2018 in Review

As a non-profit with few reserves, whether a year was a good one at ISC has a lot to do with our financial status. 2018 was a good year on that score. Most of our funding comes through software support contracts, and in 2018, our support business was strong. We gained 22 new support contracts and lost only 10. As a result, we were able to add quite a lot of staff: three new BIND 9 developers, a senior QA person, a systems and support engineer, and a documentation/communications person. That is a 20% increase in staffing, which is huge. One BIND 9 engineer left and went to one of our OEM partners, and one BIND 9 engineer who had previously left for another open source project returned to ISC.

Updated Development Environment

The biggest event at ISC in 2018 was the migration to GitLab, for both the BIND 9 and Kea DHCP development teams. The ISC-hosted GitLab replaced the separate issue tracker, wikis, and repos for both BIND and Kea with a single development platform. We also are leveraging the continuous integration capabilities of GitLab. We invested quite a bit of effort into getting our automated BIND 9 tests to run in parallel to speed them up. Now nothing is committed to BIND 9 without first passing our continuous integration tests. The Kea team is also making extensive use of the GitLab wiki pages to post design and requirements documents in the open. The adoption of GitLab was fairly disruptive, but overall it has been a positive change.

User Participation Is Up

The shift to GitLab as our primary working repository brought us several unexpected benefits. We opened our issue tracker for BIND so new issues are now open by default; only a few sensitive issues are non-public. (The Kea issue tracker was already open.) We are able to accept contributed patches more gracefully, via merge requests (we used to require email attachments). It seems like we are getting better-quality bug reports, which could be because they are public. We have been pleased to see many people create accounts on https://gitlab.isc.org and start actively opening issues, commenting on existing issues and offering to help (although anyone can browse without an account).

We Welcomed New Kea Users

Our support business is a reasonable barometer of how our open source work is being received by users. Many of our new support customers are adopting Kea DHCP, which is gratifying because this project has been under-supported for the past several years. While the IETF work on DHCP is winding down, and the standards-setters are enthusiastic about router-advertised address assignments, DHCP seems to be a continuing core requirement of most networks. We have seen a healthy interest in our “premium” Kea hooks software package, which provides a moderate-cost (US$499) way to support ISC’s open source work.

BIND Refactoring Is Well Underway

We are very satisfied with the BIND 9 refactoring we have completed so far. Our initial goal was to reduce software complexity, but we are also adapting to modern cryptography standards and taking more advantage of well-supported LINUX OS features that didn’t exist when BIND 9 was first written. In January we released BIND 9.12, with refactored query logic (query_find and resquery_response). These massive, ridiculously complex functions were split into smaller functions and we created an external library, libns. These changes simplified the query processing code and made it much easier to subsequently implement QNAME minimization. We continued with the refactoring, tackling BIND’s network interface next. The results of that will be seen in 9.14.0, where we expect to see significant improvements in performance on modern hardware.

These are the highlights of 2018 at ISC. We go into more depth, or at least more length, below.

Customer Support

Our busiest quarter for customer support was Q2 (April-June) – and it wasn’t followed by the usual summer holiday slow-down, so our support team has been very busy all year. BIND 9 is a steady workload; ISC DHCP ticket volumes are gradually decreasing, while Kea questions and issues to our support team are steadily increasing. This is to be expected, since we are ending the year with many more Kea support customers than we started it with. We averaged just over six new support issues per week, and the return of Alan Clegg, an experienced, former ISC systems and support engineer, helped us to handle the increased workload.

Screen Shot from kb.isc.org showing the BIND 9 feature matrix

Other support-related activities:

• We migrated our Knowledgebase to a new platform. The old software was unmaintained and starting to decay, and we wanted to make sure that the KB would continue to be a useful resource for our users. This was a big project for our new Marketing Communications person, Suzanne Goldlust, because the markdown and links all had to be updated and we needed to maintain many of the old URLs, since many of the documents are linked from elsewhere. So far, user response has been positive.
• We saw increasing demand for binary packages from ISC, for both BIND 9 and Kea. Our users have asked us to incorporate software dependencies (particularly for BIND 9 – DNSTAP support), provide a fresher software version for older OS releases, and offer a “supported binary” with our support contracts that includes our premium features. We responded by contributing BIND 9 packages for Debian and Ubuntu in their repositories. We have also created our own RHEL/CentOS 6 and 7 binaries, both for open source users and for our BIND 9 Subscription Edition users. We are definitely appreciating the challenge our OS packagers have had for many years, in deciding which of BIND 9’s many compile-time options to enable for their packages! We expect to iterate on this more in 2019 in an effort to improve the ease of installation and updating. (https://www.isc.org/blogs/bind-9-packages/)

• Customers and users asked for resources on performance tuning and information on the impact of the choice of OS and features on BIND performance. We have shared what we have, but we can see a need for more experimentation in this area to meet the requests. Unfortunately, this isn’t what our Perflab tool was designed to do and it isn’t really suited to this job.
• The continued inability of the DNS standards community to settle on a “solution” to the CNAME-at-zone-apex problem is frustrating BIND 9 users. (ISC has participated vigorously in the debate about CNAMEs at the apex of a DNS zone, and has put forth three proposals: https://tools.ietf.org/html/draft-ietf-dnsop-aname-01, https://datatracker.ietf.org/doc/draft-sury-dnsop-cname-plus-dname/, and https://www.isc.org/blogs/dns-and-web-architecture/.)
• BIND Customers are curious about DNS cookies and anxious about whether this change will introduce problems. DNS cookies were first added in BIND 9.10, but now that they have matured they are enabled by default.
• After much fanfare – and years of preparation and testing – the root KSK rollover in October was wonderfully uneventful. Far more angst was caused by GDPR compliance. Now we are gearing up to be ready for a possible spike in questions related to EDNS compliance issue identification and troubleshooting, related to the upcoming DNS Flag Day. (https://www.isc.org/blogs/end-to-bandaids/)

Projects – BIND 9

Ondřej Surý, our BIND 9 Development Director, drove a lot of changes in 2018, including ISC’s adoption of GitLab, the addition of three new BIND team members, a new BIND release strategy, and termination of workarounds for legacy systems.

Early in 2018 we implemented the BIND 9 release strategy announced in late 2017; you can read about it at https://www.isc.org/blogs/bind-release-strategy-updated/. The goal was to pare down the number of supported branches to (1) development/unstable, (2) stable, (3) extended support and (4) subscription edition. We accomplished this by declaring End of Life for the 9.9 and 9.10 branches in 2018. It is too early to tell if the new model will result in improving the quality of the early releases on our next stable branch, 9.14, but it has already enabled us to get more users and more feedback on the development branch.

Besides the investment in refactoring described above, 2018 was a year of tough love for BIND users. When BIND 9 was originally written, there was a wider variety of available operating systems, including HP/UX, AIX, BSD, DOS, and Linux, and we proudly supported all of them. Many of these are no longer supported well enough to be viable options for most users, so we decided it was time to cut our support for them and focus on improving BIND for the majority. We didn’t support antiquated software to be stubborn; we did it because we really cared about supporting all our open source users. But we reached a point where that no longer made sense.

Some of the changes:

• Workarounds for old versions of UnixWare, BSD/OS, AIX, Tru64, SunOS, TruCluster, and IRIX have been removed. On UNIX-like systems, BIND now requires support for POSIX.1c threads (IEEE Std 1003.1c-1995), the Advanced Sockets API for IPv6 (RFC 3542), and standard atomic operations provided by the C compiler.
• Previously, it was possible to build BIND 9 without thread support. BIND now requires threading support (either POSIX or Windows) from the operating system, and it cannot be built without threads.
• BIND 9 will no longer build on platforms that don’t have proper IPv6 support. BIND 9 now also requires non-broken POSIX-compatible pthread support; such platforms are usually long after their end-of-life date and they are neither developed nor supported by their respective vendors.
• BIND can no longer be built without DNSSEC support. A cryptography provider (i.e., OpenSSL or a hardware service module with PKCS#11 support) must be available. DNSSEC validation is now enabled by default.
• The BIND team partnered with other open source DNS projects to announce the “DNS Flag Day” in 2019, ending support for “fixups” to accommodate non-compliant DNS implementations. This is, to some extent, the culmination of the campaign Mark Andrews started back in 2015 to expose and correct the gaps in EDNS standards compliance.
• We also removed support for Windows XP(tm), finally.

Windows is still a very popular platform for BIND, but there are no engineers on the BIND team who specialize in it, nor do we have adequate automated test coverage for Windows. We had an external contributor, Rockerinthelocker (later revealed to be Thomas Jach), who contributed several needed Windows fixes and also advised us on some Windows issues.

After removing those legacy system requirements, we were able to modernize our networking stack to take better advantage of threading and reduce context-switching. Witold “hold my beer” Kręcicki spent the last three months of 2018 refactoring the networking code. Through a massive series of commits, he eventually managed to successfully replace our ancient task manager and socket code. The manager uses per-cpu queues for tasks and the network stack runs multiple event loops in CPU-affinitive threads. This greatly improves performance on large systems, especially when using multi-queue NICs. This project is equivalent to removing the bald, flat tires from a car, while it’s in motion, and replacing them with steel-belted radials. We are expecting that these new tires will really pay off when we implement DNS over TLS in 2019.

New features added in 2018 include:

• QNAME minimization, sponsored by the Open Technology Fund.
• Mirror Zones, intended to facilitate running a local copy of the root, sponsored by ICANN.
• Implementation of Serve Stale to provide additional protection against an outage like the massive DDoS that hit Dyn in October 2016, thanks to a patch contributed generously by Akamai.

At the end of 2018, we commissioned a BIND 9 project logo from Richard de Ruijter, who also designed the new NLNET Labs project logos. It was nice to be able to afford the small luxury of a professionally-designed logo!

BIND 9 by the Numbers

Releases

We issued releases in the following branches of BIND 9 in 2018:

• 9.9.13 (this branch went to End-of-Life as of July 2018)
• 9.10.7-S (this branch went to End-of-Life as of July 2018)
• 9.11.3-S, 9.11.3-S2, 9.11.4, 9.11.4-S, 9.11.5, 9.11.5-S1, 9.11.5-S2, 9.11.5-P1 (released in January 2019)
• 9.12.1, 9.12.2, 9.12.3, 9.12.3-P1
• 9.13.0, 9.13.1, 9.13.2, 9.13.3, 9/13.4, 9.13.5

There were six BIND CVEs in 2018.

Issues Opened/Resolved

Here are some approximate numbers of BIND issues created and closed in 2018:

• 777 issues were created since the migration to GitLab (some were brought over from RT)
• 508 issues have been closed
• 269 are still open

In our pre-GitLab system, there were 96 bug reports closed/rejected and resolved in 2018, and 62 new issues opened.

Our THANKS to the following people for their technical contributions to BIND 9 in 2018

Tony Finch with an impressive 16 commits in GitLab, and five in our pre-GitLab system

Petr Menšík with 12 commits in GitLab, plus one in our pre-GitLab system

Andreas Hasenack

Bhargava Shastry

Bill Parker

Håvard Eidnes

Kevin Chen

Mathieu Arnold

Paul Hoffman

Roland Gruber

Thomas Jach

Tomas Hozza

Zhaolong Zhang

We also love to see comments, suggestions, and even complaints from all our users on the bind-users mailing list and in our gitlab.isc.org BIND project.

Projects – Kea

In 2018 we finally started building a solid base of Kea DHCP support customers, and we began to see more ISC DHCP support customers prepare to migrate to Kea. Our premium hooks packages (sold on our website for US$499) turned out to be a successful experiment that we plan to continue. Our biggest thrill of the year, however, was adding a much-needed new development team member, a senior QA engineer.

• We migrated Kea to GitLab, adopting gitlab.isc.org as both the source repo and the issue tracker. We disabled issue tracking on the Kea GitHub, which we are keeping as a read-only repository. We are still doing our integration testing in Jenkins (jenkins.isc.org), but we are looking at migrating that to GitLab in the future. Our old Kea Trac site is still available at oldkea.isc.org for reference.
• ISC participated in the Google Summer of Code (GSOC) program for the first time, mentoring two student developers who contributed a new feature to Kea and released an open source dashboard for Kea. While this was a good experience for ISC, with such a small team it can be hard to spare the resources to mentor and support GSOC students adequately. (https://www.isc.org/blogs/kea-google-summer-of-code-projects/)
• We started using the ISC Perflab tool to do performance testing of Kea.
• We released a high-availability feature that removed the last remaining significant obstacle for ISC DHCP users to migrate to Kea. By implementing a less complicated HA feature instead of failover, we were able to support multiple deployment models with one feature (DHCPv4 AND DHCPv6, the standard memfile, AND DB backend options).
• Using a significant code contribution from Deutsche Telekom, we added a Cassandra database backend.
• We produced two more premium hook libraries to help manage Kea: the RADIUS integration hook and the client classification hook.
• Kea 1.5 added YANG model support through integration with the Sysrepo open source configuration datastore.
• We added a hooks API guide to the documentation.

In addition, the Kea team spent many hours working on a thorough revision of the DHCPv6 standards, which were standardized in November, 2018 as:

• RFC 8415 (was draft-ietf-dhc-rfc3315bis) https://datatracker.ietf.org/doc/rfc8415/ Dynamic Host Configuration Protocol for IPv6 (DHCPv6).

We weren’t able to invest much more work into our ISC DHCP to Kea configuration migration tool, which we are currently testing with some of our ISC DHCP support customers. It is expected that the most difficult part of an ISC DHCP configuration to migrate will be the client classification, because ISC DHCP is so flexible. We have been identifying and tracking all the differences between the two projects that potentially could trip up some users as they migrated in a GitLab milestone (https://gitlab.isc.org/isc-projects/kea/-/milestones/6).

ISC DHCP and Kea by the Numbers

Releases

We issued two major feature releases of Kea and published our last planned major branch of ISC DHCP. Kea continues to be a very active project with an emphasis on new feature development. Currently, we produce only new feature versions, and we support only two versions at a time. As our user base grows, we are considering introduction of parallel stable and development branches, with maintenance-only releases on the stable branch.

New features added in 2018 included, by release:

• Kea 1.4 – High Availability (HA), RADIUS integration, and a supported Cassandra backend.
• Kea 1.5 – The sysrepo YANG model configuration store, HA improvements, an authoritative flag, global host reservations, and a new client classification hook module.
• DHCP 4.4.0 & 4.4.1 – the last major branch https://www.isc.org/blogs/isc-dhcp-the-last-branch/ featured improvements in DDNS, the DHCP client, and Dual-stack mixed mode (https://www.isc.org/blogs/using-dual-stack-mixed-mode-dsmm-with-ddns-in-isc-dhcp-4-4/).
• DHCP 4.1-ESV-R15, 4.1-ESV-R15-P1
• DHCP 4.3.6-P1

There were two ISC DHCP CVEs and one Kea CVE in 2018.

Issues Open/Resolved

Because we migrated our issue tracker, and migrated some of the open issues, all of our bug ticket metrics are somewhat messed up, to be honest.

We opened 393 new Kea issues in Gitlab, all of them in 2018. Some of these were migrated over from our older Trac system. We closed 295 issues in our old Trac system in 2018, 184 of them were for Kea 1.4 milestones, which was completed before we migrated to GitLab. We closed 181 tickets in the Kea Gitlab instance to date. So, we closed at least 181 + 184 = 365 issues, and probably more. At any rate, it seems that we are closing issues roughly as fast as we are opening them, which is a good indication that we are keeping up.

Of the 86 new ISC DHCP tickets, 40 (nearly half) were opened as confidential tickets. 21 of the tickets opened in 2018 were resolved or closed in 2018. Overall, we resolved 26 ISC DHCP tickets and closed another four in 2018.

We thank the following people for their technical contributions to ISC DHCP and Kea DHCP in 2018

The following people submitted patches, reported bugs, or suggested valuable features or changes to ISC DHCP in 2018:

Bill Shirley

Brad Fitzpatrick

dgutier-at-cern-dot-ch

Felix Wilhelm of the Google Security Team

Fernando Soto from BlueCat Networks

Gabriel Valcazar

Peter Anvin

Indy, of the Fireball ISO open source project

Jiri Popelka of Red Hat

Naiming Shen and Enke Chen of Cisco

Pavel Kankovsky

Pavel Zhukov at Redhat

Peter Lewis

Tim DeNike of Lightspeed Communications

For Kea, our top contributor was Razvan Becheriu. Other 2018 contributors included:

Andrei Pavel

Franciszek Gorski

Franek18

Piotr Strzyewski

Plyul

Sebastian Schrader

Sunil Mayya (our GSOC student)

Research

In 2018, our research team worked on the following projects:

• ethq, a Linux NIC monitoring tool (https://www.isc.org/blogs/ethq-linux-nic-monitoring-tool/)
• dnsgen – a packet generator https://www.isc.org/blogs/dnsgen-a-dns-packet-generator/
• Updating the ISC Dig app, available on iOS; 2271 copies were downloaded in 2018 (and over 500 of these were from Japan!)
• The ISC DNS Checker app, available on iOS; 670 copies were downloaded in 2018. It was most popular in the US, Germany, and the UK.

Other Random Statistics

Presentations by ISC: 17 (https://www.isc.org/mission/webinars/)

ISC Staff: 32 in 9 countries and 8 US states

2018 email traffic on user mailing lists:

Emails for help or information to info-at-isc: 883.