The last few days of HubN work started with broken DNS and ended with a link to an AS-SET. In between were certificate chains, wildcard records, a wiki refresh and several small websites.
There wasn’t a grand feature tying it all together. Mostly, I followed the things which were confusing or broken when somebody actually used the network. A reverse lookup returned an answer from the authoritative server but failed through a validating resolver. A certificate existed but belonged to the system account. A WHOIS result omitted routes which the ASN was allowed to originate.
Each looked fairly small from the outside. DNS was particularly committed to preserving that illusion.
The Record Was There
I wrote about HubN’s DNSSEC publication path in August. September supplied some additional test cases.
The immediate failure was a reverse lookup. The root servers could return the PTR record, but the signatures needed to validate it were missing from the response. The recursive resolver returned SERVFAIL. Asking the authoritative server directly made the record look healthy, which was a good way to spend time looking in the wrong place.
The repair crossed several boundaries. Signature refresh needed to notify distribution. DNSKEY signatures needed proper renewal. Signature-only updates needed new serials, even though the underlying address or hostname had not changed. Roots reconnecting to the coordinator needed a complete, validated publication to catch up from.
The coordinator now persists the signed zone snapshots which it distributes. Working records can change while a zone is being assembled, but a candidate has to pass publication validation before replacing the saved snapshot. Reconnecting servers receive that published state.
Registry reconciliation also became more thorough. It fetches every page of records, repairs missing or stale state periodically, and returns signing failures instead of calling an incomplete sync successful. Reverse-DNS changes now publish their Registry events after the database mutation commits, including enable, disable and PTR record changes.
There was a second failure hiding in parent delegations. An old automatically generated DS record could conflict with the record the Registry sync wanted to manage. The child could have the right key while its parent still pointed at the wrong one.
That path now adopts the recognised legacy records at the exact delegation name, preserves the keys and refuses unrelated ownership conflicts. It also leaves healthy, unchanged parents alone. Repeated reconciliation should not need a new signature over half the tree just to establish that nothing happened.
The local checks exercised AAAA and PTR creation, modification and deletion through to a validating resolver. Deletion mattered: an answer disappearing is only half the job when the server also has to prove that it is gone.
Wildcards Needed More Than an Asterisk
Hosted DNS did not accept wildcard record names. Allowing a leading * in the Registry was the easy part. The authoritative servers also needed to know when to use those records, and the signers needed to describe them correctly.
A wildcard is selected relative to the closest existing ancestor of the queried name. Existing names and delegation boundaries affect that choice; the server cannot just keep stripping labels until it finds an asterisk. Empty names with descendants count too. That behaviour is spelled out in RFC 4592.
The root server now constructs wildcard answers with the appropriate denial proofs and handles the case where the wildcard exists but has no record of the requested type. Exact names and delegated zones retain their own behaviour.
Then there was the signature’s label count. For *.example.hubn., the RRSIG Labels field is 2: neither the asterisk nor the terminating root label counts. The wildcard remains part of the name being signed. A validator uses the count to reconstruct that original name after expansion. RFC 4034 has the details.
Both signing paths now agree on that. Adding one character to a record name ended up touching the Registry, the CA and the DNS servers, which seems about right for DNS.
Certificates Had Two Different Problems
ACME-issued certificates were registered under the system handle. Issuance could succeed while the certificate failed to appear among the domain owner’s resources.
ACME now resolves the registered owner of every name before signing. All names in an order must resolve to the same owner; mixed ownership needs separate orders. The lookup uses the most specific registered domain and normalises wildcard prefixes, case and trailing dots.
A Registry migration also repairs existing system-owned ACME server certificate records where the names resolve unambiguously. It changes their ownership metadata, leaving the certificates themselves alone. Unknown or mixed-owner records are left for inspection rather than assigned to whoever happens to own the first name.
The other problem was inside the certificates: missing Authority Key Identifiers. AKI identifies the issuer’s signing key and, in our issuance path, must match the Subject Key Identifier in the issuer certificate. This is part of the certificate profile in RFC 5280, and strict verification is considerably less forgiving of its absence.
The shared certificate builders now derive that identifier from the actual issuer certificate. The fix covers the service signer and internal PKI paths as well. The regression tests include strict OpenSSL verification and a Python TLS handshake, so the check goes beyond finding an extension in the output.
Fixing the builders only helps certificates issued afterwards. I also added a durable, one-time deployment job which walks the existing sub-CAs from their parents downwards and repairs the affected certificates. It preserves their keys, subjects, validity and constraints, saves the original Registry records, and records completion so a restart does not issue everything again.
That still doesn’t replace a chain already installed in a service. Existing leaves need their normal renewal and installation path. The Registry holding a corrected intermediate is useful, but it cannot reload somebody else’s TLS server by implication.
WHOIS Should Answer the Useful Question
Some of the smaller changes were about what the Registry tells people.
Hosted domains now show the nameservers from the hosting configuration, instead of empty or stale delegation fields. Their WHOIS DNSSEC flag also reflects automatic hosted signing. That flag describes the configured service; it isn’t a fresh validation of the domain every time somebody opens the page.
ASN results now include the Registry’s IRR prefixes. The web views show the list, and the text output includes route or route6 objects with the origin ASN. This uses the routing permissions already in the Registry.
The first version still missed some prefixes. It looked like an ownership problem, but the ordinary prefix query already used the allowed-origin list independently of ownership. The omission was anycast allocations, which live in a separate table.
The shared resolver now combines active ordinary prefixes and active anycast allocations which explicitly allow the ASN, removing duplicates. Owning a prefix is not enough to include it, and somebody else’s ownership is not enough to exclude it. The same resolver feeds IRRd queries and AS-SET expansion, so the correction reaches the route-filtering path too.
WHOIS also has query URLs now. A path such as /q/AS42%3AAS-DOWNSTREAMS opens the corresponding AS-SET directly. The query survives a refresh, and browser history works when moving between searches.
The AS42 site uses that link. Sending somebody to a particular object is a small improvement over sending them to a dashboard and expecting them to reconstruct what I meant.
Somewhere to Send People
There are now small internal sites for the Registry, CA, root servers and IRC, as well as AS42.
The pages give each service somewhere to explain itself. The CA page has the hierarchy and certificate downloads. The root-server page has bootstrap material and resolver guidance. IRC has connection details. Resource management stays in the portal, but reading about a service no longer has to begin there.
I also went through the wiki against the current portal workflows. It had accumulated old configuration blocks, obsolete bootstrap instructions and descriptions of processes which had since moved into the portal. The refresh replaces those with current guides for peering, identities, AS-SETs, reverse DNS and the other resources, plus a directory of places to visit.
That is where this round of work ended: a reverse lookup which can be validated, certificate records attached to the right owner, an ASN lookup which includes its anycast permissions, and a URL I can paste when somebody asks what AS42 announces.
I would still like the next small DNS change to be small.