HubN has its own DNS hierarchy, certificate authorities and RPKI repository. Most of the individual parts had been working for a while. The less convincing bit was what happened while they changed.
A manifest could be fetched just before its CRL changed. A root server could receive a zone update but fail halfway through applying it. A reverse zone could be correctly signed and still be impossible to reach because I had delegated it from the wrong side of an existing zone cut.
None of those failures produced a satisfyingly broken file. They produced collections of individually plausible files which didn’t describe the same moment in time. This week I went through that path from the CA to the validators and authoritative servers, and made the publication boundaries much less hopeful.
Publishing One RPKI Generation
An RPKI publication point is a set of related objects. The manifest lists the files which belong to the current generation, the CRL says which certificates are no longer valid, the ROAs contain the route-origin authorizations, and RRDP tells a validator how to move between repository states.
My first implementation updated those pieces in the right general order, but the order wasn’t a transaction. There were gaps where a validator could see the new manifest with the previous CRL, or a new RRDP serial whose snapshot hash still described the old object set.
That is the sort of thing which tends to pass a local test. Fetch the repository again a second later and everything has caught up. A validator arriving in the gap isn’t obliged to wait for my code to finish, though. It sees an inconsistent repository and is quite entitled to reject it.
The publisher now assembles a complete future generation first. Changed objects are staged, then the publication rows, RRDP delta, serial and snapshot hash are committed together. Identical objects are left alone, which avoids manufacturing a new serial just because the periodic job ran again.
The CA side also marks related manifests and CRLs with the same generation. The ROA service reads the trust-anchor manifest before and after fetching the CA objects. If the generation changed during the fetch, it throws the attempt away and starts again. It is a slightly annoying extra round trip which is preferable to publishing a beautifully signed mixture of two different states.
I also fixed the manifest serials so they survive a restart. Resetting a monotonic counter because a container was replaced is easy to do and difficult to explain to software which has already seen the larger number.
Fixing reverse-DNS tree traversal
The IPv6 space inside HubN lives below 6660::/16, so its aggregate reverse zone is 0.6.6.6.ip6.arpa.. I initially delegated that name directly from HubN’s private root.
That looks reasonable if I only inspect the root zone. It doesn’t work for an ordinary resolver.
The public DNS already delegates arpa.. Once a resolver follows that cut, a private delegation for 0.6.6.6.ip6.arpa. hidden in the root is no longer on the path. The resolver asks the public arpa. hierarchy instead and never discovers the HubN reverse zone.
The private root now carries an authoritative overlay for the parts of the path it needs:
.
└── arpa.
└── ip6.arpa.
└── 0.6.6.6.ip6.arpa.
└── delegated prefix zones
The coordinator builds and signs that hierarchy from the leaf upwards. Each child publishes its DS record into the closest managed parent, and only after the overlay zones exist does the root activate the first delegation. The old unreachable delegation is removed as part of the same reconciliation.
This also forced me to stop assuming that every label on the way to a zone is itself a managed zone. Between 0.6.6.6.ip6.arpa. and ip6.arpa. are several names which exist only because a descendant exists. DNS calls these empty non-terminals.
A Name Can Exist Without Having Records
The empty non-terminal case found another bug in the authoritative server.
If a name has no records but has descendants, it exists. A query for that name should return NODATA. My server was only checking for an exact record set, so it returned NXDOMAIN instead. With DNSSEC enabled it also built the wrong denial proof, which made a valid branch of the reverse tree look as though it did not exist.
The in-memory zone now checks both exact owners and descendant names. NSEC generation includes the empty non-terminals needed to prove the answer, and DS propagation walks upwards until it finds an actual managed parent rather than stopping at the first textual parent.
It is one of those DNS details which seems absurdly fussy right up until a validating resolver uses it to discard the answer.
Applying Registry updates after delivery
The coordinator distributes signed zones to several root servers. The old path treated a completed stream as fairly strong evidence that the update had landed. It wasn’t.
Zone transfers now carry a canonical digest made from the serial and decoded wire-format records. Ordering doesn’t affect the digest, but the actual DNS data does. A root server assembles the candidate away from the live zone, checks the digest, refuses serial regressions and only then swaps it into service.
If a delta is incomplete or its digest doesn’t match, the server keeps the previous good zone and asks for a full copy. The root server also reports the serials it has actually applied. That gives the coordinator something useful to compare instead of equating “I sent serial 42” with “serial 42 is serving.”
The same distinction now exists in the other trust path too. A newly written ROA row isn’t the same thing as a coherent RRDP generation, just as a signed reverse zone isn’t the same thing as a reachable, applied zone.
Where It Ended Up
The CA, ROA publisher, DNS coordinator and root servers still run as separate services. I don’t want one enormous process just to obtain transaction-like behaviour across them. The boundaries are now explicit instead:
- a CA generation can be read consistently or retried;
- an RPKI publication becomes visible as one RRDP generation;
- the private reverse hierarchy follows the path a resolver actually takes;
- signed zones are verified before replacing the live copy;
- serials describe applied state and survive restarts.
There is more machinery here than I expected when I decided a private network should have proper DNSSEC and RPKI. That has been true of nearly every part of HubN. The useful bit is that a failure now leaves the previous generation in service, rather than briefly publishing a newer state which never existed as a whole.