The HubN Registry stores allocations and requested state. Other services use that information to publish reverse DNS, generate ROAs, issue certificates and decide which BGP sessions ought to exist.
For a long time the Registry wrote a change to PostgreSQL and then published an update through Redis. That was fine as long as Redis was available and the Registry process remained alive for the few milliseconds between those two operations.
That “and” was doing rather a lot of work.
If the database commit succeeded and publication failed, the Registry had the new state but its consumers did not know to fetch it. Retrying the API request might produce the event, reject it as a duplicate or change the resource again, depending on the endpoint. None of those are especially good recovery procedures.
I replaced that path with a transactional outbox and then went through the consumers to make sure duplicate delivery and missed history were both survivable.
Putting the Event Beside the Change
Infrastructure updates are now inserted into an outbox table by PostgreSQL triggers. The resource change and its event are part of the same transaction, so either both exist or neither does.
The triggers cover the changes which have operational consequences: prefix routing and reverse-DNS settings, PTR records, anycast route origins, certificate authorities, new certificates and revocations. API handlers no longer have to remember which Redis message belongs after which database call.
A Registry worker claims pending rows in batches and publishes them. Multiple Registry replicas can run the worker at once; SKIP LOCKED keeps them from picking the same row during the normal case. Claims expire after five minutes so a dead worker can’t own an event forever.
Failed publications use bounded exponential backoff. Successful rows remain for seven days before cleanup, which has already made debugging nicer because there is an actual record of what the Registry tried to announce.
There is one uncomfortable part which can’t be removed with a table: the worker may publish an event and crash before marking it complete. The event will be sent again after its claim expires. The delivery guarantee is therefore at least once, not exactly once.
That is intentional. Losing an update is worse than asking a consumer to process the same desired state twice.
Ordering Still Matters
Duplicates aren’t the only awkward case. Suppose a prefix has reverse DNS enabled and then disabled while Redis is down. Once publication resumes, sending those two events in the opposite order would leave the DNS service doing exactly the wrong thing very reliably.
Each outbox row has an aggregate key derived from the affected prefix, anycast allocation or other resource. Workers can publish changes for different resources in parallel, but a later event for the same resource cannot pass an earlier pending one.
The event remains small. It says what changed and identifies the resource; it doesn’t attempt to be a second copy of the Registry row. Consumers fetch the current representation before acting. If several updates collapse into one current state, that is fine. They are reconcilers, not historical replay engines.
Updating the Registry event consumers
An outbox only closes the gap between PostgreSQL and Redis. It doesn’t make the code on the other end correct.
The ROA service had a particularly unpleasant edge case. A prefix allocation and an anycast allocation can legitimately describe the same ASN, prefix and maximum length. The old uniqueness rule let one source effectively take ownership of the other’s row. Deleting either source could then withdraw the shared ROA even though the other authorization still existed.
ROAs are now tracked by their Registry source. Retiring one source only removes the published object when no other active source still needs the same URI.
The service also performs periodic full reconciliations in addition to handling live events. It fetches all active prefix and anycast sources, processes them, then reads the source IDs again before withdrawing anything. If the Registry changed during the paginated scan, withdrawal is refused and the next reconciliation starts over. A failed Registry request keeps the last known ROAs in place instead of interpreting “I couldn’t fetch it” as “it no longer exists.”
That last distinction sounds obvious when written down. It was less obvious in code where an empty result and an unavailable service can both arrive near the same branch.
Rebuilding the Looking Glass Safely
The Looking Glass has the same general problem with a different kind of state. It receives routes from GoBGP and session intent from the Registry. After a collector reconnect, it needs to rebuild its view without showing a mixture of the old and new RIB.
Routes are now loaded into a new RIB generation. The previous generation remains queryable until the replacement has completed, at which point the service switches generations and retires the old one. A disconnect halfway through a dump leaves the last complete view available.
I also split two session cases which I had previously treated alike. A Registry session which is still waiting for the collector is pending and should remain visible. Collector status for a session which no longer exists is stale and should be pruned. “Not currently observed” isn’t enough information to decide between them.
The DNS coordinator follows the same pattern after the publication fixes: live events make it react quickly, while periodic comparison and applied serial reports repair anything which fell out of step.
What the Event Means Now
The Registry remains authoritative for allocations and requested state. Redis is a wake-up path, not the only surviving copy of a change and not proof that a consumer applied it.
The resulting flow is less clever and much harder to lose:
- PostgreSQL commits the resource and its outbox row together.
- A worker publishes the event, retrying when it cannot.
- The consumer fetches the current state and applies it idempotently.
- Periodic reconciliation compares the complete source again.
- The service which owns the operational state reports what actually became active.
There are still windows where two systems disagree. This isn’t a distributed transaction pretending otherwise. The difference is that those windows now have a durable repair path, and a temporary outage no longer requires me to remember which checkbox to toggle to make a service notice the database again.
Previously: Fixing HubN’s DNSSEC and RPKI Publication.