← Notebook
KNET Finding 2 minute read

The State File Was Right. The Route Was Wrong.

The failover supervisor remembered the right state while the live routing tables had quietly stopped agreeing with it.

During a dual-WAN incident, the failover state file said exactly what I expected. The live route did not.

That distinction matters because the state file is only the supervisor’s last decision. It is not proof that FRR still has the intended distances, that a later VyOS commit did not replace them, or that the kernel selected the interface implied by the state. A restart or partial transition can leave perfectly valid state attached to the wrong route.

IPv6 added a less obvious constraint. The backup line learns its default gateway from a router advertisement. I cannot reject that default route: without accepting the RA, I do not know the gateway for that line. The primary route therefore has to be owned separately and outrank the learned backup route while the primary is healthy.

I changed the supervisor so every run reconciles intent with reality. For IPv4 it reads the exact owned FRR defaults, rejects malformed routes, converges missing or duplicate distances, and then checks the kernel-selected interface. For IPv6 it keeps accepting the backup RA, tracks its lifetime, and verifies that the owned primary route and selected path agree with the state machine.

State is now written only after a transition has actually converged. If the route does not match, the transition stays retryable instead of recording a success which exists only in a file.

The same distinction is visible in monitoring now: desired state, configured route and selected route are separate signals. That is a much better description of failover than one green light labelled “up”.