One of r3’s WANs genuinely failed this morning. The supervisor moved both IPv4 and IPv6 to LTE, the kernel selected the backup route, and traffic continued to pass. Then the same supervisor reported that LTE’s IPv4 path was down too.
The live state disagreed. The IPv4 default pointed through LTE, the failover state and route distances matched, the modem gateway answered, and source-bound probes to the public targets received replies. IPv6 probes over the same backup path were clean.
Running the exact IPv4 probe exposed the contradiction. It sent three ICMP requests, received two valid echo replies, and still exited with status 124.
The timing was self-inflicted. Each target was tested with ping -c 3 -W 2 inside a hard three-second timeout. With no loss, ping usually completed before the wrapper intervened. Once LTE was carrying the connection, partial IPv4 ICMP loss or de-prioritisation left ping waiting for a missing response. The wrapper killed it before the final packet-loss summary was written.
The supervisor reduced that result to a boolean: exit status zero meant reachable; every other status meant failed. It discarded the echo replies already present in the partial output. Once two targets were misclassified, the existing two-of-three decision quite reasonably declared the path down.
Simply lengthening the timeout would have moved the problem elsewhere. A complete run performs twelve target probes serially, has a 45-second watchdog, and is scheduled once a minute. Giving every probe a longer worst case would push complete runs towards the scheduler interval and make lock contention more likely.
The fix keeps the execution deadline but separates it from the measurement. Valid echo replies now prove that a target was reachable even when the wrapper terminates ping. If the final summary is missing, the script reconstructs packet loss and mean round-trip time from the replies it did receive. A target with no replies still fails, and the two-of-three path threshold is unchanged.
The regression case reproduces the incident directly: two targets return replies with status 124, one target returns nothing, and the backup path remains reachable with degraded loss metrics. The supervisor now uses the probe evidence to decide reachability instead of treating the watchdog’s exit status as the measurement itself.