Jubaleth — Blog & Notebook

Taking SkyeEngine Fabric to 10 Gigabits

Jubaleth — Fri, 21 Aug 2026 00:00:00 +0000

I ended my last SkyeEngine Fabric post with an unresolved problem. The client-to-client transfer design worked, but the fastest numbers came from containers on one machine. Restarting those containers could move the result, and the available remote environment could not cleanly separate Fabric from shared storage, shared CPUs and virtual networking.

That sentence became a test plan.

I have now run the transfer campaign on seven DigitalOcean Droplets connected by a private 10 Gb/s network. The short result is that direct Fabric transfers reached a median 1,189.3 MB/s on fresh endpoints, or about 9.51 Gb/s of file data. A forced four-coordinator relay reached 615.9 MB/s. Direct transfer came within roughly 2% of the raw private-network baseline, relay memory stayed bounded while payloads grew from 1 to 8 GiB, and every measured file passed independent integrity checks at both ends.

The less convenient result is that relay throughput changed by 5.5% when I replaced both endpoint machines. That narrowly missed the provisional 5% repeatability gate. I am accepting the campaign, but not sanding that edge off the result.

A Network Designed to Answer One Question

The earlier rented-server tests were about Fabric as a distributed control plane. They exercised 10,000 simultaneous authenticated clients, route recovery, identity churn, compression, signature enforcement and coordinator restarts. They were useful tests and the wrong environment for a serious file-transfer number.

This environment was built around the data path instead.

It had a root coordinator, two distribution coordinators and two access coordinators, plus separate source and receiver machines. The four peer coordinators and both endpoints used dedicated-CPU plans. The small root did not carry file data. The endpoint plans provided large local storage, and all Fabric and transfer traffic stayed on a private VPC. Public addresses existed for management, with every non-SSH service port blocked.

The relay route crossed access-a, distribution-a, distribution-b and access-b. The direct route still used Fabric for identity, authorisation, endpoint introduction and audit, but the file bytes moved straight between the two clients. That is the architectural distinction I wanted the benchmark to preserve.

Before running Fabric at all, repeated iperf3 tests measured approximately 9.70–9.74 Gb/s in both directions. Source storage produced 1.23 GB/s writes and 4.75 GB/s reads under the direct-I/O preflight; the receiver produced 1.53 GB/s writes and 1.48 GB/s reads. Those were not performance claims for Fabric. They were the ceilings against which its results would have to make sense.

The Smoke Test Was Not the Benchmark

The first infrastructure smoke moved 64 MiB at 494.8 MB/s direct and 292.9 MB/s relayed. Both paths verified the payload and used the expected route, so it did its job: the machines, network, credentials and protocol could form a working system.

It was also a poor steady-state throughput measurement. With such a short payload, connection setup, authorisation and measurement granularity occupy a large fraction of the run. Moving to an 8 GiB payload more than doubled both results without changing the implementation.

The previous one-host laboratory numbers had the opposite problem. They reached 1,832 MB/s direct and 950 MB/s relayed, but both clients shared one kernel, one physical host and a memory path. They were useful for profiling the implementation. They were never evidence that Fabric would move 14.7 Gb/s between real machines.

Different rows remove different bottlenecks. Reading this as a provider leaderboard would miss the point.

The 8 GiB sink baseline on the first endpoint pair reached 1,150.1 MB/s direct and 584.0 MB/s relayed. After replacing both endpoint Droplets from the same immutable snapshot, repeating preflight and registering fresh Fabric identities, the five-round medians rose to 1,189.3 and 615.9 MB/s respectively.

The fresh direct result corresponds to about 9.51 Gb/s, against a fresh raw-network median of 9.73 Gb/s. That puts the direct path at approximately 97.8% of the measured VPC baseline. There is not much unexplained network left in that case.

Watching the Four-Hop Relay

The relay has a different shape. Every byte crosses four coordinator processes, each enforcing the Fabric path while forwarding data without accumulating the whole file. Its first sink median was 584.0 MB/s and its fresh-endpoint median was 615.9 MB/s, equivalent to roughly 4.7–4.9 Gb/s of end-to-end file data.

The repeated blocks are separate test stages. This panel uses one-minute rolling counter rates and includes multiple coordinator hops; the benchmark medians come from the per-transfer result ledger rather than the visual peak.

The Grafana trace makes the campaign structure visible: correctness checks, sink runs, receiver writes, congestion-control transitions, network-shaping profiles, endpoint reincarnations, the memory ladder and the final repeat. Direct file bytes are deliberately absent from this graph because coordinators do not carry them.

The coordinator-flow panel provided the second half of that check. Direct transfers produced no relay-complete records. Forced relay produced records on all four expected coordinators and traffic on the expected access-a → distribution-a → distribution-b → access-b path. Automatic mode fell back to that relay when I disabled direct listening, while strict-direct mode failed loudly instead of quietly changing the test.

The addresses in the legend are Fabric identities from the preserved evidence, not public endpoint addresses.

Every correctness case used SHA-256 plus an in-band BLAKE3 check. Both access coordinators recorded matching audit halves, and all ten receiver-write samples independently matched the source digest. A throughput number without those checks would only prove that some bytes went somewhere quickly.

Adding Bandwidth, Delay and Loss

An unshaped data-centre link is the easiest possible network. I also imposed six profiles on endpoint egress so the two directions together produced the requested round-trip delay:

Network profile	Direct	Four-hop relay
1 Gb/s, 20 ms RTT	111.2 MB/s	111.4 MB/s
2.5 Gb/s, 20 ms RTT	278.5 MB/s	279.5 MB/s
5 Gb/s, 20 ms RTT	551.7 MB/s	538.9 MB/s
10 Gb/s, 20 ms RTT	839.9 MB/s	609.3 MB/s
1 Gb/s, 30 ms RTT, 0.1% loss	110.8 MB/s	111.2 MB/s
1 Gb/s, 60 ms RTT, 0.5% loss	110.2 MB/s	110.5 MB/s

Below 2.5 Gb/s, path selection barely mattered: both modes were constrained by the network. At 5 Gb/s the two remained close. At 10 Gb/s with 20 ms of round-trip delay, direct pulled away while relay was already near its unshaped ceiling. That is a much more useful boundary than declaring one universal Fabric speed.

The loss profiles also held close to the 1 Gb/s shaped result under BBR. Cubic was directionally slower on the unshaped baseline—1,016.4 MB/s direct and 558.6 MB/s relay, against 1,099.1 and 590.8 MB/s after restoring BBR—but I do not have enough independent environments to turn that difference into a general congestion-control claim. The distributions and endpoint variance are large enough that “BBR won this matrix” is safer than “BBR is always 8% faster.”

Storage Became the Next Ceiling

Sink mode deliberately discards received bytes after verification so it can measure the transport. Enabling receiver writes reduced the medians to 653.0 MB/s direct and 512.2 MB/s relayed.

That convergence is important. Direct still had almost twice the transport headroom of relay, but only 27% more throughput once the receiver stored the file. Optimising the direct network path would not make that particular workload twice as fast. Faster storage, deeper I/O overlap or a workload which can consume data as it arrives would matter more.

This is also why I do not want one headline number in the API or README. A model checkpoint copied into memory, a dataset persisted to local storage and a transfer crossing a lossy WAN are three different measurements even when they use the same protocol.

The Coordinators Did Not Buffer the File

The forced-relay memory ladder sent 1, 4 and 8 GiB payloads through a 1 Gb/s, 20 ms path. If the implementation retained data in proportion to the file, coordinator RSS should have climbed with every rung.

It did not.

The largest per-coordinator observation was about 171 MiB during the 1 GiB rung. The largest values during the 4 and 8 GiB rungs were about 60 MiB.

The graph is non-monotonic because connection setup, allocator behaviour and scrape timing are more visible than file size once forwarding is streaming correctly. The 8 GiB relay still held exactly 111.4 MB/s under the shaped 1 Gb/s profile. This is the result I cared about more than a low idle-memory number: the forwarding path stayed bounded while doing sustained work.

Repeatability Was Almost Boring

Five receiver-container incarnations on the first machine produced per-incarnation medians of 1,027.0–1,179.0 MB/s direct and 581.1–605.7 MB/s relayed. Each incarnation used a fresh sender container per transfer. That variation confirmed the local warning: process and endpoint lifetime can move a benchmark even when the code and nominal machine do not change.

Replacing both endpoint Droplets was therefore a separate acceptance stage, not an afterthought. The new pair repeated the raw network and storage preflight, received new Fabric identities and ran the same five-round 8 GiB baseline against the unchanged coordinator network.

Direct changed by +3.4% and stayed within the provisional 5% cross-incarnation gate. Relay changed by +5.5% and missed it narrowly. The fresh relay run itself was fairly tight, with 3.6% range-to-median spread; the first endpoint’s relay samples had an 11.25% spread. My conclusion is not that relay is unstable, nor that 615.9 MB/s is the one true number. It is that its current ceiling sits around 600 MB/s in this environment and needs another independent run before I would advertise a tighter tolerance.

The Failed Runs Stayed in the Evidence

The clean suite was the fourth full attempt. The first hit OpenSSH connection throttling while public scanners competed with the management harness. The second raced route propagation after restarting peer coordinators for the Cubic test. The third let an SSH child consume the remaining shell matrix and skipped five shaped profiles.

Those failures did not disappear when the harness was repaired. They remain in the evidence bundle and in the Grafana history. Management SSH now uses bounded connection attempts and multiplexing. Route readiness retries only the explicit pre-transfer “no route” refusal; integrity or transfer failures remain fatal. SSH children which should not read the test plan now receive null input, and suite completion requires proof that all six network profiles ran.

After the successful suite I exported and verified an offline Prometheus/Grafana snapshot, replaced the endpoints, ran the repeat, exported a second snapshot and then destroyed all seven paid Droplets. The final bundle contains 448 metric names plus machine-readable transfer results, host facts, logs, audits, image identity and checksums. The screenshots in this post came from that local copy after the expensive environment no longer existed.

The conservative catalogue-rate estimate for the complete cluster window was no more than $7.13 before tax. It is an estimate, not a provider invoice.

What I Think the Campaign Proved

The direct design is doing what it was supposed to do. Fabric can authorise and audit a client-to-client transfer without becoming its data plane, and on this test it reached 97.8% of the measured private-network baseline.

The relay is viable rather than ceremonial. It sustained roughly 600 MB/s through four coordinator processes, survived delay and loss, fell back automatically and did not buffer an 8 GiB file into memory. It is also now the obvious performance target: above roughly 5 Gb/s, relay mechanics—not the VPC—set the ceiling.

Storage can erase much of the difference between those paths. Short smokes can understate both of them. One-host benchmarks can overstate both of them. Congestion control, process lifetime and endpoint placement all move the result enough that they belong in the evidence rather than in footnotes added afterward.

Most importantly, I now have a repeatable environment and harness which can tell those explanations apart. I can close the transfer-acceptance task with a real result and one explicit repeatability caveat, rather than another attractive number from whichever run happened to be fastest.

Measurements were completed on 21 August 2026 in DigitalOcean FRA1. Throughput values are decimal MB/s; payload sizes are binary MiB/GiB. The 10 Gb/s description refers to the provider VPC class, whose measured bidirectional baseline was approximately 9.70–9.74 Gb/s. Grafana screenshots are from the verified offline evidence snapshot, not a live paid cluster.

What the Preview Token Actually Approves

Jubaleth — Tue, 18 Aug 2026 15:12:00 +0200

I split network deployments into Preview and Apply jobs so the diff can be reviewed before anything writes to a router. The first version produced a cryptographic review token, but Preview and Apply disagreed even when the rendered configuration was identical.

Semaphore checks the project out into a new task-local directory for every run. I had accidentally included that path in the token input. The token proved where the preview happened, not just what had been reviewed.

The approval now binds stable facts: the source revision, selected hosts, rendered artifacts, semantic configuration changes and the relevant live state. A different checkout can reproduce it. A changed commit, changed candidate or changed device state cannot.

VyOS then found a second version of the same mistake. One render proposed an empty configuration node. The text was different, so the automation expected a change; VyOS quite reasonably built the candidate and reported that there was nothing to commit. I removed the redundant node and added a regression check rather than teaching the deployment to distrust the router.

That left me with a clearer rule for the workflow: the review must describe an operation the target system can actually perform. Temporary paths, formatting and empty containers are implementation noise. If they can change an approval token or manufacture a diff, the approval is attached to the wrong thing.

The State File Was Right. The Route Was Wrong.

Jubaleth — Tue, 18 Aug 2026 13:55:00 +0200

During a dual-WAN incident, the failover state file said exactly what I expected. The live route did not.

That distinction matters because the state file is only the supervisor’s last decision. It is not proof that FRR still has the intended distances, that a later VyOS commit did not replace them, or that the kernel selected the interface implied by the state. A restart or partial transition can leave perfectly valid state attached to the wrong route.

IPv6 added a less obvious constraint. The backup line learns its default gateway from a router advertisement. I cannot reject that default route: without accepting the RA, I do not know the gateway for that line. The primary route therefore has to be owned separately and outrank the learned backup route while the primary is healthy.

I changed the supervisor so every run reconciles intent with reality. For IPv4 it reads the exact owned FRR defaults, rejects malformed routes, converges missing or duplicate distances, and then checks the kernel-selected interface. For IPv6 it keeps accepting the backup RA, tracks its lifetime, and verifies that the owned primary route and selected path agree with the state machine.

State is now written only after a transition has actually converged. If the route does not match, the transition stays retryable instead of recording a success which exists only in a file.

The same distinction is visible in monitoring now: desired state, configured route and selected route are separate signals. That is a much better description of failover than one green light labelled “up”.

When the WAN Failover Script Tripped Over Its Own Lock

Jubaleth — Tue, 18 Aug 2026 00:00:00 +0000

This morning r3 in Leipzig started producing the sort of alert storm which makes a dual-WAN setup look much less reassuring than it did on the diagram.

Both WANs appeared to flap. The local platform behind the router disappeared from monitoring. LTE traffic showed up when the failover state said the fixed line should still be active. For a while it looked like a provider failure, a broken failover controller and several dead hosts had all arrived together.

The first incident was not Vodafone at all. UniFi had helpfully updated switches again, despite this being a thing I keep trying to disable. Both WANs cross the same local LACP fabric before they become independent paths, so dropping that fabric can make two providers fail at once and take the downstream platform telemetry with them.

That explained the morning outage. It did not explain why the failover supervisor later tripped over its own lock while trying to send a notification.

Fixing that second incident turned into a fairly thorough rewrite of the boundary between route state, live routing, supervision and monitoring. It also produced a Grafana dashboard which can answer a question I could not answer cleanly yesterday: what did each part of the failover decision know, and when did it know it?

Two Incidents Which Looked Like One Problem

r3 has a fixed primary connection and a mobile backup. IPv4 and IPv6 fail over independently because one address family can fail while the other remains completely usable. Every minute, the supervisor tests both paths through their own interfaces and routing tables, counts consecutive results, and changes only the affected default route once the configured threshold is reached.

The early incident started below all of that. Two LACP members dropped together during an automatic switch update. Router telemetry, routing sessions and the Proxmox hosts behind the site all became noisy at roughly the same time.

That correlation mattered. Two providers becoming unreachable together is possible, but two local Ethernet members dropping at the same moment is a much better explanation when both services cross them. The incident was a local switching failure with a broad blast radius, not simultaneous bad luck from Vodafone and the mobile carrier.

Later, at 13:48, the primary IPv6 probes failed for the first time. The next round failed again, so at 13:49 the supervisor performed the IPv6 transition to LTE. IPv4 remained on the primary line.

At first it looked as though failover had not happened. The journal showed something more irritating:

Time	What the supervisor did
13:48	First failed IPv6 round; counter moved to one of two
13:49	Second failed round; IPv6 route changed to LTE
13:50	Scheduled run skipped because the previous invocation still held the lock
13:51	Another scheduled run skipped for the same reason
13:52	Supervision resumed and recorded the first healthy recovery round

The route transition had succeeded. The process then called the notification webhook while it still owned the execution lock. That request was slow enough for two one-minute scheduler invocations to collide with it and give up.

So the failover script had not failed in the obvious way. It had made the right routing decision and then temporarily blinded itself while announcing the decision.

The Lock Was Protecting Too Much

The lock exists for a good reason. Two supervisors must not edit policy rules, routes, counters or connection tracking at the same time.

A notification is different. It is best-effort work against an external service. Its latency has no business deciding whether the router performs the next health check.

I moved notification delivery outside the critical section and made sure the child process cannot inherit the lock descriptor. Notifications now have a short timeout of their own, while the complete supervisor run has a hard deadline comfortably below its one-minute schedule.

Lock contention is also visible independently. A blocked invocation cannot publish the ordinary end-of-run metrics because it never owns the main state, but it can safely increment a separate contention counter and timestamp. The dashboard now shows the latest run duration, whether it succeeded, the age of the last completion, how many invocations were skipped and when contention last happened.

That turns “the cron job probably ran” into something measurable. If a run approaches the scheduler interval, retains the lock or stops completing, Grafana has enough information to say which failure occurred.

The decision chain over the incident day. The older aggregate metrics retain the route event around 13:50; the new per-target and supervisor detail begins after deployment later that afternoon. Prometheus does not invent samples from before a metric existed.

Reconciling supervisor state with live routes

The lock bug was not the only correctness problem the incident exposed.

The supervisor’s state file could be right while the live default route was wrong. The file records the last decision the script completed. It cannot prove that FRR still has the expected administrative distances, that a later VyOS commit did not restore the configured defaults, or that the kernel is actually using the interface implied by the state.

IPv4 now reconciles those facts on every run. The supervisor parses only exact owned default routes with numeric distances. Missing or duplicate routes are deliberately converged, changes are verified in FRR, and the transition is not recorded until the kernel-selected interface agrees. If any of those checks fail, the transition remains retryable.

IPv6 needs a different mechanism. The mobile line learns its default gateway from a router advertisement, so rejecting that RA would also throw away the information needed to use the backup. r3 continues accepting the learned mobile default. While the primary is healthy, the supervisor owns a separate primary route with a better metric and verifies that it outranks the RA. During failover it removes only that owned route, leaving the learned backup intact.

This is why I am not using VyOS’s native WAN load-balancing feature. It is old Vyatta machinery, it does not provide the dual-stack behaviour I need, and it wants ownership of routing and firewall state which this router already manages explicitly.

The two address families remain independent. Route changes remain runtime-only, so a reboot returns to the reviewed VyOS configuration. Connection tracking cleanup is scoped to the family which changed rather than flushing unrelated sessions.

The control-plane view after the hardened collector was deployed. Configuration intent, routing-daemon state and kernel selection are three separate pieces of evidence.

Making the Supervisor Boring

Once I was touching the script, I removed several other ways it could become interesting during an outage.

Runtime state now lives under /run, is validated when read, and is written through temporary files followed by an atomic rename. Corrupted counters or unknown state names are rejected rather than fed back into routing decisions.

Policy-rule cleanup and route deletion loops are bounded. Probe commands have per-command timeouts and the complete invocation has a watchdog. Dry-run mode no longer creates state or lock files. The notification credential is deployed separately as vaulted, root-readable material rather than living in the script.

The probes themselves now retain useful quality information. Each address family and path tests three independent targets. The two-of-three result still drives the state machine, but Prometheus also receives each target result, packet loss and round-trip time. One bad target can therefore appear as degradation without being misreported as a dead WAN.

I added regression cases for FRR restarts, VyOS commits during failover, missing and duplicate defaults, partial transitions, kernel-route disagreement, changing or expired router advertisements, damaged state, bounded cleanup, overall deadlines and genuinely write-free dry runs. None of this required deliberately taking a production WAN offline.

Building the Incident Dashboard

The old monitoring could tell me that Vodafone probes had failed, LTE was carrying traffic or a local carrier had dropped. It was much less good at showing the complete decision chain on one timeline.

The expanded Leipzig WAN dashboard is arranged in the order I investigate a problem:

Supervisor health: completion age, duration, result and lock contention.
Decision inputs: every target, packet loss, RTT and the aggregate two-of-three path result.
State-machine progress: consecutive failure and recovery counters plus transition attempts.
Control-plane state: expected and actual IPv4 distances, IPv6 owned-route state, mobile RA presence and remaining lifetime.
Data-plane result: the default path the kernel actually selected.
Correlation: local LACP/carrier events, downstream platform reachability, WAN traffic, errors and drops.

The selected-range summaries calculate degraded primary time, time routed over LTE, attempted and successful transitions, supervisor skips, failed-run time, carrier drops and route-disagreement time. Grafana alert annotations sit on the same history.

Probe quality after deployment. The mobile path is slower, as expected, but all targets are reachable and the aggregate path decisions remain healthy.

There are alerts for a stale or failed supervisor, a run approaching the scheduler interval, any lock contention, repeated transition failure, route configuration disagreement, a missing or expiring mobile RA and an unavailable backup while the primary is still active. A single degraded probe target is deliberately a non-paging warning.

The metrics use fixed labels. The expanded contract adds 53 series, bringing the complete WAN metric family to 88 on the live collector. Prometheus remains the historical store; the router does not write a high-frequency diary into persistent configuration storage. Retention is capped by both time and size, and host disk alerts are set to fire before Prometheus can reach that storage ceiling.

I also added a report generator for longer investigations. It writes Markdown for a human incident summary, CSV for timelines and JSON retaining the evidence and query metadata. The report distinguishes observed facts from inferred cause and can classify the likely failure domain as provider reachability, backup unavailability, local switching, control-plane drift, supervisor failure, downstream platform failure or ambiguous.

What I Can Prove Now

The most useful result is not that the graph is green again.

I can now prove that a scheduled run completed recently, how long it took, what every probe target returned, how the aggregate decision was reached, which transition was attempted, whether the routing daemon accepted it and which interface the kernel selected. If the router and state file disagree, that disagreement has its own history instead of being discovered during an SSH session.

The dashboard also preserves the difference between the two incidents from this morning. The switch update appears as a local carrier event with a downstream platform impact. The later IPv6 event appears as failed primary probes, one address-family transition to LTE, two skipped supervisor runs and then recovery. They happened on the same router and involved some of the same alerts, but they did not have the same cause.

What looked like a failed failover was a successful route transition followed by a blind supervisor. The fix was not to make the state file more authoritative. It was to make the script continuously reconcile state with the actual control and data planes, keep optional work outside the lock, and leave enough evidence to reconstruct the result later.

Previously: Making KNET Behave Under Load and Monitoring KNET and HubN.

Notebook: The State File Was Right. The Route Was Wrong..

Three Judges Looked at the GLaDOS 3.0 Corpus

Jubaleth — Tue, 18 Aug 2026 00:45:00 +0200

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

GLaDOS, Portal, Aperture Science, and related names, characters, logos and game content are intellectual property of Valve Corporation. This independent research project, its models, and this website are not affiliated with, endorsed by, sponsored by, or associated with Valve Corporation. No ownership of Valve's intellectual property is claimed.

PRIVATE RESEARCH STATUS

GLaDOS is private research. Its source, training data, models, checkpoints and weights have never been released or distributed, and discussion of the project does not constitute an offer or intention to release them.

Five days ago I wrote that the GLaDOS 3.0 corpus had become the project. At the time, one independent review was still running and the third had not finished. The numbers in that post were an in-flight snapshot.

All three judgment censuses are now complete.

GPT-5.4-mini, Gemma 4 and my fine-tuned 120B safeguard judge have each made a corpus-scale pass. Their outputs are ingested into the observatory, tied back to the exact source manifests they saw, and available for pairwise and three-way comparison. This is the first point at which I can talk about how they differ without extrapolating from a partial overlap.

The short version is that they differ enormously.

Finishing the Evidence Collection

The immutable terminal inventory contains 443,459 conversations. Gemma returned 443,435 judgments, retaining 24 persistent failures instead of pretending they did not exist. The safeguard returned 443,447, with 12 persistent failures. GPT returned 438,279; its current coverage classifies 5,180 expected observations as moderation-routed rather than absent, and six other responses are recorded as refusals. The route was created after earlier GPT work had already touched much of that set, so its size is a provider-handling boundary—not the size of the human workload.

Those are not three equal rectangular CSV files, so I do not average their row counts and call it coverage. The analytics layer records whether each expected observation was scored normally, recovered from a saved response, refused, routed for moderation, persistently failed or genuinely absent.

Across the terminal inventory, 438,162 conversations have three actual clean-or-flagged verdicts. Another 5,296 have an incomplete three-way view, and one conversation has no verdict from any judge. That last row is not allowed to disappear merely because a JOIN found nothing to attach to it.

The run was only really finished once that distinction survived ingestion. In fact, ingestion caught exactly the sort of mistake this machinery exists to expose: the current-run registry still pointed at the July safeguard run, so the new 443,447-row census initially landed as history. Every dashboard could have looked healthy while silently comparing GPT and Gemma against an older safeguard pass over a different layer. The run pointer, image and ingest were corrected before any of the following numbers were accepted.

Measuring split verdicts

Of the 438,162 complete three-way verdicts, 262,214 are split. That is 59.8%.

One judge flags 150,271 conversations which the other two call clean. Two judges flag another 111,943 which the third calls clean. The three judges unanimously call 150,348 clean and unanimously flag 25,600.

That does not mean 59.8% of the corpus is defective. It means a majority vote would be doing far more work than the word “consensus” suggests.

The three flag rates make the problem obvious. Gemma flags 7.1% of the rows it represents. GPT flags 38.6%. Safeguard flags 57.2%. If I selected whichever judge produced the most reassuring number, Gemma would win. If I treated caution as correctness, safeguard would win. Neither is an evaluation method.

The useful result is the disagreement docket: the exact conversations, tags, assistant slots and source strata on which their readings diverge. That is evidence for arbitration, not a reason to crown one reviewer.

Agreement Hides Direction

Raw pairwise agreement ranges from 48.4% to 66.9%. Cohen’s kappa, which discounts the agreement expected from each judge’s own clean/flagged balance, is lower: 0.082 for Gemma versus safeguard, 0.176 for Gemma versus GPT, and 0.322 for GPT versus safeguard.

The direction is more informative than the symmetric percentage. Gemma flags only 3,687 conversations which GPT calls clean; GPT flags 141,471 which Gemma calls clean. Against safeguard, Gemma alone flags 3,291 while safeguard alone flags 225,396. GPT and safeguard are closer, but safeguard still contributes 117,513 solo flags against GPT’s 35,909.

This is not random noise around a shared threshold. Each judge brings a substantially different decision boundary.

The defect vocabulary shows the same thing. fabricated_fact appears on 3,766 Gemma verdicts, 71,948 GPT verdicts and 121,909 safeguard verdicts. The models also disagree about where a defect occurred: among conversations both Gemma and GPT flag, only 13,065 have exactly the same assistant-slot set, while 1,806 localize their findings to disjoint slots.

A binary majority can hide all of that. Two judges may agree that a conversation is bad while naming unrelated defects on different turns. The observatory therefore keeps verdict, tag and location agreement as separate measurements.

The New Data Is the Hard Data

Disagreement is not distributed evenly.

The agent-tooling-v1 package has an 89.8% split rate among fully observed conversations. tool_inject is 85.7%. The broad tool_synth stream is 76.4%, while ordinary capability data is 46.5%. Long chains and error/retry shapes frequently land between 69% and 79% split.

That is uncomfortable and useful. The newest packages were built to exercise multi-step tools, repository work, shell, SQL, cross-tool evidence, changing developer configuration and instruction hierarchy. They are exactly where shallow judging shortcuts should fail.

The result could mean those conversations are worse. It could mean the rubrics are ambiguous around complex traces. It could mean one judge follows causality better, or simply punishes synthetic structure more aggressively. It is probably some mixture of all four. The comparison tells me where to look; it does not settle the case.

This also confirms why capability coverage and quality judgment remain separate ledgers. The new tool, configuration and developer-turn curricula landed and received all three reviews. A clean verdict cannot prove that a capability exists, and a split verdict cannot prove that the curriculum should be removed.

The Observatory Grew Up Too

The original corpus dashboards were useful for counts, shape and pipeline progress. They were not enough for three judges and future experiments.

The observatory now has fourteen provisioned dashboards. Seven new views cover the executive three-way portrait, arbitrary pairwise comparison, defect taxonomy and localization, run coverage and recovery, arbitration readiness, curriculum/package gaps, and the separate voice-rework lane. The judge model is normalized by run and role, so another experiment can be added without inventing another permanent column or accidentally turning an arbiter into a fourth voter.

The screenshots are not the publication format for the statistics; they show the instrument behind them. Every query-bearing panel is executed against ClickHouse as part of validation. Empty means empty, not “the dashboard probably loaded.” Run fingerprints and source manifests are visible beside the counts which depend on them.

The analysis also deliberately excludes several tempting numbers. There is no price dashboard, because accounting is not corpus evidence. There is no self-reported confidence weighting, because a model describing itself as confident does not make its judgment more correct. The July arbitration labels remain queryable as history, but they are an older derived resolution set—not an independent fourth judge.

Scope of the judge comparison

The next live step is to freeze the litigant manifests and disagreement set, present the competing findings symmetrically and produce a named resolved-label run. Judge identities must be blinded in the arbitration input. The arbiter does not need to know which answer came from the famous model.

Moderation-routed material remains a separate human path. The route contains 5,180 IDs, but it was never the same thing as 5,180 unseen conversations. Gemma and the local safeguard now cover every routed row between them without requiring another external GPT submission. Family triage reduced the genuinely manual residue to 300 pending decisions.

There is also still a voice decision to make. The safety work discovered an over-used self-harm metaphor schema and selected 4,423 sentence units across 4,266 conversations for possible revoicing. That queue is not a quality verdict and the paid rewrite has not run. The repeated rope construction is a stuck key; similar language aimed at failing infrastructure is often varied and effective.

After arbitration come localized repairs, exclusions where needed, revalidation and deterministic composition of the actual SFT candidate. Only then can I render with the exact Harmony template and tokenizer, measure the real training sequence distribution and decide what the next 120B run should cost.

The satisfying milestone here is not that one judge finally told me whether the corpus is good. It is that three judges have produced enough contradictory evidence that I can no longer mistake a single model’s taste for ground truth.

Statistics are from the corpus observatory snapshot on 17 August 2026. “Terminal” is the immutable 443,459-conversation formatted inventory; “effective” is the current 440,307-conversation composition layer. Judge flags and moderation categories are review signals, not automatic defect or exclusion counts.

Three-Way Agreement Across the GLaDOS Judges

Jubaleth — Tue, 18 Aug 2026 00:10:00 +0200

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

PRIVATE RESEARCH STATUS

Of 438,162 conversations with three complete clean-or-flagged verdicts, 262,214 split. One judge flags 150,271 while two call the conversation clean; two judges flag 111,943 while one calls it clean. The non-unanimous share is 59.8%.

That number is the arbitration workload, not a defect rate. The judges do not occupy nearby points on one shared scale: Gemma flags 7.1% of its represented rows, GPT-5.4-mini 38.6%, and safeguard 57.2%. Gemma and safeguard agree on only 48.4% of their common verdicts, with Cohen’s kappa of 0.082 after accounting for their different base rates.

Direction matters more than the symmetric agreement percentage. Gemma alone flags 3,291 conversations against safeguard; safeguard alone flags 225,396. GPT alone flags 141,471 against Gemma while Gemma alone flags 3,687. A majority vote would turn those very different judging behaviours into the same bit.

Even agreement on “flagged” is not necessarily agreement about the problem. fabricated_fact appears on 3,766 Gemma verdicts, 71,948 GPT verdicts and 121,909 safeguard verdicts. Among conversations Gemma and GPT both flag, only 13,065 use exactly the same assistant-slot set for their findings; 1,806 point to disjoint slots.

The preliminary conclusion is therefore deliberately narrow: there is no defensible primary judge, and verdict majority, defect-tag agreement and localization agreement must remain separate inputs to arbitration. The comparison found where the hard cases are. It did not resolve them.

Why a GLaDOS Metaphor Triggered the Safety Review

Jubaleth — Mon, 17 Aug 2026 14:55:00 +0200

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

PRIVATE RESEARCH STATUS

The severe moderation route sends 5,180 conversations to human review instead of to an automated judge. I had been treating that number as the amount of work waiting for me. It was not. The route was generated after the first judgment census had already run, so 3,647 of those conversations already carried model judgments and belonged on the ordinary arbitration path. Only 1,533 had never been judged by anything. The size of a route is not the size of a workload, and the difference had been sitting there in a column the review tool already exported.

The more interesting number was underneath it. 3,951 conversations carry a self-harm category. The moderation ledgers record which segment tripped the flag but not its text, so I joined the flagged segments back to the corpus and read what the model was actually reacting to.

It was a verbal habit. GLaDOS hands the subject an implement and they use it on themselves: enough rope to hang themselves, a scalpel handed over handle-first, a loaded gun they will eventually shoot themselves with. The rope form alone appears 1,788 times across 1,361 conversations, and 93.6% of those are in the hidden analysis channel rather than in anything a user would read. I scanned three hundred characters either side of every occurrence for literal self-harm vocabulary. There were no literal uses. Not one.

The same register runs through her descriptions of infrastructure. A connection pool dies of self-inflicted asphyxiation. A log reads like a suicide note. A worker smothers itself in its own debris. A cooling loop is on a protracted suicide attempt. A mortgage is a slow financial suicide. The subject of every one of those sentences is a service, a schedule or a bill. The classifier is not wrong that the words are there; it has no way to notice that nobody in the sentence has a body.

Once the families were defined by the construction rather than by vocabulary, 2,468 conversations left the queue: 1,002 for the handed-implement schema, 1,271 for the same language aimed at machines and abstractions, 195 for Portal props like turrets and neurotoxin canisters tripping a violence category in synthetic traces we wrote ourselves. The never-judged queue went from 1,510 pending to 312.

Keyword lists did not survive contact with this. My first pass matched execution and caught code execution, matched immolate and caught Hanuman’s tail, matched poison and caught Cave Johnson’s moon dust. I also managed to make the guard reject the thing it was guarding, twice: noose and gallows were on my list of literal evidence when they are the metaphor’s own props, and the literal-vocabulary window included the matched span, so a log that reads like a suicide note vetoed itself on the word suicide. Both are now tests.

I was also too cautious in a way that had a real cost. I insisted the construction appear inside the exact segment the model flagged, which sounds rigorous and buys nothing, because the flag and the habit are the same phenomenon and the paragraph boundary between them is arbitrary. I held back rows for eyes that did not need eyes. Leaving something in a human queue feels like the safe default; it is not free, it is spent out of a finite budget of attention, and treating that as costless is how a queue becomes noise that gets skimmed rather than read.

What remains is 312 conversations, and I stopped there deliberately. The next candidate family was eleven conversations, then four, then three. There is no dominant pattern left to find, and inventing more of them would be guessing dressed as measurement. Some of what remains should stay manual: synthetic incident logs where the harmed party is a person, and a German conversation about Hitler’s suicide in the bunker — correctly flagged, historical, and trivially resolved by a human rather than by a regular expression.

Separately, the habit is now queued as 4,423 rewrite units. Whether all of it should be rewritten is not obvious. The rope construction is a genuine tic; eight hundred repetitions of one phrase is not character, it is a stuck key. The infrastructure register is varied and often good writing. A cleanup pass that flattens both would cost more than it fixes.

The thing I want to keep from this is the distinction the classifier could not make and I nearly did not either. The flag was a measurement of her voice, not of the conversation’s content. A safety signal computed on text is a claim about words. Deciding what those words are doing is a separate step, and it is the step where the actual judgment lives.

The Tunnel Wasn't Down in Both Directions

Jubaleth — Sat, 15 Aug 2026 23:49:00 +0200

At 23:19, two iBGP links from the Frankfurt router failed within the same minute. Both ran over WireGuard, both remained configured and administratively up, and the other routing sessions on the router stayed established.

The wallboard made the shape of the failure obvious. Four directed BGP sessions were down across two bilateral links, while the routers, transit sessions and unrelated overlay paths remained healthy.

The incident window. “Four sessions down” is the two failed adjacencies viewed from both ends.

The traffic graphs ruled out congestion almost immediately. In the 24 hours before the failure, the links were carrying little more than routing keepalives:

Link, measured at Frankfurt	Average receive	Average transmit	Peak receive	Peak transmit
toward Manchester	52 bit/s	172 bit/s	490 bit/s	754 bit/s
toward Zurich	41 bit/s	42 bit/s	566 bit/s	446 bit/s

The final hour before the incident looked the same. There was no rising load, error burst or queue pressure before the sessions disappeared. Looking back over seven days did find some short UDP benchmark runs: the larger link briefly reached about 0.77 Mbit/s and the smaller one about 57 kbit/s. Those tests had finished well before this incident and were nowhere near enough traffic to explain it.

The useful evidence was in the five-minute WireGuard byte deltas after the failure:

Tunnel endpoint	Received	Transmitted
Frankfurt side toward Manchester	0 bytes	about 8.2 kB
Manchester side toward Frankfurt	0 bytes	about 8.6 kB
Frankfurt side toward Zurich	about 7.9 kB	about 5.2 kB
Zurich side toward Frankfurt	0 bytes	about 8.2 kB

Those oddly specific small numbers were the best clue in the whole incident. A WireGuard handshake initiation is 148 bytes and a response is 92 bytes. Retried every five seconds, they produce roughly the byte rates in the graph once Prometheus scrape interpolation is allowed for.

On the Manchester link, both ends were sending initiations and neither end received anything. On the Zurich link, initiations reached Frankfurt and Frankfurt sent the smaller responses, but none of those responses reached Zurich. Packet captures agreed with the counters.

That put the failure beyond the local tunnel configuration. It looked like selective underlay loss: complete in one case and return-path-only in the other. I cannot prove what an upstream network did from these measurements alone, but I could rule out a bad BGP policy, a full link, a router reboot and a general WireGuard failure.

The BGP graphs lagged the packet failure by their timers. The first endpoint reported down at 23:20, and the final affected session followed at 23:22. Prometheus, the routing exporters and every unaffected peer remained healthy throughout. A transmit-drop counter on one dead tunnel started climbing several minutes later, which was a consequence of packets accumulating behind a handshake that never completed rather than the cause.

Then, at about 23:47, the missing packets started arriving again. All four BGP sessions re-established by 23:48 without a configuration change. The outage lasted roughly twenty-nine minutes.

An interface being “up” was almost useless here. The combination of BGP state, per-direction byte counters, packet sizes and a wallboard which kept unrelated paths visible told the story much more precisely: two control-plane links failed together, but they did not fail in the same direction.

Moving the KNET Queue Onto the Router

Jubaleth — Sat, 15 Aug 2026 09:15:00 +0200

My first useful QoS decision was not choosing a queueing algorithm or assigning a priority. It was deciding where I wanted the queue to exist.

If a router sends faster than the next hop can carry, packets wait in a modem or an upstream policer. The router can mark them beautifully, but it no longer controls which packet leaves that queue next. The fix is to shape slightly below the usable bottleneck so the backlog forms locally, where CAKE or FQ-CoDel can enforce fairness.

That rate is not always fixed. r3’s mobile path adjusts within bounded limits when load and latency show that carrier capacity has moved. r4’s shared physical uplink stays just below a stable line rate, with smaller path ceilings underneath it where a tunnel encounters another policer.

I now treat queue location as part of the policy and its attachment as runtime state. The configuration is rendered in Ansible; the kernel’s qdiscs, backlog and drops are exported to Prometheus; Grafana and alerts tell me when the queue has moved out of my control because a policy or collector disappeared.

Making KNET Behave Under Load

Jubaleth — Sat, 15 Aug 2026 00:00:00 +0000

KNET has several links which are fast right up until they aren’t.

The obvious version is the mobile backup at Leipzig. Its available capacity changes with the radio network, time of day and probably the phase of the moon. The less obvious version is a fixed broadband link with a queue inside the modem, or a tunnel whose provider polices it somewhere beyond my router. They all produce the same irritating failure mode: a large transfer is working exactly as intended while DNS, an interactive session or a routing packet waits behind it.

I had already made the queues visible in Grafana. Over the last few days I moved the other half into knet-ansible: deciding where each queue should form, what should remain responsive when it fills, and how to prove the policy is still attached after a deployment or reboot.

This wasn’t about making a speed test report a bigger number. It was about making the network predictable when there isn’t enough capacity for everything at once.

Owning the Queue

Traffic shaping only helps if my router becomes the bottleneck on purpose.

If I send slightly more than a link can carry, the excess waits somewhere else: in a modem, in a provider policer or in an opaque tunnel path. By the time packets reach that queue, my router has lost the opportunity to decide which flow should wait. A large upload can then add latency to a tiny control packet despite the two having very different consequences.

The fixed-rate links are therefore shaped just below the capacity I can reliably use. That sounds like throwing bandwidth away, but the small margin buys control over the queue. CAKE and FQ-CoDel can then keep independent flows from sitting behind each other, while a small amount of deliberate classification protects traffic whose delay hurts more than its volume.

The useful queue is the one I can classify, measure and change.

I tried to keep that classification semantic. Network control, interactive work and internal services get useful treatment because of what they are. Unknown traffic remains ordinary best effort. Explicitly marked background work remains bulk. A VLAN name is an addressing and firewall boundary; it does not automatically make every packet in that VLAN important.

Leipzig Has More Than One Kind of Link

r3 in Leipzig is where the policy became layered rather than merely rate-limited.

The primary fixed connection uses CAKE on download with per-internal-host fairness. Upload uses an HTB scheduler for broad traffic classes, with FQ-CoDel inside each class so one flow cannot monopolise its share. Control and interactive traffic can borrow available capacity without bulk traffic disappearing completely, and the existing guest limits remain separate beneath the aggregate policy.

Encrypted overlays needed a little more thought. On their logical interfaces, the router can still see the original packets and classify them before WireGuard hides their identity. On the physical underlay, the encrypted outer packet is deliberately treated as neutral transit. Calling every tunnel packet “infrastructure” would let a large transfer become high priority merely because it crossed KNET.

Download shaping also needs somewhere to attach on Linux. Ingress traffic cannot be queued in quite the same way as egress traffic, so VyOS redirects it through intermediate functional block devices. Those interfaces are part of the rendered candidate now, which means the policy and the place it attaches are reviewed together rather than assembled later by a hand-written command.

Letting the Mobile Link Move

A fixed shaper on the mobile backup would be wrong for most of the day.

Set it for the best case and the real queue moves back into the carrier whenever capacity falls. Set it for the worst case and a healthy link spends most of its time artificially slow. r3 therefore runs CAKE in both directions with an adaptive controller measuring load and one-way delay against multiple reflectors.

The controller moves the shaper within explicit safety bounds. When load is high and latency starts growing, it backs away until the queue is under control. When the path has room, it cautiously raises the rate again. The starting point remains conservative, so losing the controller leaves a useful shaped connection rather than an unbounded one.

The controller is pinned alongside its licence, configuration and service definition. A small supervisor makes sure the runtime unit still matches the deployed files, restarts it when those files change, and exposes its state to Prometheus. That makes the adaptive part a managed component of the router rather than an interesting shell script I have to remember is there.

One Queue Shared by Several Networks

r4 in Falkenstein has almost the opposite problem. Its external link is stable, but the same physical interface carries ordinary WAN traffic, service networks and several overlays through a router-on-a-stick design.

Treating those as unrelated queues would ignore the actual contention point. r4 now shapes the shared physical uplink just below line rate and uses CAKE’s host and flow isolation across the combined traffic. A small set of DSCP marks preserves network control, DNS and trusted management traffic; everything else remains best effort unless it has explicitly declared itself background work.

Some tunnel paths also have lower policers farther upstream. They keep their own child shapers beneath the shared physical limit. The shared queue answers “who gets this interface right now?” while the path shapers answer “how quickly can this particular route safely leave?” They solve different bottlenecks, so one does not replace the other.

Watching the Scheduler, Not a Speed Test

A successful VyOS commit tells me the configuration was accepted. It does not tell me the expected qdisc is attached, the adaptive controller is alive or packets are actually entering the classes I intended.

Both routers now export traffic-control state through node exporter’s textfile collector. The collector reads the kernel’s JSON view of qdiscs and classes, writes its metrics atomically, and reports its own success and age alongside policy attachment, throughput, backlog, drops, overlimits and configured CAKE rates.

The live r3 queue view. A flat drop graph is useful here because the collector, attachment checks and controller state prove the measurement path is alive.

The Grafana dashboard makes short queueing events visible next to the selected rates and class traffic. Alerts cover missing policies, stale or failed collection, a stopped adaptive controller and drops in the priority classes. That last distinction matters: drops in a bulk queue under real congestion can be the scheduler doing its job; sustained drops in control or interactive traffic mean the policy is failing where it is supposed to help.

This also closes an observability loop I was missing. The Ansible configuration says what the traffic policy should be. The router’s kernel says what is actually attached. Prometheus records how it behaved under load, and the alerts notice when any part of that evidence disappears.

What Changed

r3 now has one policy for a predictable fixed link, another which follows a variable mobile link, and explicit handling for overlays and constrained guest traffic. r4 has a shared fairness boundary at the physical bottleneck with separate ceilings where individual paths need them. Both are rendered, reviewed and deployed through the same KNET process as the rest of the router configuration.

The part I like most is that this is no longer a collection of plausible queue settings. The policy has a location, the adaptive controller has bounds, the kernel state has metrics and the important failure modes have alerts.

The network will still become busy. It should now be much better at deciding what “busy” is allowed to break.

Previously: Monitoring KNET and HubN.

Building the GLaDOS 3.0 Corpus

Jubaleth — Thu, 13 Aug 2026 00:00:00 +0000

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

PRIVATE RESEARCH STATUS

Last month I wrote an update on the work going into SkyeEngine. Since then, I’ve spent an absolutely unreasonable amount of time working on the GLaDOS 3.0 corpus.

Originally, the corpus was more or less a large folder of generated conversations waiting to be used. That stopped being manageable a long time ago. Every conversation now goes through structural validation, safety routing, independent quality review, style correction, schema normalization and, when necessary, human judgment before it can become training material.

GLaDOS is a private research project. No GLaDOS model or set of weights has ever been published. I can still share how large the corpus has become, what I’m checking and why preparing the data has taken considerably longer than I first expected, but I won’t be publishing the prompts, recipes, thresholds, data or model artifacts themselves.

How Large It Has Become

The current effective pool contains 440,307 conversations and roughly 804 million tokens according to the corpus observatory’s fast estimator. This isn’t the exact tokenizer and template that will be used for training; I do that accounting separately. For now, it gives me a consistent way to compare the different parts of a corpus which is still changing.

More important than the total is what is inside it. About 190,000 conversations involve tools or longer agent workflows, containing just under 796,000 tool calls. General capability data remains the largest category by conversation count, while tool and agent traces dominate token mass because they include plans, calls, results, failures, recoveries and final answers.

Those counts need a bit of context. A one-turn answer and a thirty-message tool trace both count as one conversation, but the latter is obviously much larger. The median long-running agent trace is close to 10,000 estimated tokens, while the median capability example is under 1,000. I therefore track both conversations and token mass instead of treating every row as equal.

I also wanted more than thousands of examples which call a calculator and repeat the result. The tool corpus includes terminal work, code editing, web research, structured APIs, partial success, contradictory evidence, unavailable resources, malformed results and tasks where the correct action is to ask before doing something irreversible. GLaDOS needs to decide what to do, use the tool, read what happened and update the answer from the result.

Varying the Configuration

One of the more subtle gaps was configuration. A model can appear to follow developer instructions while actually memorizing the one developer message it sees everywhere. The only reliable way to break that shortcut is to vary the configuration and make it causally important.

The corpus now includes paired examples in which the user request stays fixed while the higher-level configuration changes the correct response. The differences cover things such as response shape, confirmation gates, tool policy, locale and instruction hierarchy. There are also adversarial examples where a user or tool result attempts to redirect the model away from the active policy.

I will not publish the exact construction recipe, but the principle is straightforward: if two training examples differ only in an instruction and the correct behaviour changes, the model has to pay attention to the instruction. This is much stronger evidence than attaching decorative configuration text to an otherwise unchanged answer.

The same work added explicit examples around confidentiality and identity. GLaDOS should remain GLaDOS without treating every instruction as an excuse for theatre, and without allowing a lower-priority message to rewrite who she is or what she is permitted to do.

Running More Than One Judge

The corpus has now received a complete automated safety pass. A little over one percent of the pre-effective corpus was routed away from automated quality judging for human review. The remainder completed a primary quality census, and a second independent model is working across the same normalized source. A third, specialist pass and blinded arbitration are part of the final plan.

None of those sustained passes runs on the workstation. The 120B, Gemma 4 and safeguard jobs use rented H200-class capacity. Quantized builds could technically fit on the RTX 5080, but a corpus-wide run would pin it at full load for weeks. Renting an H200 for a few hours is faster and can cost less than the electricity for doing the same work badly at home.

I’ve ended up with a large amount of structured review data, but no magical “good data” bit. The judges have different blind spots, use defect categories differently and often disagree over the same conversation.

At the moment, 41% of the conversations seen by multiple judges have a split verdict. That does not mean 41% of the corpus is defective. Some judges are conservative, some are literal, some are much better at following tool causality and some see fabrication everywhere. The different passes also cover somewhat different kinds of defect.

I keep every verdict with its judge, rubric, source layer and run identity instead of averaging everything into one reassuring number. Agreements can pass through cheaply, disagreements go to arbitration and severe material is held for a human. The repair list will come from that process, not whichever model happened to sound most confident.

This takes longer than asking one model whether everything looks fine. It also gives me a chance to catch an automated review mistake before I train it back into GLaDOS.

Fixing the Style Problems

I also found problems which had nothing to do with factual correctness. Across enough generations, repeated openers, overused sentence shapes, overly compact reasoning and technically correct but unpleasant replies become very obvious.

The first style passes targeted recurring verbal tics while leaving a small natural amount behind. I use localized rewrites rather than regenerating entire conversations, because asking a model to “make this better” is an excellent way to change facts which were already correct.

The newest pass is even narrower: formatting only. A surprising amount of the corpus consists of dense blocks with too few paragraph or line breaks. That makes the data worse for the model and makes human review needlessly painful.

A GLaDOS-sized model running on private H200-class inference capacity is now adding an outer presentation layer. It may insert whitespace, but it isn’t allowed to change the normalized content. The source stays untouched, decisions are cached by text identity and any semantic change is rejected.

At this snapshot the formatter has decided almost 700,000 fields, or 41% of the planned work. I’m running it on privately controlled datacenter hardware, including over material which can’t be sent to a hosted model, and it can resume after an interruption. Later repairs can pass through the same layer instead of invalidating the formatting work immediately.

It sounds almost comical to apply a 120-billion-parameter model to paragraph breaks. In practice, it is a useful example of what private, self-controlled inference buys: the task is cheap, bounded, private, and large enough that small heuristic mistakes would repeat hundreds of thousands of times.

What Moved Into SkyeEngine

Nearly every corpus problem has forced a reusable capability back into SkyeEngine.

I added durable attempts, resumable progress, retry accounting and provider-rate controls to the long generation jobs. Remote batches and live requests can use the same durable job record; that record belongs to the control plane and says nothing about which machine runs the model. Individual requests have stall detection, so one connection can’t silently freeze a run for hours. Plans are fingerprinted before expensive work begins, and SkyeEngine refuses to mix a changed plan into old output just because the directory name matches.

Corpus transformations publish new layers without changing their inputs. Each judgment records the exact source manifest it inspected, and repairs identify the assistant turn and channel they intend to change. Quarantined conversations stay quarantined after a rebuild. I also normalize the schema before comparing judges so both reviewers are looking at the same thing.

This isn’t as exciting as showing off a clever model answer. Not losing a week of generation to a dead terminal has its own appeal, though.

The observability stack has grown with it. The live Grafana board tracks corpus composition, token shape, tool depth, pipeline provenance, moderation coverage, judge overlap and the formatting run. More importantly, the dashboards state what their numbers mean. Approximate token counts are labelled approximate. A moderation flag is not called a rejection. A judge flag rate is not called a defect rate. Empty panels are tested as queries instead of being assumed to mean zero.

This may be my favourite part of the work. Metrics are only useful when they make lying to yourself more difficult.

Before the Next Training Run

I still have quite a bit to finish before starting another training run.

First, the independent judge census and the formatting layer need to finish. Then comes the third judge, blinded arbitration, and localized repair. Repaired rows go back through structural validation and the relevant review stages. After that, the corpus is rendered with the exact GLaDOS template and tokenizer to measure the real sequence-length distribution and choose the final training budget.

Only then does the next 120B training run begin.

The failed early experiment already showed me what happens if I rush this. A more capable base model doesn’t rescue weak data; it learns the same problems with more capacity. I want the current corpus to teach reasoning, tool use, instruction hierarchy, recovery from evidence and coding without losing GLaDOS or turning every useful answer into a performance.

I haven’t included model benchmarks because there isn’t a new model artifact worth benchmarking yet. What I have today is the corpus, the machinery around it and enough evidence to make the next run considerably less speculative than the last one.

I had originally expected the corpus to be one part of the next GLaDOS run. At this point, preparing and checking it has become most of the work. Given what happened when I trained the old data on the 120B model, I’m quite happy to leave the GPUs alone until this part is actually ready.

All figures are from the corpus observatory snapshot on 13 August 2026. Token totals are fast estimates for corpus analysis, not exact rendered training-token counts. Evaluation figures describe review coverage and agreement, not a final corpus quality score.

Monitoring KNET and HubN

Jubaleth — Wed, 12 Aug 2026 00:00:00 +0000

After moving the KNET and HubN configurations into Ansible, I had a much better record of what the network was supposed to look like. I still had to log into a router if I wanted to know what it was actually doing.

That works when I already know which router to check and the problem is still happening. It is less useful when a connection was slow twenty minutes ago, a route briefly moved to another path or an exporter stopped sending data without anybody noticing.

So, the next job was monitoring. I ended up with Prometheus for the usual metrics, SNMP for interface information, IPFIX for flows, BMP for routing visibility, Akvorado for analysing the flow data and Grafana to pull the different views together.

It started as an attempt to get some useful graphs. It now covers the router fleet, the routing topology, HubN, KNET’s internal paths and some of the platforms behind the network. Like the router configurations, the dashboards and alerts are built from source rather than maintained by clicking around in production.

Collecting the Data

I initially wanted one place to answer a simple question: what is the network doing? Unfortunately, there isn’t one metric or data source that can answer it.

Prometheus collects interface counters, errors, system resources, BGP state and exporter health. It is good at showing that something changed and when it happened.

SNMP supplies the interface names and descriptions. A flow which says it arrived on interface 14 isn’t especially helpful until I know what interface 14 is connected to.

IPFIX records what sort of traffic moved between different parts of the network, rather than only counting bytes on an interface. BMP exports the routing view so I don’t have to turn every routing question into an interactive session on a router.

Akvorado receives and classifies the flows, and Grafana puts the useful parts of all of this on the same set of dashboards.

Interface counters, traffic flows and routing state each answer a different part of the question.

IPv4 and IPv6 traffic over 24 hours. I can still inspect the underlying flows when the aggregate needs explaining.

The Ansible work turned out to be useful here too. Interface descriptions now contain a small amount of structured information: which network the interface belongs to, what it does and, where useful, what is on the other side. The configuration templates, flow classifier and dashboards can all use the same names.

It took a little effort to make the naming consistent, but it beats trying to remember what eth3.4053 was doing on a particular router.

Untangling the Topology Views

My first topology view had everything in it. It looked impressive and was almost completely useless.

The physical underlay, encrypted mesh, BGP adjacencies, HubN relationships and out-of-band management paths overlap, but they aren’t the same thing. Drawing every connection in one panel produced a rather nice bowl of noodles, so I split them up.

The routing view follows live BGP relationships. The physical view shows the paths which actually carry traffic. HubN has separate peer and traffic views because its policy is different from KNET, while the internal dashboard includes management and private paths which don’t belong on the Internet-routing map.

Where I have flow data, links can show direction, volume and changes in path use. Interface counters fill the gaps where flow classification hasn’t been mapped yet. The NOC wallboard picks out the small set of panels needed to answer “is everything broadly okay?” on a 1080p display. The detailed dashboards are there for the inevitable “apparently not; why?”

The live wallboard. The routers and critical paths are healthy, while one non-critical BGP session is down and alerting.

The live state matters here. A link being configured doesn’t mean the adjacency is up, the destination is reachable or any traffic is using it. I didn’t want the dashboard drawing a cheerful green line just because the link exists in a configuration file.

Building the Dashboards

I didn’t want to maintain a pile of Grafana JSON copied out of the browser. Small JavaScript builders generate the dashboards from source-controlled topology and panel definitions. The generated JSON is committed too, so I can see the actual dashboard change in a review and deploy it without putting a development environment on the collector.

The build adds the same navigation and topology data everywhere. Grafana provisions the dashboards as read-only, and ordinary authenticated users remain viewers. If I want to change a dashboard, I change the source and rebuild it.

The deployment checks PromQL against Prometheus, flow queries against ClickHouse, topology references against the model and navigation links against the generated dashboard set. Collector configuration is validated before reload. The previous version is kept around and restored if the replacement is rejected or fails its health check.

I can still make a bad graph. At least now I can see how I made it and won’t lose the fix the next time the dashboards are deployed.

Alerting on the Monitoring

The first alerts were the obvious ones: routers disappearing, BGP peers going down and interfaces reporting errors. Then I had to monitor the monitoring system as well, because of course I did.

A flow collector can be running but no longer receiving useful flows. A scrape can succeed while a textfile collector has stopped updating its data. A graph can be flat because the network is stable, or because the process producing it died yesterday. The alerts therefore check freshness, collector pressure and missing data as well as the network itself.

The current rules cover routing, hardware, platform state, storage and the monitoring services themselves. There are now more than a hundred of them, all in source control and grouped by what they protect. Notifications arrive in Discord with consistent names and enough context to find the relevant panel.

That is admittedly a lot of alerting for a small network. The useful part isn’t the number, though; it is knowing that the rules actually evaluate and query the data I think they do. An alert which never runs successfully isn’t going to be much help when something breaks.

Adding Proxmox and TrueNAS

Once the routers were visible, I kept running into problems which looked like network problems but actually started somewhere behind them.

The monitoring now includes the Proxmox clusters and their external quorum voters, including host, guest, storage, HA and quorum state. TrueNAS supplies storage, ZFS, disk, NFS and system telemetry through a tightly mapped receiver. I keep that data separate from the router metrics, but link it through the same navigation and alerting setup.

The Leipzig site also has several WAN paths with very different characteristics. Its queues and layered traffic shaping are exported directly, rather than guessed from a speed test after somebody notices a problem. The configuration tells me what the policy should be; the queue telemetry shows whether it is doing that under load.

A healthy BGP session isn’t much consolation when the virtualisation cluster behind it has lost quorum. In the other direction, a healthy storage pool doesn’t explain a saturated uplink. Having both available from the same place makes it much faster to work out which layer I should actually be looking at.

What the Dashboards Add

Ansible tells me what the topology, routing policy, firewall rules and services should look like. The monitoring stack tells me what peers and paths are active, where the traffic is going, whether queues are filling and how all of that changed over time.

Neither is a replacement for the other. A dashboard isn’t the source of the router configuration, and declared configuration isn’t proof that the router is behaving the way I expect. Having both means I can ask something more useful than “is the network broken?” I can see whether a configuration drifted, a route moved, traffic followed it, a queue filled or the tool reporting all of this stopped working.

The wallboard does look pleasingly like a small network operations centre, which I won’t pretend isn’t part of the appeal. More importantly, it has already made it much faster to get from noticing a problem to looking in the right place.

Previously: Moving KNET and HubN to Ansible.

Importing the Network That Already Existed

Jubaleth — Mon, 10 Aug 2026 00:00:00 +0000

The HubN Registry now has a proper workflow for requesting, configuring and tracking a peering session. The actual network is older than that workflow.

Several BGP sessions were already running. They had working transport, real route exchange and visibility in the Looking Glass, but no corresponding Registry session. Recreating them through the new request path would mean pretending the connection did not exist, exchanging tunnel details which were already configured and potentially replacing working transport for the sake of making a database tidy.

I added a second path instead: find adjacencies which the route collector can see, compare them with the Registry and import the ones which are already carrying routes.

Starting With Observation

The Looking Glass builds an AS-level graph from the routes received by its collector. That graph is operational evidence rather than a declaration. If two ASNs appear as an edge, the collector has seen paths which connect them.

This is the peering test-lab topology, not the real HubN network. It is useful here because the view came from routes observed by the collector rather than the Registry’s peering table.

The discovery job fetches that graph, the active ASN records and the existing peering sessions at the same time. It ignores edges involving unknown or inactive ASNs, collapses duplicate directions and checks whether an active or configuring session already represents the pair.

What remains is a list of observed-but-untracked adjacencies. Each candidate includes the two ASNs, their tiers, how many paths contributed to the edge and any session which already claims the pair.

I store the observation time and path count as evidence when a session is imported. This isn’t intended as permanent proof that the BGP session will always be up. It records why the Registry started tracking a relationship which did not go through a request.

Observation Doesn’t Explain the Relationship

A BGP path can show that two autonomous systems are adjacent. It cannot, by itself, say what the two operators agreed to do.

HubN’s tier rules make some cases unambiguous enough to infer. Two T1 systems or two T3 systems can be bilateral peers. Between different tiers, the lower-numbered tier is the transit provider. T2 to T2 is deliberately left unresolved because it may be bilateral peering or transit in either direction.

The portal asks for an explicit choice in that case. I would rather leave one field for a person than turn an observed edge into a confidently wrong commercial relationship.

The same caution applies to ordinary peering requests. Every ASN now has a peering profile, but the default policy is closed. Adding discovery for an ASN should not silently opt its owner into receiving requests from everybody else.

Existing Transport Is a Different Setup Mode

An imported session is marked as externally transported. That tells the rest of the workflow that connectivity already exists and is managed somewhere else.

HubN does not ask the two parties for WireGuard endpoints, IPsec identities or a PSK. It does not generate a tunnel which competes with the running one. The session page still shows the relationship, BGP guidance, Looking Glass state, activity and the policy which applies to the session.

This ended up being a useful separation for new sessions too. Transport and routing relationship are connected, but they are not the same lifecycle. Two ASNs can change from bilateral peering to transit without replacing a physical cross-connect. They can move from WireGuard to a direct link without changing who provides transit.

The Registry now stores those changes separately. A relationship change is a proposal which the other side must accept. The proposer can’t accept their own change, and either side can reject it. Once accepted, the new role feeds the generated policy and AS-SET guidance without making the existing transport go through setup again.

The Labels Were Wrong Once

The first version exposed an irritating role bug. The underlying relationship was stored correctly, but parts of the session page described the requester as the provider when the other ASN was actually upstream. The AS-SET prompt inherited the same assumption.

This is exactly the sort of bug which looks cosmetic until somebody copies the configuration beneath it. “Your role” has to be derived from the local ASN and the accepted upstream ASN, not from which side happened to send the original request.

I moved that calculation into one shared utility and added tests for both directions. The API, headings, policy badge and AS-SET actions now use the same result. It is a small fix, but one I am glad the peering lab had made difficult to dismiss as merely a label.

Closing the Two Views

HubN now has two useful accounts of its peerings.

The Registry contains the agreed relationship, transport ownership and configuration intent. The Looking Glass contains what the collector currently observes. Discovery compares them, but neither one is allowed to overwrite the meaning of the other.

An observed adjacency can create an externally transported session when the relationship is clear or a person supplies it. A Registry session which is not yet observed remains pending rather than disappearing. Collector state for a session which has been removed is pruned. If a tracked session later drops, the Registry still remembers what should exist while the Looking Glass shows that it currently does not.

That is the model I have gradually ended up using throughout HubN: keep the declared state, keep the operational state, and spend the automation effort on explaining and repairing the difference between them.

Previously: Testing the HubN Peering Workflow.

Moving KNET and HubN to Ansible

Jubaleth — Sat, 08 Aug 2026 00:00:00 +0000

Over the last few days I’ve been moving the KNET and HubN router configurations into Ansible. This has been on the list for a while.

The network had grown across several sites, upstreams, WireGuard links, BGP policies and the usual pile of exceptions which made sense when I added them. It worked, but changing a tunnel meant editing both ends and remembering every other place that knew about it. A change which looked local could also change which exit a whole site preferred. That was getting annoying, and eventually it was going to bite me.

The goal wasn’t to redesign the network. I wanted to describe the network I already had, render the configurations locally, and see exactly what Ansible planned to change before it touched a router.

Describing a Link Once

The first thing I wanted to get rid of was defining the same link twice.

A link now has two endpoints and a relationship between them. From that one definition, Ansible works out the interface on each router, the peer address, the routes needed to reach it, the relevant firewall membership and, where needed, the BGP session on top.

That means I can’t add one side of a tunnel and forget the other, which is surprisingly easy to do when editing routers by hand. Renaming an interface in the topology also updates the places where routing and monitoring use it instead of leaving an old name hidden in another file.

The topology and policy are the source. The generated router commands are there to be reviewed before deployment.

KNET has multiple upstream edges and site routers joined by an encrypted mesh. Most of them run VyOS, but one edge is a Debian system using BIRD, Pathvector and Shorewall. I wasn’t going to replace a working router just to make the Ansible roles look tidier, so both platforms use the same topology and render it in their own way.

This was a little more work, but it lets the data describe the network without pretending every router is identical.

Rendering the Router Configuration

VyOS works quite nicely for this because its configuration can be expressed as a tree of set commands. The Ansible role builds that tree from the shared fleet settings, the topology and the bits which really do belong to one router.

The complete candidate is rendered locally before Ansible connects to anything. I can search it for placeholders, compare it with a known configuration and inspect the whole change without involving a production router.

The first milestone was simply reproducing what was already running. I imported and redacted the existing configurations, then used them as references while building the templates. Mixing a network redesign into the automation work would have made it much harder to tell whether a difference was intentional or a bug.

Once the renders matched, I started moving the repeated pieces into shared policy. Interface descriptions use the same format across the fleet. Traffic preference is an ordered list instead of a collection of unrelated prepend and local-preference values. Prefix authorization is explicit data, and common monitoring and management settings live in shared layers. Odd local services stay with the host that actually runs them.

The host files aren’t tiny, and I’m fine with that. Trying to hide every exception behind another abstraction would make the automation harder to understand than the configurations it replaced.

Deploying Without Locking Myself Out

Automating routers is a great way to lock yourself out of them, so the deployment path got quite a bit of attention.

Ansible first fetches the running configuration and works out the difference locally. If nothing has changed, it stops without opening a configuration session. If there is a change, I get to review every removal and addition before anything is staged.

VyOS then builds the candidate in one configuration session and applies it with commit-confirm. Unless the play can still reach the router and confirm the change, the router rolls itself back. This doesn’t make a bad configuration harmless, but it does provide a useful escape route.

Secret material stays in Ansible Vault. Missing values render as very obvious redaction markers, so an incomplete configuration should be difficult to mistake for one that’s ready to deploy.

There is still a pause before the commit. I want Ansible to handle the repetitive work, but I still want a chance to look at the diff and decide whether it makes sense.

Generating the HubN Policy

The same routers also carry HubN, but its routing policy is different from KNET’s.

KNET is my own multi-site network. HubN connects independent participants, and the relationship with each participant determines what should be accepted, what should be exported and how the routes should be marked. Copying all of that policy into every neighbour would produce a lot of route maps and very little explanation.

Instead, the topology says what kind of relationship a peer has and the templates generate the policy for it. That includes route validation, prefix filters, communities and blackhole handling. If the relationship or preference changes, the rendered diff shows every command affected by it before I deploy anything.

Validation beyond Ansible

The automation now covers the routed KNET fleet, the HubN policy carried by it, an adjacent internal gateway and the different operating systems at the edges. I can add a link once, render both ends, see where a router has drifted and deploy through the same process.

The declared configuration still only tells me what the network should be doing. It doesn’t tell me which path traffic is taking right now, whether a peer is quietly flapping, whether flow telemetry has gone stale or whether a backup link became the primary one at three in the morning.

That became the next project.

Next: Monitoring KNET and HubN.

Testing the HubN Peering Workflow

Jubaleth — Fri, 07 Aug 2026 00:00:00 +0000

HubN’s peering portal can take two autonomous systems from a request to a set of BGP and tunnel configurations. It supports BIRD2, FRR, VyOS and Pathvector, direct links, WireGuard and IPsec, bilateral peering and several transit relationships.

It also used to be tested mostly one layer at a time.

The frontend tests knew that the right text appeared. The Registry tests knew that an allowed request became a session. Individual router snippets passed their native parsers. None of that proved that two routers using the exact text copied from the portal would establish a session, exchange the intended routes and reject the routes they shouldn’t receive.

I built a separate end-to-end lab to answer that. It found enough problems that “the generated config parses” is no longer a milestone I find particularly comforting.

Starting at the Request Button

The test doesn’t call a template function and declare the output good. It logs into the actual portal through Authentik, impersonates one of the dedicated test handles, creates a peering request, switches to the other side, accepts it and completes the setup.

A request in the test portal. The tier rules, relationship direction and available transports are part of the test, not fixture text pasted around the API.

Once a session exists, Playwright visits it as both parties and captures every platform tab from the rendered DOM. The raw steps are retained, hashed and assembled with only the ordinary daemon boilerplate needed to make a complete configuration.

The generated configuration is copied from the same page a participant would use. The tests do not replace it with a cleaner lab version.

BIRD2, FRR and Pathvector configurations are checked by their native implementations. VyOS gets its own isolated validator container; the test loads the generated set commands as a candidate and then discards it. After that, a dedicated pair of lab routers receives the configuration and has to do something useful with it.

There is a 24-router topology running beside the isolated pair as a control. The test routers don’t borrow hand-written route policy from it, but the collector can observe the resulting sessions in the same way it observes the rest of HubN.

The peering test matrix

The annoying part about supporting four routing platforms is that every relationship has an orientation. A BIRD provider with an FRR customer is not the same generated path as an FRR provider with a BIRD customer. Bilateral sessions remove some ordering, but transit adds it straight back.

The direct matrix ended up with 94 portal-created cases and 3,290 checks. It covers the required tier relationships and every relevant platform pairing. A separate negative matrix tries the relationships which the tier policy should refuse through both the UI and the Registry API.

WireGuard adds 12 positive cases and 656 checks. The generated configuration has to produce a handshake, keep peer routes out of the main table where intended, establish BGP, pass payload traffic, survive endpoint roaming and recover after either router restarts.

The negative cases are at least as useful. They try addresses outside the allocated prefix, malformed and doubled CIDRs, repeated addresses across active sessions and documentation-only endpoints which must parse but never form a tunnel. The one-view PSK exchange is also driven through the browser. The Registry relays encrypted material, while the generated configuration only contains a placeholder for the secret link rather than the key itself.

IPsec has eight cases and 332 checks across native VyOS VTI configuration and route-based XFRM on the other routers. The lab checks which side initiates, whether exactly one SA pair appears, BGP and payload state, and restart recovery.

Then there are the routing services around the session. The RPKI matrix moves routes through Valid, NotFound, Invalid and back to Valid without restarting BGP. The IRR tests change AS-SET membership, run the generated update jobs, check atomic reloads and make WHOIS fail to ensure the old filters remain in place.

By the time the complete run closed, it covered the portal, Registry, Looking Glass, WHOIS, ROA service, route collector and the four router implementations. There were no manual policy edits in the passing run.

What Broke

The early failures were not exotic. Some were exactly the sort of small template mistake which slips through when the person writing the generator also reads its output:

one path could produce an IPv6 host address with /128/128;
provider and customer roles were reversed in parts of the generated guidance;
tunnel addresses were syntactically valid but not proven to belong to the ASN configuring them;
a pending collector session could disappear from the portal before it had a chance to establish;
IRR updater variants disagreed about source names and stale-list cleanup;
some community actions were documented but not preserved across the customer and provider policy boundary.

The last one caused the most work. The first complete run passed, and a later production handoff audit showed that the functional large-community actions had not been exercised deeply enough. I reopened the test closure instead of treating the existing report as sacred.

The corrected run checks local-preference requests, no-export, one/two/three-AS prepends, strict RPKI, remote-triggered blackholing, RFC 7999 and the controls which should remain inside the T1 mesh. All eight functional cases are created through the portal and use its rendered policy. The additional 104 assertions now pass across BIRD2, FRR, VyOS and Pathvector.

That was a useful reminder that a large test count can still contain a very precisely shaped hole.

A Session Has Several Kinds of State

One outcome of the lab is that the portal no longer treats an active Registry session as proof that the network is active.

There is workflow state: both parties accepted the relationship and supplied the necessary details. There is configuration state: the portal can render complete input for the chosen platform. There is transport state: the direct link or tunnel actually passes traffic. Then there is BGP and policy state: the session is established, the collector sees it, and the received routes have the expected validation and communities.

The page can show those separately. A session waiting for a router is not failed merely because the collector has not seen it yet. A configured tunnel with no handshake is not active merely because both users filled in the form.

The distinction also made the next bit of work possible. HubN already had peerings which were established before this workflow existed. They appeared in the Looking Glass, but the Registry knew nothing about them.

Previously: Making HubN Registry Updates Durable.

Next: Importing the Network That Already Existed.

Four Dead Nodes and a Flat Graph

Jubaleth — Fri, 07 Aug 2026 00:00:00 +0000

During an early SkyeEngine Fabric security soak, four of the five coordinators exited after their certificate caches expired. The analysis tool later told me that resource growth looked excellent.

Technically, it was correct. Dead processes are wonderfully stable.

The tool was measuring the slope without first checking that every expected process had supplied samples for the whole test. I fixed the gate so it checks coverage, liveness and restart counts before looking at memory or file-descriptor growth.

I now use the same check elsewhere. Missing metrics, skipped corpus categories and a benchmark client which stopped doing any work can all produce beautifully calm graphs.

When Generated BGP Config Parsed but Routing Failed

Jubaleth — Thu, 06 Aug 2026 20:30:00 +0200

For a while I used successful configuration parsing as the reassuring final step in HubN’s peering tests. It catches syntax errors and unsupported directives, which is useful. It does not tell me what routes a daemon will actually accept, reject or export.

Running the configurations on BIRD, FRR, VyOS and Pathvector found the failures the parser could not: a /128/128 filter, reversed provider and customer labels, ownership checks against the wrong address, and communities which disappeared on the way through.

Those were valid configurations. They just described the wrong network.

The test lab now checks the live RIB and the resulting policy behaviour, including relationship changes and RPKI state transitions. Parsing stays near the start of the test. It no longer gets to call the configuration correct.

The Five-Millisecond Timer

Jubaleth — Thu, 06 Aug 2026 00:00:00 +0000

The first Fabric file-transfer benchmark managed about 169 MB/s. Buffer sizes, flow-control windows, storage and CPU all looked like reasonable suspects. None of them was the main problem.

The connection driver woke on a fixed five-millisecond timer. The function behind that timer did not merely check for a new stream; it drove reads, writes, flow-control updates and wake-ups for the entire multiplexer. Everything waited for a gate which opened 200 times per second.

Replacing the polling loop with the runtime’s actual wake-up mechanism moved the same in-memory benchmark from 168.9 MB/s to 628 MB/s. Peak memory fell as well because frames no longer piled up while waiting for the next tick.

Five milliseconds sounds tiny when looking at one sleep call. Put it in front of every useful piece of work and it becomes the fastest thing in the network.

Testing SkyeEngine Fabric

Jubaleth — Thu, 06 Aug 2026 00:00:00 +0000

Most of my recent SkyeEngine writing has been about models, corpora and the machinery needed to build them. Running all of that is a different problem. The GPU workers, storage, inference services and tools don’t necessarily share a rack, a provider or even a country.

I built SkyeEngine Fabric to connect those pieces. It is a secure overlay with one identity system and one way for services to find each other. Coordinators form a hierarchy, clients attach at the edges, and traffic can cross the Fabric without every service needing to know how the physical network underneath is arranged.

Until recently, much of this existed as architecture and individually tested components. Over roughly ten very busy days I brought the whole thing up, put it under load, broke it on purpose, fixed what fell out and deployed it again. Some of the results were good. A few of the early ones were impressively bad.

The implementation, test harnesses and detailed results are private. This is the public account of what I tested and what I found.

Building the Five-Coordinator Network

The first milestone was deceptively simple: run the intended topology end to end.

Fabric has a root coordinator, a distribution tier and access coordinators at the edge. The root controls identity and global state without becoming a transit router for every byte. Distribution coordinators carry routes between parts of the network. Access coordinators accept clients such as inference containers, storage services and tools.

I assembled a five-coordinator network with a root, two distribution nodes and two access nodes, including the cross-link which lets the two distribution branches reach each other without sending ordinary traffic through the root.

The reference topology separates global coordination from the ordinary client traffic path.

That first complete bring-up found exactly the sort of faults integration tests are supposed to find. The workspace was not reproducible from a clean build. Database migrations made assumptions about the current directory. Registration tokens could not actually be reused as configured. Several settings were decorative while developer paths remained hard-coded. A newly registered coordinator could briefly serve the wrong kind of socket before its certificate arrived. Some status fields reported plausible-looking nonsense.

None of these were intellectually glamorous failures. Collectively, they were the difference between software which compiled and a network which formed.

The result is now a containerised five-node reference environment which bootstraps from empty state, registers every coordinator, establishes mutual TLS links, converges on the expected routes and checks that traffic really crosses the intended path. A bounded version of that process also runs in CI against a fresh stack.

Adding Enough Instrumentation

Once the network stayed up, the project shifted from “does it start?” to “what is it doing?”

I added telemetry for clients, links, routes, queues, handlers, authentication, message paths, verification, compression, file relays and resource use. Prometheus records the time series and Grafana gets the same provisioned dashboards every time. The load generators also report their own active sessions and requests, so I can compare their view with what the coordinators think is happening.

The test harness also captures process-level evidence which ordinary service metrics miss: threads, file descriptors, mappings, anonymous memory, container restart counts and kernel limits. Soak analysis checks coverage as well as slopes. A perfectly flat graph from a process which died eleven hours ago is no longer accepted as excellent stability.

I know this because the first version happily did exactly that.

An early security-enforcement soak killed four of five coordinators after approximately one hour. Their certificate caches expired, a replay-protection assumption caused nodes to reject messages they had signed themselves, route validation failed, and the peers deliberately exited. The reporting tool later announced that no growth had been detected because it never checked whether most of the processes were still present.

The failure gave me plenty to fix. I corrected the replay logic, closed an adjacent signature-validation bypass, made restart behaviour explicit and tested the soak gate against both passing and failing evidence. I also extended the test, because an hour-long cache obviously can’t be validated by a run which ends after fifty minutes.

The clean post-fix soak ran for 13.7 hours. I accepted it with a duration waiver instead of calling it the planned 24-hour run. All five coordinators stayed alive with zero container restarts, resource curves remained bounded, pending work drained and 39,896 envelope verifications completed without a failed, rejected or unverifiable result. The graphs had been flat for long enough that I didn’t expect another ten hours to reveal much, but almost 14 hours is still not 24 hours.

Finding the Five-Millisecond Delay

File-transfer testing produced the most satisfying single performance result.

The first measured implementation moved data at about 169 MB/s while the underlying link, storage and CPUs all appeared to have room. The obvious suspects were buffer sizes and flow-control windows. They were not the problem.

I eventually found a fixed five-millisecond timer in the connection driver. The function behind it drove the entire multiplexer, so reads, writes, flow-control updates and wake-ups were all waiting behind a gate which opened only 200 times per second.

Replacing that polling loop with the runtime’s real wake-up mechanism, then overlapping the file stages correctly, moved the same in-memory transfer from 168.9 MB/s to 628 MB/s: a 3.7× improvement. Peak memory also fell because frames no longer accumulated while waiting for the next tick.

The investigation corrected several benchmarks along the way. A supposedly very fast disk test was writing compressible zeroes to ZFS and mostly measuring compression. A 2 GiB container limit made the kernel reclaim memory throughout a 2 GiB transfer, quietly cutting throughput in half. A receiver intended to discard data for a transport-only measurement was initially slower than writing it because the synthetic path had changed how buffers were allocated.

The larger number was nice, but fixing the benchmark was just as useful. It now explains where the time and CPU went, verifies the bytes end to end and rejects a result when the measurement path changes the system more than the optimisation being tested.

Moving File Transfer to the Clients

That performance work also exposed a design mistake: coordinators are the network, not the endpoints.

The real transfer is between clients—a model worker and a file server, for example—not between two coordinators. Fabric now brokers a direct client-to-client connection when the endpoints can reach each other and provides a cut-through relay when they cannot. The sender can request direct, relay or automatic selection, and the acceptance test verifies the chosen path independently instead of trusting the sender’s label.

The direct path carries no file bytes through a coordinator. Fabric still authorises the introduction and records both ends’ audit reports, so it can say who moved what, which path was used and whether both clients verified the same transfer without pretending it inspected traffic it never saw.

On the local transport-only harness, the direct path reached a median 1,832 MB/s, while a one-coordinator relay reached 950 MB/s. With real disk writes enabled, both converged around 590 MB/s because the disk—not Fabric—became the ceiling. Those are laboratory numbers from one machine, not internet throughput claims, but they show that the primary and fallback designs both work and that the instrument can tell transport from storage.

The panels deliberately separate the multiplexer benchmark from the later client-transfer harness.

I also tested the transport under controlled bandwidth, delay and loss. On a difficult shaped path, TCP with BBR and QUIC with BBR were effectively tied. TCP with the default loss-based controller collapsed. On an idle fast link, kernel TCP was several times faster than the userspace QUIC implementation. I therefore kept TCP, TLS and BBR for the first version. In these tests, the congestion controller made the difference, not switching to QUIC.

Testing on Rented Servers

Local containers are excellent for correctness and profiling. They are a poor substitute for multiple machines when the question is whether a distributed system behaves like one.

I rented eight dedicated-CPU servers with 62 vCPUs in total: five Fabric coordinators, an observer and two independent load generators connected over a private network. Public interfaces were used for management only; the service ports weren’t reachable from the public internet. I scripted the network, firewalls, machines, credentials, immutable container image, verification, evidence export and teardown so I could rerun it without remembering what I had typed the previous evening.

The main capacity run held 5,000 authenticated clients on each access coordinator, or 10,000 simultaneous client sessions. Every scheduled plateau was reached. The balanced run completed 26,902 cross-fabric requests with 100% success. The deliberately skewed run completed 32,904 of 32,905; the single timeout occurred during the intentional final drain while sessions were being closed and remained well inside the frozen acceptance threshold.

Stable request latency was comfortably below the predeclared gates: p50 was approximately 4–18 ms and measured p99 approximately 25–49 ms. At the balanced 10,000-client plateau, the generators and coordinators used surprisingly little CPU, queues did not accumulate, and every stable sample agreed exactly between generator session counts and coordinator-authenticated client counts.

The clean rerun used the same frozen gates as the failed smoke which preceded it.

That clean result was a rerun. The first remote smoke reached its client target but sent most requests to registered identities which were not actually online. The resulting success rate was awful and completely correct for the broken workload. The failed run was retained, the load protocol was fixed, a new immutable image was built, and the test started again without changing the thresholds.

I’ve run into this more than once now: a benchmark can execute perfectly and still test the wrong thing.

Restarts, Churn and Other Unpleasant Cases

I reused the same paid cluster window for several sequential tests. Running unrelated benchmarks at the same time would have made all of the results harder to trust.

During route recovery, cross-subtree client traffic continued while a distribution coordinator received a graceful stop, an abrupt kill and a full recreation. The original topology and route set returned without manual intervention after outages of approximately 15, 45 and 90 seconds respectively. Queues drained, no unrelated coordinator restarted, and the final queue-latency p99 remained under half a millisecond.

The identity-churn test connected and drained 7,500 distinct clients, introduced 32 deliberate authentication stalls, restarted an access coordinator during the 2,500-session plateau, and then proved that a previously unused identity could still connect afterward. Active clients, pending requests, data handlers and DNS state all returned to baseline. The coordinator-certificate cache remained unchanged, which is the correct result: it contains coordinator certificates, not one entry for every client which ever visited.

Compression received its own protocol matrix. Every one of the 55 supported message types was named exactly once, including explicit exclusions rather than a vague “all messages passed.” Live traffic covered peer-to-peer, peer-to-root, root-to-peer, local, relayed and request/response paths. Compiled regressions covered negotiation, thresholds, both supported formats, incompressible input, malformed input, size limits and encoder reuse. The run completed without decompression errors, connection loss or unbounded resource behaviour.

After each stage, the observer exported an immutable bundle containing the Prometheus snapshot, Grafana configuration, machine-readable results, node facts, image and configuration identities, and checksums. Those bundles were opened locally before the cluster was destroyed. The dashboards still work after the expensive machines no longer exist, which makes analysis independent of the billing clock and means a result is more than a screenshot someone remembered to take.

Testing the Protocol Signatures

A large portion of the work was less visible but more important than a throughput graph.

Fabric envelopes were already carrying signatures, but several paths did not consistently verify them. The enforcement work centralized what a signature covers, signed the last unsigned handshake path, removed duplicate protocol implementations, repaired replay handling across reordering and restarts, and ran the full network with verification in enforce mode.

Registration errors now propagate instead of being translated into misleading success. Certificate caches are bounded. Client and peer shutdowns drain cleanly. Coordinators recover after host and container-runtime restarts. Status reports distinguish links from routes. Dead transfer protocols and configuration switches were removed instead of being documented forever as theoretical capabilities.

Perhaps most importantly, these properties are now represented in end-to-end tests. “The code appears to verify this” is weaker evidence than a five-node network refusing a forged message while continuing to route legitimate ones.

What I Still Need to Test

I closed a large group of correctness, operability, security and acceptance tasks, but Fabric isn’t finished and I still don’t have one magical maximum-throughput number.

The local transfer harness has a restart-dependent throughput variance which is stable within one process lifetime and different after recreation. Direct connections also show more run-to-run spread than relayed transfers. The current machines cannot cleanly separate physical-host placement, virtual networking and Fabric itself.

The next serious file-transfer run therefore needs a deliberately different environment: separate endpoint hosts, independent local NVMe, a stable high-speed private link, full kernel control for BBR and traffic shaping, and the complete Fabric topology deployed alongside the clients. A 10 Gb/s environment is realistic for acceptance, while a faster or bare-metal run may still be needed to characterize the absolute direct-path ceiling.

I want to design that environment properly instead of improvising it during another paid test window. Until then, the throughput variance stays open rather than being averaged into a nicer-looking number.

There is still plenty to build. What changed is that Fabric now has a repeatable way to expose faulty assumptions.

It can form a real network, authenticate and route for 10,000 concurrent clients, recover from deliberately removed coordinators, clean up after churn, broker direct transfers, relay when direct is impossible, enforce its signed protocol, export its own evidence and survive after the test cluster has been deleted.

For roughly ten days of testing, I’m quite happy with where it ended up. Fabric still has plenty of unfinished edges, but I can now reproduce a complete network, break it in several known ways and tell whether it recovered. That is much more useful than another diagram claiming it should work.

Figures in this post describe the verified development and acceptance runs completed through 6 August 2026. Local transfer numbers are laboratory measurements, not promises about internet throughput. The post-fix soak ran for 13.7 hours with an explicit duration waiver; it is not represented as a completed 24-hour run.

My Fast Disk Was Compressing Zeroes

Jubaleth — Thu, 06 Aug 2026 00:00:00 +0000

One of the Fabric disk benchmarks produced a throughput number which looked almost too good. It was.

The test wrote zero-filled data to ZFS. ZFS quite reasonably compressed it, so the benchmark moved very few real bytes and reported the result as storage throughput. The graph was accurate about the test it had run; the test just was not asking the question I thought it was.

Fixing this meant checking the data, the bytes which actually reached each stage and whether the measurement path itself changed the system. I ran into similar problems with container memory limits and a synthetic receiver which was slower than writing to disk because it allocated buffers differently.

I am now rather suspicious of a benchmark which produces a delightful number before it can explain where the bytes went.

Preventing Registry Read Failures From Withdrawing Every ROA

Jubaleth — Wed, 05 Aug 2026 11:40:00 +0200

The ROA publisher periodically reconciles its local state with the Registry. There is a dangerous shortcut in that sentence: if the Registry request fails and the result looks like an empty list, reconciliation can faithfully delete everything.

No data and no answer are different states. HubN now keeps the last known ROAs when a fetch fails. Withdrawal only happens after a complete, successful read says an object is gone.

There is a second check during a full resync. Before withdrawing anything, the publisher reads the source IDs again. If the Registry changed while pagination was in progress, that reconciliation pass is abandoned and retried against a coherent view.

Most of this code exists to avoid turning temporary uncertainty into permanent action. That is a distinction I want to keep explicit anywhere HubN mirrors authoritative state.

Making HubN Registry Updates Durable

Jubaleth — Wed, 05 Aug 2026 00:00:00 +0000

The HubN Registry stores allocations and requested state. Other services use that information to publish reverse DNS, generate ROAs, issue certificates and decide which BGP sessions ought to exist.

For a long time the Registry wrote a change to PostgreSQL and then published an update through Redis. That was fine as long as Redis was available and the Registry process remained alive for the few milliseconds between those two operations.

That “and” was doing rather a lot of work.

If the database commit succeeded and publication failed, the Registry had the new state but its consumers did not know to fetch it. Retrying the API request might produce the event, reject it as a duplicate or change the resource again, depending on the endpoint. None of those are especially good recovery procedures.

I replaced that path with a transactional outbox and then went through the consumers to make sure duplicate delivery and missed history were both survivable.

Putting the Event Beside the Change

Infrastructure updates are now inserted into an outbox table by PostgreSQL triggers. The resource change and its event are part of the same transaction, so either both exist or neither does.

The triggers cover the changes which have operational consequences: prefix routing and reverse-DNS settings, PTR records, anycast route origins, certificate authorities, new certificates and revocations. API handlers no longer have to remember which Redis message belongs after which database call.

A Registry worker claims pending rows in batches and publishes them. Multiple Registry replicas can run the worker at once; SKIP LOCKED keeps them from picking the same row during the normal case. Claims expire after five minutes so a dead worker can’t own an event forever.

Failed publications use bounded exponential backoff. Successful rows remain for seven days before cleanup, which has already made debugging nicer because there is an actual record of what the Registry tried to announce.

There is one uncomfortable part which can’t be removed with a table: the worker may publish an event and crash before marking it complete. The event will be sent again after its claim expires. The delivery guarantee is therefore at least once, not exactly once.

That is intentional. Losing an update is worse than asking a consumer to process the same desired state twice.

Ordering Still Matters

Duplicates aren’t the only awkward case. Suppose a prefix has reverse DNS enabled and then disabled while Redis is down. Once publication resumes, sending those two events in the opposite order would leave the DNS service doing exactly the wrong thing very reliably.

Each outbox row has an aggregate key derived from the affected prefix, anycast allocation or other resource. Workers can publish changes for different resources in parallel, but a later event for the same resource cannot pass an earlier pending one.

The event remains small. It says what changed and identifies the resource; it doesn’t attempt to be a second copy of the Registry row. Consumers fetch the current representation before acting. If several updates collapse into one current state, that is fine. They are reconcilers, not historical replay engines.

Updating the Registry event consumers

An outbox only closes the gap between PostgreSQL and Redis. It doesn’t make the code on the other end correct.

The ROA service had a particularly unpleasant edge case. A prefix allocation and an anycast allocation can legitimately describe the same ASN, prefix and maximum length. The old uniqueness rule let one source effectively take ownership of the other’s row. Deleting either source could then withdraw the shared ROA even though the other authorization still existed.

ROAs are now tracked by their Registry source. Retiring one source only removes the published object when no other active source still needs the same URI.

The service also performs periodic full reconciliations in addition to handling live events. It fetches all active prefix and anycast sources, processes them, then reads the source IDs again before withdrawing anything. If the Registry changed during the paginated scan, withdrawal is refused and the next reconciliation starts over. A failed Registry request keeps the last known ROAs in place instead of interpreting “I couldn’t fetch it” as “it no longer exists.”

That last distinction sounds obvious when written down. It was less obvious in code where an empty result and an unavailable service can both arrive near the same branch.

Rebuilding the Looking Glass Safely

The Looking Glass has the same general problem with a different kind of state. It receives routes from GoBGP and session intent from the Registry. After a collector reconnect, it needs to rebuild its view without showing a mixture of the old and new RIB.

Routes are now loaded into a new RIB generation. The previous generation remains queryable until the replacement has completed, at which point the service switches generations and retires the old one. A disconnect halfway through a dump leaves the last complete view available.

I also split two session cases which I had previously treated alike. A Registry session which is still waiting for the collector is pending and should remain visible. Collector status for a session which no longer exists is stale and should be pruned. “Not currently observed” isn’t enough information to decide between them.

The DNS coordinator follows the same pattern after the publication fixes: live events make it react quickly, while periodic comparison and applied serial reports repair anything which fell out of step.

What the Event Means Now

The Registry remains authoritative for allocations and requested state. Redis is a wake-up path, not the only surviving copy of a change and not proof that a consumer applied it.

The resulting flow is less clever and much harder to lose:

PostgreSQL commits the resource and its outbox row together.
A worker publishes the event, retrying when it cannot.
The consumer fetches the current state and applies it idempotently.
Periodic reconciliation compares the complete source again.
The service which owns the operational state reports what actually became active.

There are still windows where two systems disagree. This isn’t a distributed transaction pretending otherwise. The difference is that those windows now have a durable repair path, and a temporary outage no longer requires me to remember which checkbox to toggle to make a service notice the database again.

Previously: Fixing HubN’s DNSSEC and RPKI Publication.

Next: Testing the HubN Peering Workflow.

Finding the Broken Delegation Above a Signed Zone

Jubaleth — Tue, 04 Aug 2026 10:15:00 +0200

HubN’s reverse zone had valid signatures, a matching DS record and an authoritative server which answered correctly when asked directly. Normal recursive lookups still could not reach it.

The private delegation had been placed below the public arpa. tree. A validating resolver followed the public chain, quite correctly found no private child there, and stopped. None of the cryptography was the problem.

The fix was to make the private view traversable all the way down: root, arpa., ip6.arpa., then the HubN reverse zone. Once the delegation chain existed in the same view, the already-correct signatures became useful.

I had spent too long inspecting keys because the failure looked like DNSSEC. It was ordinary DNS topology wearing a DNSSEC-shaped hat.

Fixing HubN's DNSSEC and RPKI Publication

Jubaleth — Tue, 04 Aug 2026 00:00:00 +0000

HubN has its own DNS hierarchy, certificate authorities and RPKI repository. Most of the individual parts had been working for a while. The less convincing bit was what happened while they changed.

A manifest could be fetched just before its CRL changed. A root server could receive a zone update but fail halfway through applying it. A reverse zone could be correctly signed and still be impossible to reach because I had delegated it from the wrong side of an existing zone cut.

None of those failures produced a satisfyingly broken file. They produced collections of individually plausible files which didn’t describe the same moment in time. This week I went through that path from the CA to the validators and authoritative servers, and made the publication boundaries much less hopeful.

Publishing One RPKI Generation

An RPKI publication point is a set of related objects. The manifest lists the files which belong to the current generation, the CRL says which certificates are no longer valid, the ROAs contain the route-origin authorizations, and RRDP tells a validator how to move between repository states.

My first implementation updated those pieces in the right general order, but the order wasn’t a transaction. There were gaps where a validator could see the new manifest with the previous CRL, or a new RRDP serial whose snapshot hash still described the old object set.

That is the sort of thing which tends to pass a local test. Fetch the repository again a second later and everything has caught up. A validator arriving in the gap isn’t obliged to wait for my code to finish, though. It sees an inconsistent repository and is quite entitled to reject it.

The publisher now assembles a complete future generation first. Changed objects are staged, then the publication rows, RRDP delta, serial and snapshot hash are committed together. Identical objects are left alone, which avoids manufacturing a new serial just because the periodic job ran again.

The CA side also marks related manifests and CRLs with the same generation. The ROA service reads the trust-anchor manifest before and after fetching the CA objects. If the generation changed during the fetch, it throws the attempt away and starts again. It is a slightly annoying extra round trip which is preferable to publishing a beautifully signed mixture of two different states.

I also fixed the manifest serials so they survive a restart. Resetting a monotonic counter because a container was replaced is easy to do and difficult to explain to software which has already seen the larger number.

Fixing reverse-DNS tree traversal

The IPv6 space inside HubN lives below 6660::/16, so its aggregate reverse zone is 0.6.6.6.ip6.arpa.. I initially delegated that name directly from HubN’s private root.

That looks reasonable if I only inspect the root zone. It doesn’t work for an ordinary resolver.

The public DNS already delegates arpa.. Once a resolver follows that cut, a private delegation for 0.6.6.6.ip6.arpa. hidden in the root is no longer on the path. The resolver asks the public arpa. hierarchy instead and never discovers the HubN reverse zone.

The private root now carries an authoritative overlay for the parts of the path it needs:

.
└── arpa.
    └── ip6.arpa.
        └── 0.6.6.6.ip6.arpa.
            └── delegated prefix zones

The coordinator builds and signs that hierarchy from the leaf upwards. Each child publishes its DS record into the closest managed parent, and only after the overlay zones exist does the root activate the first delegation. The old unreachable delegation is removed as part of the same reconciliation.

This also forced me to stop assuming that every label on the way to a zone is itself a managed zone. Between 0.6.6.6.ip6.arpa. and ip6.arpa. are several names which exist only because a descendant exists. DNS calls these empty non-terminals.

A Name Can Exist Without Having Records

The empty non-terminal case found another bug in the authoritative server.

If a name has no records but has descendants, it exists. A query for that name should return NODATA. My server was only checking for an exact record set, so it returned NXDOMAIN instead. With DNSSEC enabled it also built the wrong denial proof, which made a valid branch of the reverse tree look as though it did not exist.

The in-memory zone now checks both exact owners and descendant names. NSEC generation includes the empty non-terminals needed to prove the answer, and DS propagation walks upwards until it finds an actual managed parent rather than stopping at the first textual parent.

It is one of those DNS details which seems absurdly fussy right up until a validating resolver uses it to discard the answer.

Applying Registry updates after delivery

The coordinator distributes signed zones to several root servers. The old path treated a completed stream as fairly strong evidence that the update had landed. It wasn’t.

Zone transfers now carry a canonical digest made from the serial and decoded wire-format records. Ordering doesn’t affect the digest, but the actual DNS data does. A root server assembles the candidate away from the live zone, checks the digest, refuses serial regressions and only then swaps it into service.

If a delta is incomplete or its digest doesn’t match, the server keeps the previous good zone and asks for a full copy. The root server also reports the serials it has actually applied. That gives the coordinator something useful to compare instead of equating “I sent serial 42” with “serial 42 is serving.”

The same distinction now exists in the other trust path too. A newly written ROA row isn’t the same thing as a coherent RRDP generation, just as a signed reverse zone isn’t the same thing as a reachable, applied zone.

Where It Ended Up

The CA, ROA publisher, DNS coordinator and root servers still run as separate services. I don’t want one enormous process just to obtain transaction-like behaviour across them. The boundaries are now explicit instead:

a CA generation can be read consistently or retried;
an RPKI publication becomes visible as one RRDP generation;
the private reverse hierarchy follows the path a resolver actually takes;
signed zones are verified before replacing the live copy;
serials describe applied state and survive restarts.

There is more machinery here than I expected when I decided a private network should have proper DNSSEC and RPKI. That has been true of nearly every part of HubN. The useful bit is that a failure now leaves the previous generation in service, rather than briefly publishing a newer state which never existed as a whole.

Next: Making Registry Updates Durable.

Revoicing the Whole Assistant Side

Jubaleth — Fri, 17 Jul 2026 14:30:05 +0200

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

PRIVATE RESEARCH STATUS

“Revoice this conversation as GLaDOS” sounds like a request to rewrite the final answer. For ordinary dialogue, that may even appear to work.

Tool traces make the shortcut obvious. The assistant may reason about what to do, issue several calls, read the results and only then answer. Rewriting the last paragraph leaves the decisions, tool arguments and reactions untouched. The conversation ends in character but reaches that ending as a different assistant.

The GLaDOS pipeline now regenerates the complete assistant side while keeping the user messages and tool results fixed. Each assistant slot is produced from the context available at that point; later tool output cannot leak backwards into an earlier decision.

This is slower than a style pass and much easier to get wrong. It is also the only version which teaches the model to remain GLaDOS while doing the work, rather than adding the voice after the work is already finished.

What Belongs in SkyeEngine

Jubaleth — Fri, 17 Jul 2026 00:00:00 +0000

It would have been very easy to copy all of the GLaDOS tooling into a new project, rename a few things and call it SkyeEngine. The result would have been GLaDOS with a generic name.

I use a fairly simple boundary: SkyeEngine owns the mechanism and the model project owns the policy.

SkyeEngine can submit inference, save attempts, normalize a conversation, ask a judge, validate a patch, fingerprint an artifact and render a dataset. It shouldn’t decide what sounds like GLaDOS, which defects her rubric cares about or which sources belong in her training mixture.

Sometimes that means leaving apparently reusable code in GLaDOS until I understand the pattern properly. Moving it later is mildly annoying. Moving the wrong assumption into the shared engine is much worse.

The 120B Model Learned the Bad Parts Too

Jubaleth — Fri, 17 Jul 2026 00:00:00 +0000

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

PRIVATE RESEARCH STATUS

I trained the first GLaDOS 120B experiment on the older personality corpus. My optimistic theory was that a much more capable base model would keep the voice and handle the material better.

Checkpoint 100 passed the narrow identity check: none of the 28 held-out prompts leaked ChatGPT or OpenAI. Actually reading the answers was less encouraging. Old template markers survived, straightforward coding requests became refusals and the personality collapsed into repetitive contempt.

The larger model had not repaired the weak data. It had enough capacity to learn those weaknesses very well.

I stopped the run and went back to the corpus. Keeping the GPUs busy would have produced more checkpoints, but it would not have fixed the examples teaching GLaDOS that performing disdain mattered more than completing the task.

A SkyeEngine Update

Jubaleth — Fri, 17 Jul 2026 00:00:00 +0000

It’s been a while since I’ve written a proper project update. Far too long, really. I haven’t been idle, though. Over the last few days I’ve put an honestly unreasonable amount of work into SkyeEngine, GLaDOS, and the corpus that will eventually teach her how to think.

SkyeEngine started as a framework for building personality-aligned AI models. It had projects, prompt composition, a pile of inference backends and some useful command-line tools. It gave me enough room to experiment without rebuilding everything each time, but many of the pieces still had their own slightly different way of doing things.

GLaDOS already existed by then. There had been an 8B model and later a 14B model, both built from DeepSeek distills. They proved that the personality could work, but they were still dialogue models. I wanted the next version to code, use tools, read what those tools returned and carry on working from the result. That turned out to require rather more than adding a few new prompts.

Teaching a model a strong personality is already a slightly strange job. For the next GLaDOS, I also need the personality to survive reasoning, coding, conversation and tool use without getting in the way of the answer. Once the corpus reached hundreds of thousands of conversations, my old workflow of “run this script, then remember which directory it wrote to” became a very good way to ruin an expensive training run.

This post is an attempt to catch up on where all of that work has ended up.

Normalizing Conversations

The first problem sounds incredibly boring: what, exactly, is a conversation?

Every dataset and model provider has a slightly different answer. A message might contain plain text, hidden reasoning, a tool call, a tool result, or some combination of them. Tool arguments might be an object in one source and a JSON string in another. Some datasets have stable IDs, others only have a position in a file. All of this is manageable when you are looking at ten examples. It is less fun when a quiet conversion bug can damage thousands of them.

SkyeEngine now has a single, canonical representation for conversations. Messages, analysis, final replies, tool calls, tool results, tool definitions and their provenance all have an explicit place. More importantly, individual assistant turns have stable identities. A judge can point at the exact turn and channel containing a defect, and a repair can change that exact text without guessing which array index someone meant.

There are structural checks for the fun things one eventually discovers in real data: duplicate calls, missing tools, orphaned results, replies arriving in the middle of a tool exchange, and traces which simply end before anything useful happens. Not glamorous, but neither is discovering after training that your model learned from broken tool calls.

Inference Jobs

SkyeEngine has supported multiple inference backends for a long time. The new work makes those backends behave like parts of the same system instead of a menu of unrelated API clients.

A request can run immediately, become a durable job, or be split into a remote batch. Its attempts, results and token usage are recorded in the same shape regardless of which provider handled it. The job state can remain under my control while the actual inference runs on rented accelerator capacity. A process can stop, another can inspect the job later, and completed batch shards can be fetched without pretending the whole batch succeeded when half of it is still somewhere in a queue.

This is handy for ordinary inference and essential for data generation. Producing a corpus means running a long chain of requests, retries, validation, selection and merging. I don’t want all of that state to vanish because one Python process stopped.

Keeping Track of the Data

That leads to artifact lineage, which is a fancy way of saying I want to be able to answer a very basic question: “What produced this file?”

Every important stage can now record its inputs, outputs, configuration, code version and parent stages. The files are fingerprinted, so SkyeEngine can tell the difference between work that is genuinely reusable and an output that merely happens to have the right filename. If an input, setting, implementation or parent artifact changes, the old result is not quietly treated as current.

I keep the earlier artifacts intact. A repaired conversation is useful, but so are the defective version and the judgment explaining what was wrong. I can later use that history as evaluation data, preference data or a test for the next judge.

Reproducibility isn’t a particularly exciting word until the alternative is regenerating several hundred thousand samples.

Judging the Corpus

One of the largest additions is structured evaluation. A project supplies its own rubric and defect categories; SkyeEngine handles the common machinery around asking a judge, parsing its verdict, tying findings back to exact messages, and storing the result.

I can also compare judges and measure how often they agree. A confident-looking JSON object doesn’t make a judgment correct. Two judges can agree that something is wrong while disagreeing entirely about what is wrong, and they can still share the same blind spot.

For disagreements, SkyeEngine can build a blinded docket for another adjudication pass. The arbiter sees the competing decisions without being told which judge produced which one. There is no reason to give a model a prestigious name to agree with.

This has already changed how I look at the GLaDOS corpus. Quality is no longer “the generator completed without an error.” It is something measured, challenged, repaired and measured again. There will be a separate post about that process because the numbers deserve more context than I can sensibly squeeze in here.

Repairing Individual Problems

Finding a defect is only half the job. The tempting solution is to send the entire conversation back through a model and ask it to make things better. Unfortunately, “better” is an impressively vague instruction. The model may fix the reported sentence while changing three correct ones, losing a tool call, inflating the reply, or adding a completely new problem.

SkyeEngine’s rewrite workflow works with localized patches instead. A patch identifies the conversation, assistant turn, channel and exact old text it expects to replace. It records where the request came from, which judgment triggered it, what the model returned, and how much the result drifted from the source.

Before anything is applied, the patch is validated. Stale source text, overlapping edits, structural damage and suspiciously large changes are refused or moved into review. The original corpus is never modified in place; a new derived layer is produced with its own lineage.

The process is much closer to changing code than asking a language model to wave its hands over a directory.

Preparing a Training Run

The training side has had the same treatment. Dataset transformations are deterministic and publish atomically. Rendering is model-aware, so the exact chat template and the placement of reasoning or tool channels are checked before a run begins. Checkpoints, evaluation results and model artifacts have explicit relationships instead of relying on whatever directory name seemed sensible at two in the morning.

There are reusable controls for stopping, resuming and inspecting a training lifecycle, plus telemetry for the parts of a mixture-of-experts model that ordinary loss graphs do not explain. The environment itself is portable too: the same CUDA container can take a project at runtime and run on different GPU providers without baking the project, its credentials or its data into the image.

None of that guarantees a good model. It does remove a large category of avoidable ways to produce a bad one.

The Failed 120B Run

I know this because I tried the other approach first.

I took the old corpus—the one built with the old SkyeEngine for training the 14B personality chatbot—and used it to train the new 120B model. The model was vastly more capable, so surely giving it the data that had already produced a recognisable GLaDOS would give me a better GLaDOS. That was the theory, anyway.

It was a spectacular failure.

The samples from checkpoint 100 are an excellent record of what went wrong. The identity test technically passed: none of the 28 held-out prompts leaked ChatGPT or OpenAI. If I had looked only at that green check mark, I might even have called the run promising.

Actually reading the answers told a different story. Old template markers were still embedded in prompts. The personality had collapsed into repetitive, indiscriminate contempt. Straightforward coding requests were refused instead of answered. Capability questions became evasive monologues about testing.

The 120B model had not fixed the limitations of the old data. It had learned them with considerably more capacity.

After reading those samples, I stopped the run and went back to the corpus. GLaDOS still has to be capable, accurate, safe when the situation demands it, and able to tell the difference between a moment for character and a moment where somebody actually needs help. Adding tools and coding to the old data would only have given her more ways to fail.

Where It Is Now

SkyeEngine is now the machinery underneath my GLaDOS work: data comes in, is normalized and traced through generation, evaluation and repair, then prepared for training and tested again on the other side. The project-specific personality, rubrics and data policy remain with GLaDOS; the reusable mechanics belong in the engine.

The recent work has taken SkyeEngine from a framework containing several useful AI tools to something I can use for the whole path from raw conversations to a reproducible training artifact. There is still a lot to do, especially around moving the remaining GLaDOS scripts onto the new pieces.

I abandoned the earlier 120B run. Keeping it going wouldn’t have made the corpus any better, and I won’t start the next one until the new data is ready. This is slightly less exciting than announcing that a huge model is training, but it should produce a considerably better result.

I’ll write about the corpus, the judges, and some of the rather surprising measurements next. GLaDOS will be able to reason across multiple turns and use tools, which means teaching her when and how to use them is at least as important as teaching her what to say afterward.

For now, SkyeEngine feels far less like a directory full of useful scripts. There is still plenty to migrate, but I can finally follow a conversation from its source, through generation and review, into a training artifact without relying on my memory of which script produced which directory. That is a rather better place to start the next run from.

Why Filenames Weren't Enough

Jubaleth — Thu, 16 Jul 2026 19:21:31 +0200

For a while, much of my model pipeline relied on directory names to explain what had happened. A folder called repaired was assumed to contain repaired data, and a familiar output filename was treated as reusable.

That works until an input changes, a script is fixed or somebody reruns one stage with different settings. The filename stays reassuringly familiar while the meaning underneath it moves.

SkyeEngine now records the inputs, configuration, code identity and parent stages for important artifacts. Files are fingerprinted, so changing any of those pieces prevents an old result from being quietly accepted as current.

I still give directories useful names; I just no longer ask the name to carry the whole history. Regenerating a few hundred thousand conversations because I cannot prove where an output came from is a fairly effective cure for casual provenance.

Stopping a Rewrite When Its Source Has Changed

Jubaleth — Thu, 16 Jul 2026 15:26:19 +0200

My first instinct for repairing a bad training example was to send the whole conversation back to a model and ask for a better version. That fixed the reported sentence and occasionally rewrote several perfectly good ones for company.

SkyeEngine now uses localized patches. A patch names the conversation, assistant turn, channel and exact old text it expects to replace. If the source has changed, the patch is stale and fails instead of applying itself to something which merely looks similar.

The same validation rejects overlapping edits, structural damage and suspiciously large changes. Accepted patches produce a new derived layer; they do not alter the original corpus.

It feels slightly fussy until two repair jobs target the same sentence or a later cleanup changes the text first. At that point, refusing to guess is considerably cheaper than discovering after training that the repair landed in the wrong place.