Source
Immutable imported material and the provenance needed to understand where it came from.
SkyeEngine
The tools I use to generate, check, repair and train model data without tying the whole project to one provider.
Began asGLaDOS tooling
Design ruleBackend neutral
WorkflowData to deployment
Network layerSkyeEngine Fabric
// WHY IT EXISTS
Why I built SkyeEngine
SkyeEngine started with the scripts I wrote while training GLaDOS. They worked, but too much depended on filenames, shell history and me remembering what I had run. That stopped being manageable once the corpus reached hundreds of thousands of conversations.
I moved the reusable machinery into SkyeEngine. GLaDOS still owns her identity, prompts, rubrics and training policy. Inference, data handling, judging, repair, rendering and training mechanics belong in the engine.
// CANONICAL RECORD
The conversation record
A record can contain ordinary text, hidden analysis, tool calls, tool results, schemas and several assistant turns. SkyeEngine gives those parts explicit identities, so a judge or repair job addresses the exact turn and channel it means instead of relying on an array offset which changes after the next transformation.
Structural validation catches duplicate calls, orphaned results, undefined tools, interrupted exchanges and records which stop before a trainable answer. Source selection and project policy remain with the owning project; the portable record and validation rules belong here.
CONVERSATIONSTRUCTURE / VALID
USERCONTENTStable turn identity
ASSISTANT / ANALYSISREASONINGAddressable channel
ASSISTANT / TOOLCALL + RESULTLinked identities
ASSISTANT / FINALRESPONSETrainable outcome
// THE WORKBENCH
The data lifecycle
Each step leaves an inspectable artifact for the next one. A familiar filename is never enough to prove that work is current.
01Adapt→
02Generate→
03Validate→
04Judge→
05Repair→
06Render→
07Evaluate
// PROVENANCE
Artifact lineage and repairs
Every important stage records inputs, outputs, configuration, code identity and parents. Fingerprints prevent a familiar filename from impersonating current evidence.
IMMUTABLE LAYER
Judgment
A separate record of the exact defect, rubric and target turn which triggered intervention.
Repair
A localized proposal with its own lineage, never an in-place mutation of the source layer.
Evaluation
Post-repair evidence which can disagree without rewriting the history which produced it.
// EXECUTION
Execution modes and resumable jobs
Neutral intent stays in the control plane. Each backend owns its translation into provider-specific requests and responses.
Immediate
Run a request now when the result is small enough to remain attached to one process.
Durable
Persist attempts, state and partial failures so a process exit does not erase what actually happened.
Remote batch
Split slow work across a provider or accelerator backend without representing a half-fetched batch as complete.
Engine components
Forge
Dataset preparation, canonical transformations and model-native rendering primitives.
Sentinel
Portable structural checks plus explicit attachment points for project-owned rules.
Engine core
Backend-neutral execution, controlled rewrites, structured evaluation and lifecycle orchestration.
// FROM ONE MACHINE TO MANY
SkyeEngine Fabric
Fabric is the distributed execution plane for the point where a model-building workflow stops fitting on one machine. It connects GPU workers, storage, inference services and tools across machines and providers without making every service understand the physical network underneath.
5Coordinator reference topology
10,000Simultaneous authenticated clients
26,902 / 26,902Balanced requests completed
0Coordinator restarts in stable plateaus
Coordinators provide one signed identity and discovery system, distribute routes and introduce clients. The root owns identity and global state without becoming the transit path for every byte. Distribution coordinators carry cross-fabric routes; access coordinators attach workers, storage and tools at the edges.
Transfers happen between clients. Fabric brokers a direct connection when the endpoints can reach each other and falls back to a cut-through relay when they cannot. Both ends report what they verified, so the system retains an audit trail without claiming to have inspected traffic which never crossed a coordinator.
Identity + global state
Owns signed identity and the shared view of the Fabric without carrying ordinary cross-fabric traffic.
Routes + transit
Connects regions of the Fabric and carries routes between access tiers through explicit cross-links.
Client edge
Attaches inference workers, storage services and tools while keeping their physical placement behind the Fabric boundary.
CURRENT MIGRATION STATE
SkyeEngine now has canonical conversation data, artifact tracking, structured evaluation, controlled rewrites, durable jobs and model-aware training preparation. I am still migrating the remaining GLaDOS orchestration onto it. Some scripts already use the new libraries; that does not mean the full workflow has moved or passed end-to-end testing.