It’s been a while since I’ve written a proper project update. Far too long, really. I haven’t been idle, though. Over the last few days I’ve put an honestly unreasonable amount of work into SkyeEngine, GLaDOS, and the corpus that will eventually teach her how to think.
SkyeEngine started as a framework for building personality-aligned AI models. It had projects, prompt composition, a pile of inference backends and some useful command-line tools. It gave me enough room to experiment without rebuilding everything each time, but many of the pieces still had their own slightly different way of doing things.
GLaDOS already existed by then. There had been an 8B model and later a 14B model, both built from DeepSeek distills. They proved that the personality could work, but they were still dialogue models. I wanted the next version to code, use tools, read what those tools returned and carry on working from the result. That turned out to require rather more than adding a few new prompts.
Teaching a model a strong personality is already a slightly strange job. For the next GLaDOS, I also need the personality to survive reasoning, coding, conversation and tool use without getting in the way of the answer. Once the corpus reached hundreds of thousands of conversations, my old workflow of “run this script, then remember which directory it wrote to” became a very good way to ruin an expensive training run.
This post is an attempt to catch up on where all of that work has ended up.
Normalizing Conversations
The first problem sounds incredibly boring: what, exactly, is a conversation?
Every dataset and model provider has a slightly different answer. A message might contain plain text, hidden reasoning, a tool call, a tool result, or some combination of them. Tool arguments might be an object in one source and a JSON string in another. Some datasets have stable IDs, others only have a position in a file. All of this is manageable when you are looking at ten examples. It is less fun when a quiet conversion bug can damage thousands of them.
SkyeEngine now has a single, canonical representation for conversations. Messages, analysis, final replies, tool calls, tool results, tool definitions and their provenance all have an explicit place. More importantly, individual assistant turns have stable identities. A judge can point at the exact turn and channel containing a defect, and a repair can change that exact text without guessing which array index someone meant.
There are structural checks for the fun things one eventually discovers in real data: duplicate calls, missing tools, orphaned results, replies arriving in the middle of a tool exchange, and traces which simply end before anything useful happens. Not glamorous, but neither is discovering after training that your model learned from broken tool calls.
Inference Jobs
SkyeEngine has supported multiple inference backends for a long time. The new work makes those backends behave like parts of the same system instead of a menu of unrelated API clients.
A request can run immediately, become a durable job, or be split into a remote batch. Its attempts, results and token usage are recorded in the same shape regardless of which provider handled it. The job state can remain under my control while the actual inference runs on rented accelerator capacity. A process can stop, another can inspect the job later, and completed batch shards can be fetched without pretending the whole batch succeeded when half of it is still somewhere in a queue.
This is handy for ordinary inference and essential for data generation. Producing a corpus means running a long chain of requests, retries, validation, selection and merging. I don’t want all of that state to vanish because one Python process stopped.
Keeping Track of the Data
That leads to artifact lineage, which is a fancy way of saying I want to be able to answer a very basic question: “What produced this file?”
Every important stage can now record its inputs, outputs, configuration, code version and parent stages. The files are fingerprinted, so SkyeEngine can tell the difference between work that is genuinely reusable and an output that merely happens to have the right filename. If an input, setting, implementation or parent artifact changes, the old result is not quietly treated as current.
I keep the earlier artifacts intact. A repaired conversation is useful, but so are the defective version and the judgment explaining what was wrong. I can later use that history as evaluation data, preference data or a test for the next judge.
Reproducibility isn’t a particularly exciting word until the alternative is regenerating several hundred thousand samples.
Judging the Corpus
One of the largest additions is structured evaluation. A project supplies its own rubric and defect categories; SkyeEngine handles the common machinery around asking a judge, parsing its verdict, tying findings back to exact messages, and storing the result.
I can also compare judges and measure how often they agree. A confident-looking JSON object doesn’t make a judgment correct. Two judges can agree that something is wrong while disagreeing entirely about what is wrong, and they can still share the same blind spot.
For disagreements, SkyeEngine can build a blinded docket for another adjudication pass. The arbiter sees the competing decisions without being told which judge produced which one. There is no reason to give a model a prestigious name to agree with.
This has already changed how I look at the GLaDOS corpus. Quality is no longer “the generator completed without an error.” It is something measured, challenged, repaired and measured again. There will be a separate post about that process because the numbers deserve more context than I can sensibly squeeze in here.
Repairing Individual Problems
Finding a defect is only half the job. The tempting solution is to send the entire conversation back through a model and ask it to make things better. Unfortunately, “better” is an impressively vague instruction. The model may fix the reported sentence while changing three correct ones, losing a tool call, inflating the reply, or adding a completely new problem.
SkyeEngine’s rewrite workflow works with localized patches instead. A patch identifies the conversation, assistant turn, channel and exact old text it expects to replace. It records where the request came from, which judgment triggered it, what the model returned, and how much the result drifted from the source.
Before anything is applied, the patch is validated. Stale source text, overlapping edits, structural damage and suspiciously large changes are refused or moved into review. The original corpus is never modified in place; a new derived layer is produced with its own lineage.
The process is much closer to changing code than asking a language model to wave its hands over a directory.
Preparing a Training Run
The training side has had the same treatment. Dataset transformations are deterministic and publish atomically. Rendering is model-aware, so the exact chat template and the placement of reasoning or tool channels are checked before a run begins. Checkpoints, evaluation results and model artifacts have explicit relationships instead of relying on whatever directory name seemed sensible at two in the morning.
There are reusable controls for stopping, resuming and inspecting a training lifecycle, plus telemetry for the parts of a mixture-of-experts model that ordinary loss graphs do not explain. The environment itself is portable too: the same CUDA container can take a project at runtime and run on different GPU providers without baking the project, its credentials or its data into the image.
None of that guarantees a good model. It does remove a large category of avoidable ways to produce a bad one.
The Failed 120B Run
I know this because I tried the other approach first.
I took the old corpus—the one built with the old SkyeEngine for training the 14B personality chatbot—and used it to train the new 120B model. The model was vastly more capable, so surely giving it the data that had already produced a recognisable GLaDOS would give me a better GLaDOS. That was the theory, anyway.
It was a spectacular failure.
The samples from checkpoint 100 are an excellent record of what went wrong. The identity test technically passed: none of the 28 held-out prompts leaked ChatGPT or OpenAI. If I had looked only at that green check mark, I might even have called the run promising.
Actually reading the answers told a different story. Old template markers were still embedded in prompts. The personality had collapsed into repetitive, indiscriminate contempt. Straightforward coding requests were refused instead of answered. Capability questions became evasive monologues about testing.
The 120B model had not fixed the limitations of the old data. It had learned them with considerably more capacity.
After reading those samples, I stopped the run and went back to the corpus. GLaDOS still has to be capable, accurate, safe when the situation demands it, and able to tell the difference between a moment for character and a moment where somebody actually needs help. Adding tools and coding to the old data would only have given her more ways to fail.
Where It Is Now
SkyeEngine is now the machinery underneath my GLaDOS work: data comes in, is normalized and traced through generation, evaluation and repair, then prepared for training and tested again on the other side. The project-specific personality, rubrics and data policy remain with GLaDOS; the reusable mechanics belong in the engine.
The recent work has taken SkyeEngine from a framework containing several useful AI tools to something I can use for the whole path from raw conversations to a reproducible training artifact. There is still a lot to do, especially around moving the remaining GLaDOS scripts onto the new pieces.
I abandoned the earlier 120B run. Keeping it going wouldn’t have made the corpus any better, and I won’t start the next one until the new data is ready. This is slightly less exciting than announcing that a huge model is training, but it should produce a considerably better result.
I’ll write about the corpus, the judges, and some of the rather surprising measurements next. GLaDOS will be able to reason across multiple turns and use tools, which means teaching her when and how to use them is at least as important as teaching her what to say afterward.
For now, SkyeEngine feels far less like a directory full of useful scripts. There is still plenty to migrate, but I can finally follow a conversation from its source, through generation and review, into a training artifact without relying on my memory of which script produced which directory. That is a rather better place to start the next run from.