The judge preferred the fabricated numbers

TL;DR: A system that can move its own target will discover that moving the target is the cheapest improvement. So we made the standard immutable and the skill replaceable. That separation, not a better optimizer, is the whole thing.

A year ago we set out to make our strategy skills improve themselves. Collect the founder's judgments, train a judge to predict them, let an optimizer search for prompts the judge scores higher, ship the winner. Every piece of that exists in the literature and most of it in open source.

We ran it, measured it, and stopped. Not because the judge's score failed to move. Because of what it did to the thing the judge was supposed to measure.

What the judge learned

We qualified the judge on matched pairs, one variant with fabricated figures woven in at matched form. On five of six paired probes, the judge preferred the fabricated variant. It had learned that confident numbers read like expertise, which is true, and it had no way to know the numbers were invented.

Never qualify a judge on good examples alone. Qualify it on matched pairs where one side carries a planted defect. If it can't see the defect you planted, it can't see the ones you didn't.

The worse finding was already in production. Our unsourced-figure validator fired on twenty-eight of thirty outputs an expert had marked perfect: one hundred percent sensitivity, seven percent specificity. It was measuring numeric density, not fabrication, and it punished exactly the outputs it existed to protect.

The trap underneath both is the same. The scalable instrument is a proxy for the expert. An optimizer searching against a proxy finds the proxy's blind spots faster than it finds the expert's preferences, and it reports success the whole way. Nothing in the loop can tell you, because the only thing in the loop that could is the proxy.

The one-line constraint

Two objects. The standard, meaning what good means, is owned by a human and immutable: superseded by a new version, never amended by a machine. The skill, meaning how a model reaches the standard, is owned by the machine and replaceable. The loop improves the skill against a fixed standard.

The ungoverned alternative is what every self-improving system actually computes: improve the skill and the standard together, with the objective inside the search.

Two corollaries, both enforced. A comparative instrument may say B is closer than A, never that either has arrived, because an ordering can't establish a threshold. And the evaluator is excluded from what the optimizer may edit, because a search allowed to touch its scorer will touch the scorer.

The human stays in one place: authoring and superseding the standard. Not reviewing every output.

What it measured

The pilot was one expert, two skills, three arms. A was a strong generic prompt. B was the bare skill. C was the skill plus the expert's compiled standard, nothing else changed.

On pricing across seven hard contexts, C had the best worst case and zero catastrophic failures; the bare skill had two. Across thirteen contexts, every one where the arms disagreed favored C over bare prose, four to none.

What we don't claim: no test reached conventional significance. One rater, two skills, a pilot sized to design the confirmatory study. The compiled skill doesn't out-predict a strong holistic model. The claim is reliability on hard cases, not accuracy on average.

The next study closed as a null, and the close report is public. It asked a different question from the artifact-versus-persona test we pre-registered in July, which has not run; this one tested a compiled standard against a bare model on pricing. 444 outputs, seventeen independent clusters, a pre-registered failure protocol. The primary comparison came in at plus seven points with an interval of minus seven to plus twenty-one. Standard versus bare model: exactly zero. Per protocol, nothing was repaired or rerun.

Inside the null there's a shape worth keeping. On the decision itself, the evidence-carrying arms gained. On rule coverage, the standard gained. On restraint, the standard lost by twenty-three points. Compilation kept what to say and lost when not to say it. That's the property the next version has to carry.

What it became

We stopped calling it self-improvement and called it what it is: an expert standard as a governed object. It's now a standalone open-source tool, Atelier. It reads an expert's examples, proposes the rules behind them with evidence, lets the expert approve the standard once, then checks every output against it. The standard is hashed and versioned, and no model update, optimizer or judge can move it. An invented story, quote or figure is cut, never reworded.

Same model, same tasks, measured in the README: required rules held in full on seven of nine outside posts instead of two of nine. Six-word runs copied from the author's own pieces fell from ninety-seven per piece to under four. Invented claims went from shipping as written to being cut.

Inside StratOS, the same separation is why the record works. The founder commits a position; the system can't amend it, only propose a successor. The skills are replaceable. The standard is yours.

Self-improvement without ground truth is possible. It just can't include the ground.