We probably invalidated our own pre-registered study

TL;DR: We bet the company's skill design on typed artifacts instead of personas, and in July we wrote the test down before collecting any data. Then we found that the condition the test depends on, running before any prompt tuning, probably lapsed in August while we shipped. There are three honest options and one dishonest one, and we're picking in public.

The bet we never tested

Every StratOS skill is a typed artifact: an output contract, hard rules, a statement of when to use it and when not to, a shape the server owns. None of them opens with a persona. A July audit found forty skills and zero persona preambles.

That's a founding bet, made early and never tested.

So in July we wrote the test down before collecting data. The question: does framing a skill as a typed artifact produce more consistent and better-scoped output than framing it as a role, at quality that isn't worse?

Three arms, swapped at the single seam where the prompt gets assembled. Same skill, same input, same template, same model, same temperature. Pure artifact, which is the product today. The contract plus a calibrated light voice paragraph. And role-led, a persona prompt written to the best current standard and adversarially reviewed, so it isn't a strawman. That third arm doesn't exist in the product at all. We'd be building our own counterfactual.

Four hypotheses, directions committed in advance. Consistency: variance across repeated runs lower under artifact framing. Scoping: boundary violations lower under artifact framing, where a violation means a protected framework name, an internal leak, or a section outside the contract. Quality non-inferiority, margin fixed before data. And an exploratory one: is there a voice optimum sitting on top of the contract, or does any persona cost consistency?

Why the headline needs no judge

Our quality judge is anchored on one person's taste. That's right for an early product where the founder is the standard, and useless as an objective instrument.

The failure we designed against is simple. An instrument built by the same taste that produced the output can't return a surprising answer. The fix is placement, not caution: the headline rides metrics that need no judge.

Consistency is arithmetic over replays. A boundary violation is a deterministic check anyone can rerun with the released prompts and validators. Those two carry the publishable result. Quality is secondary and gated on external raters, founders and advisors in the segment scoring a blind sample, because "consistent and scoped but worse" would matter and our own judge couldn't find it.

The conflict of interest is written into the protocol: the team that chose artifact framing is running the test. The mitigations are the design. Frozen hypotheses and analysis, a strong role arm, full release of data and code, external raters, and a binding commitment to publish whichever way it falls.

The condition that lapsed

The protocol sequences the study before any prompt tuning, because tuning confounds it. A tuned artifact prompt against an untuned persona prompt measures the tuning, not the framing.

The trigger was the kickoff of our skill-tuning program. That kickoff never fired. Prompt and harness changes ran continuously through August anyway, because the product had to ship. So the condition in the frozen protocol may already be unsatisfiable as written, and running the study next month doesn't fix that.

A protocol gated on an internal event is only as good as the roadmap's habit of producing that event, and ours quietly stopped. The fix isn't better planning. Date the lapse, name it, choose from what remains.

Choosing in public

Three honest options. Run it on today's harness and state in the write-up that the condition wasn't met. Amend the pre-registration with a dated amendment against a frozen snapshot of the pre-tuning harness, which exists in version control. Or retire the study and register what replaces it.

One option damages the paper: run it and not mention the condition. A pre-registration is worth exactly its refusal to be adjusted quietly after the fact.

We have an opinion about the result, built from a year of watching artifact-framed skills hold rules under pressure, and we're not stating it; a direction announced after the design is the thing the pre-registration forbids. The choice about the condition is different. That one has to be made in the open, so here it is.

We take the second option. The study runs against the frozen August snapshot of the harness, the one in version control from before the continuous changes began, with a dated amendment on page one that says why. The first option would compare an artifact prompt with months of tuning against a persona prompt with none and call the difference framing. The third throws away a written test for a condition we can still meet. The amendment names the snapshot by its commit, records the date the condition lapsed, and is published before a single output is generated.

If the snapshot turns out not to run cleanly, the paper is this post, the frozen protocol and the reason, and that is also a result.

This is a different study from the pricing null we reported in The judge preferred the fabricated numbers. That one tested a compiled standard against a bare model and has closed. This one tests how a skill is framed, and has not started.