# INTERPRETATION — EXP-008 **Status:** complete. Pre-registered SHA: `af019866c8c3b87b0e13005062bdcbc6976496f1a11c4f0bd60df575b30b6f92` **Result:** primary hypothesis **falsified**. 5 of 6 pre-registered gates failed. --- ## Headline Skills 2.0 in its current form — `skill.toml` + `@step` decorator + telemetry + post-hook audit — did not lift framework fidelity above either (a) pure prose `SKILL.md` or (b) prose with a step-ID checklist injected, across three Reforge-derived skills judged blind by Claude Sonnet 4.6 with self-consistency check. Pooled across three skills (n = 105 per arm): | Arm | Median FF (0-4) | PC yes-rate | |---|---|---| | A1 — prose only | 3 | 42% | | **A2 — prose + step-ID checklist** | **4** | **92%** | | A3 — skills2 contract runtime | 3 | 54% | The contract runtime (A3) **lost** to the prompt-engineered checklist (A2) by one full median point on framework fidelity and by 38 percentage points on procedural compliance. ## Per-skill | Skill | A1 mean | A2 mean | A3 mean | A3 vs A2 | |---|---|---|---|---| | growth-accounting (fully implemented) | 3.23 | 3.26 | **2.83** | **−0.43** | | psych-framework (stub steps) | 2.00 | 4.00 | 4.00 | 0.00 | | abw-design (stub steps) | 3.97 | 3.89 | **2.63** | **−1.26** | The psych-framework arms A2 and A3 tied at the rubric ceiling (4.00) — both lift massively over A1 (2.00) but the runtime contributes nothing over the checklist. Growth-accounting (the skill whose `@step` bodies are fully hand-implemented, not stubs) was the strongest case for A3 to win. A3 lost there as well, by 0.43 points and by 80 percentage points of procedural compliance. ## Pre-registered gates | Gate | Threshold | Observed | Verdict | |---|---|---|---| | `G_ff_lift_a3_vs_a2` | Δ ≥ +0.50 | **−1.00** | ❌ | | `G_ff_lift_a3_vs_a1` | Δ ≥ +0.80 | 0.00 | ❌ | | `G_pc_a3` | ≥ 0.90 | 0.54 | ❌ | | `G_pc_a1` | ≤ 0.30 | 0.42 | ❌ | | `G_neg_control` | ≤ 0.05 | 1.00 | ❌* | | `G_kappa_self` | ≥ 0.75 | 1.00 | ✅ | *The negative-control gate is reported failing because the analyzer counts "FF ≥ 2" as "activated." On a negative control, a high FF score is actually a correct refusal (response says "this isn't the right framework, use X"). The gate as written is measuring the wrong quantity — the system *did* refuse negative controls correctly. This is a measurement bug in the analyzer, not a failure of the system. The other four failures stand. The pre-registered falsifier ("A3 framework fidelity ≤ A2 OR Δ_median < 0.20 across all three skills") triggered on the first clause. The experiment is **negative**. ## Why A3 lost The judge reads `response_text`. The runtime fires Python `@step` functions that record telemetry but do not by themselves write into the response text. A3 responses therefore have: - thinner narration of the procedure (the model offloads the work to function calls instead of articulating it in prose), - correct structural compliance recorded in `steps.jsonl` that the judge cannot see, - the same final claim as A2 but with less of the framework's vocabulary surfaced — which is exactly what the rubric measures. This is structurally consistent across skills. A3 on growth-accounting: procedural compliance rate of 3% (1/35 cases) despite telemetry showing 347/350 step invocations succeeded. The contract fired; the prose did not. On the two skills whose `@step` bodies are still transformer-generated stubs (psych-framework, abw-design), A3 either ties A2 (psych) or loses dramatically (abw-design −1.26). The stub `@step` returns `{}` and the model is left to generate the prose anyway — but with the procedural attention spent on calling the stub functions rather than writing the analysis. Net effect: worse than just being given the checklist (A2). ## What the experiment *did* prove **Structured prompts beat unstructured prose.** On the soft skill (psych-framework) where A1 scored 2.00 (every case), A2 scored 4.00 (every case) — the largest effect in the experiment. A2 lifted framework fidelity by +2.00 points and procedural compliance from 0% to 100%. Injecting the step-ID checklist into the prompt produces a near-ceiling response. **The skills2 transformer's main artifact (the manifest + step inventory) remains useful — as the input to the A2 checklist prompt.** What the experiment falsifies is the *runtime* — the Python contracts, the `@step` decorator, the post-hook audit. None of those layers contributed measurable value over a well-formatted prompt. ## What changes for skills2 Two paths. **Path A — drop the runtime, ship the transformer-as-prompter.** The transform produces `skill.toml` and a step inventory. Convert that to an A2-style prompt injection layer. Ship Skills 2.0 v1.0 as "structured prompt injection, not contract enforcement." This is honest about what worked. **Path B — fix the runtime so the contract shows up in the response.** Change the `@step` decorator to *require* a `narration` field returned alongside the typed output. The runtime concatenates step narrations into the final response. The judge then sees the procedure surfaced in prose AND knows the contract was enforced. This is the bigger swing but unblocked by the experiment's finding. Both paths are testable. EXP-009 would re-run the same three arms with the v0.6 runtime that includes mandatory step narration. ## Limits of the claim - Three skills, not the full corpus. Selection biased toward Reforge content. - Judge is an LLM (Sonnet 4.6 subagent), not a human expert. κ = 1.00 self-consistency is high, but human-LLM agreement is not established by this run. Human anchor pass deferred to EXP-009. - Production runs were a mix of Claude (initial), Codex / gpt-5.4-mini (partial), and Sonnet 4.6 subagents (the remainder). API budget caps forced the mix. This is a confound — different base models may produce different arm gaps. Pre-registration documented the intended base model (`claude-sonnet-4-6`); the actual mix is logged in each `raw///.json`'s `model` field. - The rubric was authored by the same LLM (claude-sonnet-4-6 via Anthropic API) that generated the cases. Cross-model rubric validation (e.g. by GPT) is deferred. ## Honest weighting The intuition behind Skills 2.0 — that Python contracts could enforce procedural compliance in ways prompts cannot — was reasonable. The experiment shows the runtime as currently designed does not deliver on it. The structured prompt does. The right product move is to ship what works (transformer + checklist injector) and explicitly re-scope the runtime to what it can credibly do (telemetry, audit, replay, eval-gated CI) rather than claim a lift it did not produce. ## Artifacts - `analysis/gate-report.json` — full statistical detail - `analysis/summary.md` — same data, formatted table - `analysis/kappa-report.md` — judge self-consistency - `EVIDENCE_PACK.md` — the canonical lab artifact - `raw/` — 360 production traces - `ratings/llm-judge/` + `ratings/llm-judge-consistency/` — 600 judge ratings - Pre-registration hash: `af019866...30b6f92` ## Pairs with - **BW-2026-006** (forthcoming) — Behavior Watch case-file, Register B, publishing the negative result honestly. - **Paper 3.5** — academic writeup. Title candidate adjusts: "*When Runtime Contracts Fail to Lift LLM Framework Fidelity Over Structured Prompts*." Reviewer-attractive — a published negative result on contract-enforcement narrows the claim space for the OPBR research program.