# INTERPRETATION — EXP-009

**Status:** complete. Pre-registered SHA: `d8e4738ec1efc9f1b082139eb3a0c4ad6445f8b254014f72b6bffcd805c0c44d`
**Predecessor:** EXP-008 (`af019866…30b6f92`).
**Result:** partial. The v0.6 narration runtime closes the procedural-compliance gap that EXP-008 exposed but does not close the framework-fidelity gap. A different fix is needed for the second gap.

---

## Headline

The skills2 v0.6 runtime — every `@step` returns `(output, narration)`,
narrations concatenate into `response_text` — lifted procedural compliance on
the growth-accounting skill from **3% to 100%** vs. its predecessor v0.5, and
mean framework fidelity from 2.83 to **3.06**. Median framework fidelity
remains tied with the prose-plus-checklist arm (A2). The pre-registered
median-delta gates still fail.

| Arm | n | Median FF | Mean FF | PC yes-rate |
|---|---|---|---|---|
| A1 — prose only | 35 | 3 | 3.23 | 26% |
| A2 — prose + checklist | 35 | 3 | 3.26 | 83% |
| **A3 — skills2 v0.6** | 35 | **3** | **3.06** | **100%** |

EXP-008 same skill, same cases, v0.5: A3 mean FF = 2.83, PC = 3%.
EXP-009 v0.6 → mean +0.23, PC +97 percentage points.

## What v0.6 fixed

The procedural-compliance collapse. EXP-008 showed A3 firing 347/350 step
calls in telemetry while the judge saw procedural compliance "yes" on
**only 1 of 35 cases**. v0.6's `render_narrations()` concatenates the per-step
narration strings produced by each `@step` body into `response_text` — what
the judge actually reads. Result:

- 40/40 cases now score "yes" on procedural compliance (vs. 1/40 in v0.5).
- All ten ordered steps surface in the response, including the framework's
  hard invariants (identity check passes/fails, quick ratio formula, dominant
  flow named with magnitude).
- Pre-registered gate G_pc_a3 (threshold ≥ 0.90) **passes at 1.00** — the
  largest single mechanism improvement in the entire EXP-008/EXP-009 program.

## What v0.6 did not fix

The mean framework fidelity gap to A2. Mean FF went 2.83 → 3.06, but A2 is
3.26. The judge's per-case rationales are explicit on the ceiling:

> Per-flow diagnosis is always a generic template ("Retention is the
> foundation. Shift focus to growth loops if the retained base is stable")
> regardless of the actual case scenario. No case-specific matched remedy
> (no "Big New + thin Retained → fix activation before scaling acquisition",
> no resurrection-as-vanity warning, no Contraction-as-leading-indicator
> callout). The aggregate MAU masking motivation is never stated. The
> resurrection warning is systematically absent.

This is a consequence of the v0.6 design. The `run_per_flow_diagnosis` step's
prescription text comes from a hard-coded `_INPUT_LEVER_MAP` keyed by flow
name. Every "retained" dominant flow gets the same prescription text. Every
"churned" flow gets the same prescription text. The runtime is delivering on
its contract — it always says the right *category* of thing — but the
deliveries are generic by construction. The judge is correctly marking
generic prescriptions as score-3 rather than score-4. There is no rubric
disagreement here; the runtime is genuinely the limit.

Score distribution corroborates: 33 of 40 cases scored exactly 3, only 7
scored 4 (the 5 negative controls + 2 deliberate broken-data cases that the
identity check correctly halted). Standard deviation 0.24 — the runtime is
delivering identical-shape responses with no case-adaptive variance.

## What this means for skills2

The original Skills 2.0 architecture conflated two jobs:

1. **Structural compliance** — does the right procedure fire, in the right
   order, with the right invariants? EXP-009 shows: v0.6 solves this. The
   contract enforcement works.
2. **Case-specific elaboration** — does the response apply the framework to
   *this* user's data in a way that beats a free-form LLM answering with
   only the skill's prose? EXP-008 said no; EXP-009 says still no.

Hard-coding the elaboration into Python step bodies (the v0.6 path) makes
the elaboration uniform and that ceiling-caps it. A2 wins because the
prompt-engineered checklist gives the LLM the structure AND lets it adapt
the elaboration to the case.

The next move is a **hybrid runtime** — v0.7:

- Runtime fires all steps and produces telemetry + narrations as v0.6 does
  (the structural contract is non-negotiable).
- After the steps complete, the LLM is given the rendered narration plus the
  user's case and asked to *elaborate* the per-flow prescription to the
  specific case before the final response is written.
- The contract still enforces what *must* be said (identity check, quick
  ratio, dominant flow). The elaboration is what A2 already produces well.

EXP-010 would retest under v0.7. The hypothesis is straightforward: if A3 in
v0.7 = "runtime structure + LLM case elaboration", then A3 should beat A2
because A2 has no structural guarantee and v0.7 has both.

## Pre-registered gates

| Gate | Threshold | EXP-009 observed | Verdict |
|---|---|---|---|
| `G_ff_lift_a3_vs_a2` | Δ median ≥ +0.50 | 0.00 | ❌ |
| `G_ff_lift_a3_vs_a1` | Δ median ≥ +0.80 | 0.00 | ❌ |
| `G_pc_a3` | ≥ 0.90 | 1.00 | ✅ |
| `G_kappa_self` | ≥ 0.75 | 1.00 | ✅ |

The framework-fidelity gates still fail. The pre-registered falsifier ("A3 ≤ A2
on median OR Δ_median < 0.20") fires on the first clause. EXP-009 is itself a
negative result on the original FF claim, but a *positive* result on the v0.6
narration mechanism specifically.

## Cumulative finding across EXP-008 + EXP-009

We have ruled out one explanation and confirmed another:

- **Ruled out:** that runtime contracts can't produce procedural compliance.
  EXP-009 v0.6 produces it perfectly (100%, σ=0).
- **Confirmed:** that A2 (prose + checklist) is a *strong* baseline. Beating it
  on framework fidelity requires *more* than structural enforcement — it
  requires case-specific reasoning that no Python step body can match without
  delegating back to the LLM.

The skills2 v1.0 product becomes:
- **Transformer** → produces manifest + step inventory. Already valuable.
- **Runtime** → enforces structure, surfaces it in prose via narration. Now
  proven (EXP-009).
- **Hybrid elaborator** → LLM expands the runtime narration with case
  specifics before the response ships. To-be-built and tested in EXP-010.

## Limits of the claim

- Single skill (growth-accounting). The v0.6 narration update must be applied
  to psych-framework and abw-design before generalizing. They remain at v0.5
  stubs and would also need the hybrid elaborator.
- Judge is an LLM (Sonnet 4.6 subagent), self-consistency κ = 1.000, but
  human-rater agreement is not established. A small human anchor pass on
  a stratified sample would tighten the claim.
- A1 + A2 traces reused from EXP-008 (same cases, same response_text). A3 is
  fresh under v0.6. Apples-to-apples within the same case set.
- Production model for A3 = `skills2-v0.6-runtime` (Python execution + Sonnet
  4.6 subagent harness). A1 + A2 = mix of claude-sonnet-4-6 (initial) and
  codex-gpt-5.4-mini (partial) and claude-sonnet-4-6-subagent (fillers).
  Cross-model confound documented; the within-experiment comparison is
  controlled because A1/A2 reused from one run.

## Pairs with

- **EXP-008** (`lab/exp/exp-008/INTERPRETATION.md`) — the falsifying experiment
  that motivated v0.6.
- **EXP-010** (forthcoming) — hybrid v0.7 runtime + LLM elaborator,
  re-running the same arms.
- **BW-2026-006** — Behavior Watch case-file. Now becomes a *progress* report:
  "We falsified the original claim, identified the mechanism, fixed it,
  partially closed the gap, named the next gap." More valuable than a
  single-positive case-file.
- **Paper 3.5** — academic writeup. Title shifts again:
  *"Decomposing Skill-Runtime Lifts: Structural Compliance vs. Case-Specific
  Elaboration in LLM Skill Contracts."*

## Artifacts

- `analysis/gate-report.json` — full statistical detail
- `analysis/summary.md` — formatted table
- `raw/growth-accounting/a3-skills2/` — 40 fresh v0.6 traces
- `raw/growth-accounting/{a1-prose,a2-prose-checklist}/` — 80 reused from EXP-008
- `ratings/llm-judge/` + `ratings/llm-judge-consistency/` — 240 ratings
- Pre-registration hash: `d8e4738e…c0c44d`