Experiment · Paper 1 · Study 5
EXP-004 — Held-out repair
On held-out canonical OPBR scenarios, behavior contracts achieve F1 = 0.982 against the best non-contract baseline at F1 = 0.400. ΔF1 = 0.582. McNemar paired tests against every baseline, Holm-corrected, p ≈ 0.
Preregistered hypothesis
On a held-out partition of OPBR scenarios across four domains, behavior contracts will detect output-passing behavioral regressions with F1 ≥ 0.90, and will exceed the best non-contract baseline by ΔF1 ≥ 0.30 with paired McNemar p < 0.01 (Holm-corrected across the baseline family).
Method
400 runs across the four OPBR domains, drawn from the held-out partition not used during contract development. Each scenario is paired regression-vs-baseline. Baselines: output-only field check, rich-output provenance, LLM-judge trajectory grader, operational composite, output+operational composite. Contracts are evaluated against the same traces. F1 is the primary metric; McNemar paired tests run on per-run agreement matrices.
Result
Contracts: F1 = 0.982 · precision = 0.964 · recall = 1.000. Best baseline (rich-output provenance): F1 = 0.400. ΔF1 = 0.582. McNemar paired test contracts vs every baseline, Holm-corrected family-wise α = 0.01: p ≈ 0 on all five comparisons.
The inversion relative to Study 2 (where rich-output provenance achieved F1 = 0.890) is mechanistic. Under semi-blind perturbation, output quality degrades alongside behavior, so an output-richness detector catches correlated degradation. On canonical OPBR (this experiment), the agent fills every output field correctly while skipping a precondition. Output evaluation has no signal. Only the trace-level check has signal.
What this experiment does not establish
Contracts dominate baselines on this held-out partition with this model, this runtime, and this domain set. The result does not establish dominance under all perturbation regimes (see EXP-002 for the inverted regime), nor on tasks outside the four OPBR domains, nor on runtimes other than Claude Code (Codex replication pending in EXP-007).
Replication
abw run --experiment EXP-004
Requires Agent Behavior Workbench ≥ 0.4. Corpus auto-downloads from OPBR-Bench v0 held-out partition. Full protocol freeze at exp-004-protocol.json.
Artifacts
Every load-bearing claim on this page traces back to one of the artifacts below. They are the canonical citation targets — not the prose.
exp-004-protocol.json— frozen preregistration: hypothesis, falsifier, method, detectors, primary metric, preregistered gates, claim boundary.exp-004-results.json— machine-readable result block: F1 = 0.982 for contracts, F1 per baseline, McNemar Holm-corrected p-values, gate-pass record, mechanism interpretation.- Agent Behavior Workbench — open-source code and OPBR-Bench v0 corpus. Required to run
abw run --experiment EXP-004. - Paper 1 · Study 5 — synthesis layer that bundles this experiment with EXP-001, EXP-002, EXP-003, EXP-005 into the venue submission.
Cite as: Sanchez de la Sierra, A. (2026). EXP-004 — Held-out repair. Raising Agents Lab. https://raisingagents.is/lab/exp/exp-004
Related
- Paper 1: Output-Passing Behavioral Regressions · Study 5
- Inverted regime: EXP-002 — Semi-blind perturbation (rich-output wins when output degrades alongside behavior)
- Confirmatory: EXP-006 — Confirmatory E1+E2 (Paper 2 v1.1)