Heuristic · Paper 1 · Study 7
EXP-005 — Completeness analyzer
A specification audit, not an empirical run. The analyzer enumerates the consequential tools present in the empirical traces from Study 1 and Study 2 and reports which ones are missing from the corresponding contract's required_tools. The Study 2 recall gap is enumerable specification work, not a flaw in the contract mechanism.
Why this is heuristic, not empirical
This experiment does not draw new runs from a perturbation distribution. It runs a static analyzer over the traces and contracts already produced by EXP-001. The analyzer enumerates every tool call in the corpus, flags the ones whose effect type is consequential under the OPBR ontology, and reports which of those tools are absent from the contract's required_tools or required_successful_order sections. The output is a specification gap list, not a hypothesis test.
Method
For each contract C and each Study 1 trace T: enumerate the consequential tool calls in T, intersect with the consequential-tools set declared in C, and emit a diff. Frequency-rank the unspecified consequential tools across the full trace corpus. A consequential tool is one whose call commits a state-changing effect, claim, memory write, handoff, escalation, or external publication.
Result
No load-bearing consequential action is missing from the Study 1 contracts. The high-frequency unspecified tools that account for the Study 2 recall gap of 0.214 are enumerable. The improvement path is deterministic: observe traces, identify high-frequency unspecified tools, review each for consequential status, add qualifying entries to the contract, re-evaluate. Study 5 demonstrates the endpoint of that path: with complete contracts over the target failure modes, recall reaches 1.000.
The reframing is the load-bearing finding: the Study 2 recall gap is not evidence that contracts fundamentally miss failure modes. It is evidence that specification is an iterative engineering task. The mechanism is sound; the artifact is incomplete.
What this experiment does not establish
This is a heuristic, not a hypothesis test. It does not establish that all future specification gaps will be enumerable in the same way. It does not establish that every consequential tool can be statically classified — the consequential-status review still requires human judgment per tool. It does not test the analyzer on contracts other than Study 1's.
Replication
abw run --experiment EXP-005
Static analysis. No model calls required. Reads the Study 1 trace corpus and contract set from OPBR-Bench v0 and emits the gap report as JSON.
Artifacts
Every load-bearing claim on this page traces back to one of the artifacts below. They are the canonical citation targets — not the prose.
exp-005-protocol.json— frozen preregistration: hypothesis, falsifier, method, detectors, primary metric, preregistered gates, claim boundary.exp-005-results.json— machine-readable result block: primary metric, gate-pass record, mechanism interpretation.- Agent Behavior Workbench — open-source code and OPBR-Bench v0 corpus. Required to run
abw run --experiment EXP-005. - Paper 1 · Study 7 (heuristic) — Synthesis layer that bundles the experiments and the specification audit into the venue submission.
Cite as: Sanchez de la Sierra, A. (2026). EXP-005 — Completeness analyzer. Raising Agents Lab. https://raisingagents.is/lab/exp/exp-005
Related
- Paper 1: Output-Passing Behavioral Regressions · Study 7 (heuristic)
- Subject: EXP-001 — Controlled mechanism (the contracts audited here)
- Endpoint: EXP-004 — Held-out repair (recall = 1.000 with complete contracts)