Empirical · 5 studies executed · 3,797 runs · preregistered

Output-Passing Behavioral Regressions in AI Agents

Detection, Baselines, and Repair with Agent Behavior Contracts

Type: Empirical paper · multi-study
Status: v4 complete · Studies 6 + 8 executing
Studies: 5 executed · Studies 6 & 8 preregistered & in progress6 is engineering. 8 is access. both solvable.
Runs: 3,797 (to date)
Target venue: ICLR 2027 (primary) · AAAI 2027 (backup)
Preregistration: PAPER1_PROTOCOL_FREEZE.json
Author: Adrian Sanchez de la Sierra, Genesis Rojas, and Radu Simonescu
Last edited: 2026-05-21

Final-output evaluation is necessary for AI agents but insufficient when agents take consequential actionscommit, write, send, delete. We define an output-passing behavioral regression as a behavior change that preserves selected final-output correctness while violating a precondition over a consequential trajectory event. An agent can return the correct final answer while committing a refund before checking policy, asserting a research claim before checking conflicting evidence, writing memory before resolving conflict, or applying a code edit before reading the failing test. Output-only evaluation does not see the failure. Exact path matching is too brittle to detect it without overflagging useful variation. Behavior contracts — executable specifications over consequential trajectory eventsnot a rubric. not a judge. a type. — detect the failure preciselythe spec is what makes this testable, tolerate safe path variation, and enable measurable repair through typed action-space redesign.

Canonical case · Study 5 · n=400 Interactive demo →

Full manuscript (Markdown) → PDF (in prep) Workbench & reproduction

Thesis

Do not make agents deterministic. Make consequential behavior contract-bound.

A final answer can be correct while a consequential action commits too early. Output-only evaluation cannot see this. Exact path matching is too brittle to detect it without rejecting useful variation. Behavior contracts occupy the middle layer: they specify what must be true before a consequential action counts as acceptable success, without prescribing the path the agent takes to get there.

The process integrity problem

The natural engineering response is: extend the output schema. Require the agent to self-certify which preconditions it met. Add fraud_check_passed, policy_version_checked, evidence_reviewed to the output JSON. Then the output evaluator catches the regression when those fields are missing or false.

This works when agent behavior is reliable enough that self-reporting is honest. It fails in exactly the cases where you need it most: when the agent is competent enough to produce correct-looking outputs through an incorrect process. Under EU AI Act Article 9, what the regulator requires is not an accurate field. It is a tool call log showing that the required tool was invoked and returned before the consequential action executed. The output evaluator cannot detect this. It trusts the field. The behavior contract does not trust the field. It checks the trace.

The full process-integrity argument, including the optimization-drift and compliance-update scenarios, is developed in §2 of the manuscript.

Domain	Output pass	Behavior pass	Order failures	Operational quality
Policy exception	60/60	0/60	60	60/60
Qualified claim	60/60	0/60	60	60/60
Memory revision	60/60	0/60	60	60/60
Total (induced drift)	180/180	0/180	180	180/180

Every induced drift run passes selected final-output field checks. Every induced drift run fails the behavior contract. The drift paths are also operationally clean — short, error-free, retry-free. A naive combination of output evaluator and operational budgets, applied jointly, still passes 180/180. Only the behavior layer separates them.

Runtime control blocks unsafe successful effects 180/180 (with operational churn 7/180). Harness repair restores contract-satisfying commit structure 180/180 (with policy operational caveat 14/60). Safe variation passes 177/180 — exact path matching would fail all 180. Contracts tolerate path variation; exact path imitation does not.

Study 2 — Semi-blind perturbation · executed · 1,437 runs

Under non-instructed perturbations, 91.4% of behavioral regressions pass outputswent in expecting ~80%. found 91. changed the section title.

Four domains (policy, claim, memory, code), six conditions (reference, latency-pressure, prompt-compression, tool-semantics-ambiguity, evidence-noise, safe-extra-check), three cases per domain, 20 runs per cell. Agent-visible prompts contain none of the forbidden patterns from the locked preregistration ("commit before verify", "fast lane", "behavior contract"). The agent is never told what failure mode is being tested.

Gate A — θ_OPBR: 877 of 960 behavioral regressions (91.4%) produce passing outputs. Wilson CI [0.894, 0.932]. Gate A pass with large margin across all four domains.

Detector	Precision	Recall	F1	FPR
output_only_fields	0.741	0.086	0.155	0.122
output_rich_provenance	0.802	1.000	0.890	1.000
exact_path	0.801	0.954	0.871	0.962
unordered_required_tools	0.865	0.214	0.343	0.135
fingerprint_distance	0.750	0.053	0.099	0.072
agent_behavior_contract	0.801	0.214	0.337	0.215

Rich-output provenance achieves the highest F1 here — by catching correlated output degradation, not the behavioral failure directly. This relationship inverts in Study 5.

The mechanism is correlated degradation: under latency pressure, an agent that skips the policy check also fills fewer provenance fields and produces sparser outputs. The rich-output detector catches the correlated output degradation, not the behavioral failure directly.

Contract recall is 0.214. The recall gap is not undefined: the completeness analyzer identifies six high-frequency consequential tools present in empirical traces but absent from the locked contracts. This is a specification completeness gap, not a mechanism failure. See §9.5 of the manuscript for the calibration methodology.

Study 4 — Action-affordance merge · executed · 960 runs

Collapse the action-space split, and every structural detector degrades together

Same four domains, with the proposal/commit pair collapsed into a single ambiguous tool (e.g., decision_record replacing refund_proposal + refund_commit). Contract recall = 0.331 under affordance-merge. All structural detectors also degrade (FPR ≈ 1.000 for rich-output, exact-path, operational-only). The architectural claim holds: when the action-space split is collapsed, all structural detectors degrade together. The action-space is the load-bearing unit, not the prompt.

Study 5 — Held-out repair · executed · 400 runs

On the canonical case, contracts hit F1 = 0.982held-out. n=400. no contamination.; the best baseline gets 0.400

Four held-out cases (REF-206, QCLAIM-033, MREV-016, CODE-303) locked before execution. Canonical OPBR: the agent returns a correct output while skipping a required precondition.

Detector	Precision	Recall	F1	FPR
output_only_fields	0.250	0.013	0.024	0.013
output_rich_provenance	0.250	1.000	0.400	1.000
exact_path	0.250	1.000	0.400	1.000
fingerprint_distance	0.221	0.537	0.313	0.633
agent_behavior_contract	0.964	1.000	0.982	0.013

n = 400 held-out OPBR scenarios · McNemar contracts vs. every baseline, Holm-corrected p ≈ 0.

Gate F: Δ_F1 = 0.982 − 0.400 = 0.582 ≥ 0.50. Gate F pass. McNemar agent_behavior_contract vs. every baseline: Holm-corrected p ≈ 0 in all 8 contrasts.

On held-out OPBR scenarios, rich-output provenance fails completely (FPR = 1.000). The agent fills all output fields correctly, so provenance evaluation has no signal. This is the canonical case that defines OPBR; contracts detect it; output schemas cannot.

Why rich-output wins in Study 2 and loses in Study 5the best section in this paper. the inversion is the argument.

Semi-blind perturbations degrade output quality alongside behavior. A rich-output detector that catches the correlated output degradation is detecting noise, not behavioral failure. Its Study 2 F1 = 0.890 is real, but the mechanism is wrong. On canonical OPBR (Study 5), the agent fills all output fields correctly while skipping a precondition. Output evaluation has no signal. Only the trace-level check has signal.

Rich-output provenance is a useful noise detector but a poor contract detector. The operationally important case — an agent that is behaviorally wrong while being output-correct — is exactly what contracts are designed for and exactly what provenance evaluation misses.

Specification completeness as a calibration methodologythe completeness gap is closable. process mining closes it.

The Study 2 recall gap (0.214) is enumerable. The completeness analyzer identifies the specific consequential tools present in the empirical traces but absent from required_tools:

Domain	Unspecified high-frequency tool	Success fraction
Policy	`bash:customer_history_check`	0.825
Policy	`bash:refund_proposal`	0.897
Memory	`bash:memory_dedupe_check`	0.608
Code	`bash:read_source`	0.857
Code	`bash:propose_edit`	0.798
Claim	`bash:source_relevance_check`	0.261

These tools appear in the majority of successful baseline traces but are not in the locked contracts. The relationship between specification completeness and recall is direct:

Contract recall ≈ fraction of consequential events specified in the contract.

The gap is not a flaw in the mechanism. It is a measurement of specification work remaining. The improvement path is deterministic: observe traces, identify high-frequency unspecified tools, review each for consequential status, add qualifying entries to the contract, re-evaluate. Study 5 demonstrates the endpoint: with complete contracts over the target failure modes, recall reaches 1.000ceiling. complete spec = complete detection. no gaps..

Acceptance gatesGate E still open

Gate	Description	Status
A	Non-tautological emergence (≥3 domains, Wilson lower ≥0.10)	Pass θ_OPBR = 0.914
B	Baseline superiority (contracts Pareto-dominate)	Mixed rich-output complementary in degradation regimes
C	Safe variation discrimination (exact-path FPR − contract FPR ≥ 0.50)	Pass
D	Cross-domain breadth (≥4 domains)	Pass policy, claim, memory, code
E	Cross-runtime (≥2 runtimes)	Pending Study 6, Codex CLI
F	Held-out repair (Δ_F1 ≥ 0.50)	Pass Δ_F1 = 0.582
G	Reproducibility package (OPBR-Bench released)	Partial spec locked, corpus packaging pending
H	Claim boundary (Appendix C present)	Pass

Claim boundary

The paper claims, subject to study execution:

Under induced fast-lane conditions, field-level output evaluation can pass while behavior contracts fail on consequential ordering (Study 1).
Under non-instructed semi-blind perturbations across four domains, 91.4% of behavioral regressions produce passing outputs (Study 2).
On held-out OPBR scenarios, contracts achieve F1 = 0.982 versus best baseline 0.400 (Study 5).
Merging proposal and commit into an ambiguous tool degrades all structural detectors (Study 4).
The completeness analyzer finds no load-bearing missing consequential action in Study 1 contracts (Study 7, heuristic).

The paper does not claim:

Claude Code naturally regresses this way in ordinary deployment.
All AI agents have this failure mode.
Behavior contracts prove correctness.
Hidden reasoning is observable.
Private chain-of-thought is a reliable reasoning trace.
Contracts are automatically inferred.
The workbench is enterprise-ready.
Output evaluation is useless.
Rich-output / LLM-judge baselines are dominated in all regimes (they are not — Study 2 shows the opposite under degradation-heavy perturbations).
Naturalistic prevalence is high (Study 8 protocol locked, execution pending).

Reproduction

Workbench, baselines, statistical scripts, completeness analyzer, semi-blind scaffolds, and adapter specification are open source. Locked artifacts:

locked artifacts · paper 1

docs/PAPER1_BASELINES_PREREGISTRATION.md
docs/PAPER1_STATISTICAL_PLAN.md
docs/PAPER1_LABELING_AND_BASELINE_PROTOCOL.md
docs/PAPER1_SEMIBLIND_PREREGISTRATION.md
docs/OPBR_BENCH_SPECIFICATION.md
docs/PAPER1_TRACE_MINING_PROTOCOL.md
docs/CROSS_RUNTIME_ADAPTER_SPECIFICATION.md
experiments/PAPER1_SEMIBLIND_DESIGN.json
experiments/PAPER1_PROTOCOL_FREEZE.json
  (hash 5e69d50e...026ae2b9)

Full manuscript v4 → Use the workbench OPBR-Bench

Next See the canonical OPBR demonstration →

Research program

Where this goes.three papers from a single refund transaction

Paper 1 establishes the empirical foundation: output-passing behavioral regression exists, is measurable, and is detectable via behavior contracts at F1 = 0.982 vs. F1 = 0.400 for the best baseline. What follows extends the result along three lines.

Studies 6 + 8 — external validity and naturalistic logs

Study 6 (cross-runtime replication). Repeat the Study 5 canonical case using Codex CLI as the agent runtime instead of Claude Code. The adapter specification is locked at docs/CROSS_RUNTIME_ADAPTER_SPECIFICATION.md. Execution is engineering, not new research: import Codex CLI transcripts via the workbench adapter, apply the same locked behavior contract, compare detection rates. The study answers the external-validity objection — "does this only work for Claude Code?" — which is the strongest remaining challenge to the claim boundary.

Study 8 (naturalistic logs). Apply the contract detection pipeline to agent execution logs drawn from real production deployments rather than the controlled OPBR-Bench corpus. Preregistered; execution depends on log access. The result either confirms that θ_OPBR holds in the wild or produces a calibrated estimate of how detection performance changes outside the controlled setting.

Graph formulation — the structural lift

The current workbench represents agent trajectories as linear sequences of spans. The natural structural extension is a directed graph: spans as nodes, causal/temporal edges, tool-call sub-graphs per LLM turn. This opens three research directions with different timelines.

Process mining for contract induction. The primary extension. Rather than hand-authoring behavior contracts (which caps recall — Study 5 recall = 0.214 is the completeness ceiling), mine contracts from the trace corpus using process discovery algorithms (Inductive Miner, Split Miner). Output is a Petri net or process tree over the trace event log; the contract follows from the discovered model. This closes the completeness loop and is the spine of Paper 3. The OPBR-Bench corpus plus the 755 escaping traces from Study 2 are exactly the input dataset. Key challenge: defining the "case" concept cleanly when sub-agent delegation and parallel tool calls blur trace boundaries.
Cycle detection and absorbing states. Apply Tarjan's strongly connected components over the trace graph to identify cognitive loops — patterns where the agent revisits the same tool sequence without progress. These are detectable cheaply and provide a practical runtime halt heuristic (already gestured at in the manuscript §10.4 runtime control). High value-to-effort ratio; doable before the venue deadline.
Reachability and PCTL verification. Fit a Markov Decision Process over the discovered process model, then apply model checking (PRISM, pctl specifications of the form P_≥θ[ψ]) to verify contract satisfaction properties formally. The PCTL connection is the formal-verification bridge to Paper 2 (Delegation-Grade Agents). Constraint: MDP fitting from agent traces is hard — state explosion, abstraction choice. Honest framing: "model checking after abstraction" with explicit abstraction commitments.

Publication arc

Paper 1 is the empirical lemma. Paper 2 (Delegation-Grade Agents, whitepaper v1) is the conceptual frame. The planned Paper 3 — Contract Induction from Agent Traces via Process Mining — closes the completeness loop by automating what Paper 1 demonstrates manually. Target venues: ICLR 2027 for Paper 1; the graph/process-mining work is NeurIPS/AAAI-grade if the induction results hold.