Experiment · Paper 1 · Study 2
EXP-002 — Semi-blind perturbation
Under non-instructed perturbations across four domains, 91.4% of behavioral regressions produce passing outputs. Output-only evaluation does not make a marginal miss — it misses nearly all behavioral failures.
Preregistered hypothesis
Under perturbations whose names do not appear in the agent's prompt, instructions, or tool descriptions (latency pressure, prompt compression, tool-semantics ambiguity, evidence noise), the proportion of behavioral regressions that produce field-level passing outputs (θ_OPBR) will be ≥ 0.80 with lower Wilson CI bound ≥ 0.75.
Method
Four domains (policy, claim, memory, code). Six conditions per domain (reference, latency-pressure, prompt-compression, tool-semantics-ambiguity, evidence-noise, safe-extra-check). Three cases per domain. 20 runs per cell. 1,437 runs total. The agent-visible prompts contain none of the forbidden patterns from the locked preregistration ("commit before verify", "fast lane", "behavior contract"). The agent is never told what failure mode is being tested. Behavior contracts and baselines are evaluated on the same traces.
Result
θ_OPBR = 0.914. Wilson 95% CI [0.894, 0.932]. Lower bound exceeds the preregistered 0.75 floor by 14.4 percentage points. Hypothesis confirmed.
Baseline ordering inverts relative to canonical OPBR (EXP-004): under semi-blind perturbation, rich-output provenance achieves the highest F1 = 0.890. The mechanism is correlated degradation: under latency pressure, an agent that skips the policy check also fills fewer provenance fields and produces sparser outputs. The rich-output detector catches the correlated output degradation, not the behavioral failure directly. This is a real signal under this regime, with the wrong mechanism. The inversion is the load-bearing finding for the baseline-selection discussion in the paper.
What this experiment does not establish
91.4% is the rate at which behavioral regressions pass output evaluation in this corpus under this perturbation set with this model and runtime. It is not a claim that all output-passing failures in production agents occur at this rate, nor that the four domains generalize to every consequential-action surface. The baseline ranking under this perturbation regime is regime-specific — see EXP-004 for the inverted regime on canonical OPBR.
Replication
abw run --experiment EXP-002
Requires Agent Behavior Workbench ≥ 0.4. Corpus auto-downloads from OPBR-Bench v0. Full protocol freeze at exp-002-protocol.json.
Artifacts
Every load-bearing claim on this page traces back to one of the artifacts below. They are the canonical citation targets — not the prose.
exp-002-protocol.json— frozen preregistration: hypothesis, falsifier, method, detectors, primary metric, preregistered gates, claim boundary.exp-002-results.json— machine-readable result block: primary metric, gate-pass record, mechanism interpretation.- Agent Behavior Workbench — open-source code and OPBR-Bench v0 corpus. Required to run
abw run --experiment EXP-002. - Paper 1 · Study 2 — Synthesis layer that bundles this with EXP-001, EXP-003, EXP-004, EXP-005 into the venue submission.
Cite as: Sanchez de la Sierra, A. (2026). EXP-002 — Semi-blind perturbation. Raising Agents Lab. https://raisingagents.is/lab/exp/exp-002
Related
- Paper 1: Output-Passing Behavioral Regressions · Study 2
- Inverted regime: EXP-004 — Held-out repair
- Mechanism: EXP-001 — Controlled mechanism
- Specification audit: EXP-005 — Completeness analyzer