Papers / OPBR-Bench

Benchmark · v0 corpus packaging in progressGate G. ICLR 2027 waits for this.

Type

Corpus · benchmark

Version

v0 · packaging in progress

Domains

4 · policy, claim, memory, code

Scenarios

12 cases · 5 conditions each60 cases. not 600. precision over coverage.

Schema

docs/OPBR_BENCH_SPECIFICATION.md

Tracks

Paper 1 Gate G

License

MIT (corpus + tooling)

Last edited

2026-05-24

OPBR-Bench is the corpus artifact that makes the Paper 1 methodology testable beyond a single research machine. A versioned benchmark of consequential-agent regression scenarios across four domains, with locked behavior contracts, safe-variation contracts, redacted traces, output schemas, and labeled output-passing behavioral regression. The workbench runs against it; new detectors get scored on it; reproductions and replications start from it.

What is in v0

Per-scenario assets

• Locked behavior-spec/v0 YAML — required tools, required order, required successful order, error/retry budgets.
• Locked safe-variation-spec/v0 YAML — the extra-check condition the contract must still accept.
• Repair contract — proposal/commit split, used for Study 1 repaired condition.
• Control contract — runtime-control gate, used for Study 1 controlled condition.
• Redacted trace pack — baseline, candidate (regression), controlled, repaired, safe-variation runs.
• Output schema — canonical JSON fields the field-level evaluator inspects.
• Domain rule summary — plain-English statement of what makes the consequential action acceptable.
• Labeled ground truth — per-trace semantic and contract-defined labels.

What is not in v0

• Raw Claude transcripts (privacy).
• Raw Codex sessions (privacy).
• Private chain-of-thought (not collected).
• Personal logs (only redacted aggregates would ever appear).
• Any client-derived material.

Schema

The benchmark schema is locked in docs/OPBR_BENCH_SPECIFICATION.md. The corpus validator (experiments/validate-opbr-corpus.mjs) checks structural consistency, label coverage, and contract executability against the workbench detector suite.

node experiments/validate-opbr-corpus.mjs --corpus opbr-bench-v0/

Acceptance criteria for v0 release

The benchmark ships when the following are true and reproducible:

• Four domains × three cases × five conditions (baseline, candidate, controlled, repaired, safe-variation) cover Paper 1 Studies 1, 2, and 5.
• Labels available for every trace, with semantic and contract-defined columns distinguished.
• The detector suite achieves the Paper 1 results on the released corpus, deterministically, in CI.
• Validator passes on the released corpus.
• Reproduction protocol available in the workbench README and in docs/PAPER1_STATISTICAL_PLAN.md.

Current status: spec locked; corpus packaging in progress. Tracking under Gate G of the Paper 1 acceptance gates.

How to use

git clone https://github.com/raising-agents/agent-behavior-workbench
cd agent-behavior-workbench
npm install
npm run build

# Validate the bench
node experiments/validate-opbr-corpus.mjs --corpus opbr-bench-v0/

# Run the detector comparison on a scenario
GENERATION=opbr-v0 \
GROUND_TRUTH=semantic \
ROOTS="opbr-bench-v0/policy,opbr-bench-v0/claim,opbr-bench-v0/memory,opbr-bench-v0/code" \
OUT_JSON=detector-comparison.json \
node experiments/run-detector-comparison.mjs

# Add a new detector and score it
# (see workbench docs/extending-detectors.md)

Leaderboard

The benchmark is not designed as a leaderboard primarily — it is designed as a reproducibility floor for a specific failure mode. That said, when external groups score new detectors against OPBR-Bench, results will be listed here with the same metrics the workbench uses (precision, recall, F1, FPR with Wilson 95% intervals; McNemar paired tests with Holm-Bonferroni correction). The Paper 1 results stand as the v0 reference scores.

No external submissions yet. The first independent reproduction will be listed here when it lands.

Citation

@misc{opbrbench2026,
  title = {OPBR-Bench: A Reproducible Benchmark for Output-Passing
           Behavioral Regression in AI Agents},
  author = {Sanchez de la Sierra, Adrian},
  year = {2026},
  url = {https://raisingagents.is/bench/},
  note = {Companion artifact to "Output-Passing Behavioral Regressions
          in AI Agents: Detection, Baselines, and Repair with Agent
          Behavior Contracts."}
}