Raising Agents Subscribe →

OPEN-SOURCE INSTRUMENT

Agent Behavior Workbench

Agent Behavior Workbench is local-first, vendor-neutral, MIT-licensed tooling for behavioral regression testing and behavior contracts on AI agents. It imports agent execution traces, normalizes them into the agent-behavior-trace/v0 schema, compares candidate runs against a baseline behavior contract, and emits a static HTML report plus a CI-friendly PASS / WARN / FAIL / INCONCLUSIVE gate.

The workbench is the reproducibility surface for Paper 1 and Paper 2. Every claim in those papers reproduces from the locked protocol files (PAPER1_PROTOCOL_FREEZE.json) and the corpus shipped with the repo. The methodology is auditable, not assertable.

Version
v0.1.0 · public alpha · API surface may change
Stack
TypeScript (CLI) + Python (latent-space tooling)
License
MIT
Schemas
v0 versioned. Four public.
Scope
Tool-mediated effects (observable, reproducible). Larger object: consequential behavior.
Status
Cited by Paper 1 (5 studies, 3,797 runs) and Paper 2 (whitepaper, v1).
GitHub repo → Deep reference Interactive demo The Stack Paper 1

Quick start

Five commands. Then an HTML report.

The default demo intentionally produces a behavioral failure while the final output remains successful. This is the canonical OPBR shape: same JSON, two trajectories, contract flips.

# Clone, install, build, run the demo. git clone https://github.com/raising-agents/agent-behavior-workbench.git cd agent-behavior-workbench npm install npm run build npm run demo open reports/supportbot-demo.html

The HTML report renders the trace pair side-by-side, marks the violated precondition, and emits a verdict line you can wire into a CI gate. The same artifact reproduces every figure in Paper 1's empirical sections.

The CLI

One binary. Six commands. Composable.

CommandWhat it does
abw import claude Import a Claude Code transcript and normalize it into agent-behavior-trace/v0. --redact strips sensitive content per the redaction policy.
abw spec init Author a baseline behavior contract from a directory of baseline traces. Emits behavior.yaml with preconditions, invariants, governance policies, recovery mechanisms.
abw diff Compare candidate traces against the baseline. Emits a structured diff (diff.json) with per-event delta and contract-violation annotations.
abw gate Apply the locked contract to candidate traces and emit a CI-friendly verdict: PASS / WARN / FAIL / INCONCLUSIVE. The verdict shape inherits AgentAssay's three-valued framework with an added WARN bucket for soft-violation budgets.
abw report Render the gate result as a static HTML report. No JS runtime. Inspectable offline. Suitable for PR comments via GitHub Action.
abw demo supportbot Run the canonical OPBR demo end-to-end. Produces reports/supportbot-demo.html. The verdict-flip case that drives Paper 1's wedge.

Schemas

Four versioned schemas. Stable URLs. Public commitments.

The workbench reads and writes against four versioned schemas, all served at stable URLs under raisingagents.is/system/schema/* and locked at v0. Versioning is a public commitment: future incompatible changes will land at /v1, never overwrite v0.

agent-behavior-trace/v0 agent-behavior-contract/v0 regression-of-the-week/v0 lab-board/v0

The trace schema is the normalized form every adapter targets (Claude Code, Codex CLI, custom harnesses). The contract schema encodes preconditions, hard invariants, governance policies, recovery mechanisms — the ABC (P, I, G, R) tuple in a workbench-readable form with (p, δ, k)-satisfaction fields. The regression-of-the-week schema is the artifact format for Behavior Watch case files. The lab-board schema is the workshop-view backing for /lab/.

GitHub Action

Drop it in your CI. Block behavioral regressions on PR.

The workbench ships a composite GitHub Action so the gate can run on every PR. The action assumes baseline traces and a locked contract are committed to your repo; the candidate traces are produced by your test suite. A failing gate posts the PR comment with the violated invariant and a link to the report.

# .github/workflows/abw-gate.yml name: Agent Behavior Gate on: [pull_request] jobs: gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: raising-agents/agent-behavior-workbench@v0 with: baseline: traces/baseline candidate: traces/candidate spec: behavior.yaml fail-on: FAIL

The action is the integration surface that turns behavior contracts from a paper concept into a deployable CI gate. The v0 release ships with a worked example wired against the Claude Code before/after experiment in experiments/claude-code-before-after.

Where this fits

The workbench is one of three timescales.

The Raising Agents research program publishes at three timescales. The workbench is the reproducibility instrument; the papers are the load-bearing arguments; the bi-weekly publication is the continuous-practice surface.

Workshop artifacts

The library of trace pairs, contracts, analyzers, and primitives.

The workbench is the tool; the workshop artifacts are what it operates on. These are the named, versioned, status-tagged building blocks the lab uses internally — they support experiments but are not themselves experiments. Source of truth: system/data/lab-board.json.

loading workshop manifest…

License and contact

MIT. Use it commercially. Fork it. Vendor-neutral by design. The workbench is the integration point; the brand-load lives in the papers and the publication.

Issues, PRs, and adapter contributions land in the GitHub repo once the Raising Agents organization is published. Direct inquiry: adrian@raisingagents.is; Zartis-routed team work: adrian.sanchez@zartis.com.

For teams

Open tool, private implementation path.

The workbench is MIT-licensed and public. Applying it to production agents, internal traces, policies, and CI gates is Zartis-routed work. That lets the public artifact stay reusable while client-specific details stay off the site.

Work with us → Follow Behavior Watch →