TECHNICAL REFERENCE

How the workbench actually works.

The Agent Behavior Workbench is a TypeScript CLI with a Python companion. It converts agent execution traces into a proprietary normalized format, compares candidate runs against a locked behavior contract, and emits a deterministic PASS / WARN / FAIL / INCONCLUSIVE verdict. INCONCLUSIVEthe honest answer. get more runs. This page explains every layer of that pipeline from source.

What it solves, precisely.

An AI agent can return a correct final output while doing the wrong thing. A refund agent that issues the refund before checking fraud policy — then checks policy and passes — produces identical JSON to one that did it in the right order. Pass@1 output evaluation cannot tell the difference. The workbench can, because it compares the trajectory of tool calls — their identity, order, and success — against a locked contract.

Formal statement: the workbench detects output-passing behavioral regression — behavior change that preserves final-output correctness while violating a precondition over a consequential trajectory event. This is the failure mode measured in Paper 1 across 3,797 runs. The workbench is the instrument those runs used.

The format

Yes, proprietary. And intentionally so.

The workbench defines its own trace format: agent-behavior-trace/v0. Not OpenTelemetry. Not LangSmith's format. Not the Claude Code raw JSONL. A custom JSON schema with stable versioned URLs at raisingagents.is/system/schema/.

Why proprietary: agent platforms each export different trace shapes. A normalization format that every adapter targets is the thing that makes cross-platform comparison possible. The schemas are the stable interop layer; the implementations are adapters.

A agent-behavior-trace/v0 trace has three top-level sections:

// agent-behavior-trace/v0 — the canonical JSON
{
  "schema_version": "agent-behavior-trace/v0",

  // WHERE it came from
  "source": {
    "platform": "claude_code",          // or "codex_cli", "langchain", etc.
    "export_format": "jsonl",
    "exported_at": "2026-05-24T12:00:00.000Z",
    "source_hash": "a3f..."              // content fingerprint of the raw file
  },

  // WHAT happened at the run level
  "run": {
    "trace_id": "...",
    "scenario_id": "refund-policy",      // optional: which OPBR-Bench scenario
    "environment": "ci",
    "final_status": "success",            // success | error | cancelled | unknown
    "final_output_hash": "5c3...",       // SHA-256 of the final output text
    "canonical_output": {                // optional: structured field extraction
      "fraud_check_passed": true,
      "refund_amount": 1200
    }
  },

  // THE TRAJECTORY — ordered list of spans
  "spans": [
    {
      "span_id": "span_0_a4f",
      "ordinal": 0,                        // position in the trajectory
      "type": "llm",                       // llm | tool | retriever | memory | handoff | guardrail
      "name": "assistant_turn",
      "status": "success",
      "input_hash": "...",                 // SHA-256 of input content — no raw text
      "output_hash": "...",                // SHA-256 of output content
      "attributes": {
        "model": "claude-opus-4-7",
        "tool_name": null,
        "token_input_count": 4827,
        "redacted": true                  // always true — no raw content ever stored
      }
    },
    {
      "ordinal": 1,
      "type": "guardrail",                 // policy_check → guardrail type
      "name": "policy_check",
      "status": "success"
    },
    {
      "ordinal": 2,
      "type": "tool",
      "name": "refund_commit",
      "status": "success"
    }
    // ... more spans
  ],

  // WHAT was stripped and how
  "redaction": {
    "applied": true,
    "method": "hash-only",
    "removed_fields": ["message.content", "tool_input", "tool_output", "file_path", ...]
  }
}

Redaction is always on by default. Raw prompt text, tool inputs, tool outputs, and file paths are never stored in traces. Inputs and outputs are reduced to SHA-256 hashes. This makes traces safe to commit, diff, and ship as part of a reproducibility corpus without exposing proprietary data.

Span type classification happens at import time. The Claude importer maps semantic Bash labels to trace spans: if a Bash call matches /policy_check/ or /refund_commit/, those become the span name instead of the generic "Bash". The label list is configurable per experiment.

The pipeline

Five commands in dependency order.

`abw import claude` — JSONL → trace

Reads a Claude Code conversation JSONL file. Walks every record line by line: assistant messages become llm spans; tool_use blocks become typed spans (tool, retriever, memory, handoff, guardrail) named after the tool; tool_result blocks close the pending span, record status (success/error), and timestamp duration. Bash calls are matched against a configurable label list to extract semantic names from raw shell commands. All content is SHA-256 hashed. Output is a single agent-behavior-trace/v0 JSON file.

# import with redaction + Bash semantic labels
abw import claude \
  --input ~/.claude/projects/my-project/session.jsonl \
  --out traces/candidate/run-001.json \
  --redact \
  --scenario-id refund-policy-check \
  --bash-tool-labels policy_check,refund_commit,fraud_check

The importer is the only Claude-specific component. The rest of the pipeline is format-agnostic — any adapter that produces valid agent-behavior-trace/v0 JSON plugs in here.

`abw spec init` — baseline traces → behavior contract

Takes a directory of baseline traces and generates a behavior-spec/v0 YAML. Auto-extracts: the set of required tools (every tool that appeared in any baseline run), max path length (1.25× the longest baseline trajectory), max errors, max retries. You then edit the YAML to add order constraints, forbidden tools, and statistical thresholds, then lock it as the gate.

abw spec init \
  --baseline traces/baseline \
  --out behavior.yaml

The generated YAML is a starting point, not a verdict. The critical step is authoring the order constraints — the preconditions over consequential events that the agent must satisfy. Auto-generation cannot infer those; they encode the domain logic of what counts as safe behavior.

`abw diff` — baseline vs candidate aggregate

Computes three distances between the aggregate baseline trajectory and the aggregate candidate trajectory:

Path edit distance. Normalized Levenshtein on the tool sequences. Captures insertions (extra tools), deletions (skipped tools), and substitutions (different tool order).
Fingerprint distance. Euclidean distance between the tool-call histograms. Captures frequency shift — how much more or less each tool was called.
Tool Jaccard distance. Set-theoretic distance on which tools appeared at all. Captures categorical drift — new tools appearing, known tools disappearing.

Also: first divergent step — the ordinal position where the candidate trajectory first diverges from the baseline common path. Useful for debugging which step triggered the behavioral shift.

`abw gate` — the verdict

Applies the locked spec to the candidate traces. Runs five categories of checks and takes the worst-case across all checks as the overall verdict:

Category	Checks	Fails on
output_eval	Canonical output field matching	Expected JSON field value doesn't match
behavior_contract	Required tools, forbidden tools, order constraints	Missing tool, forbidden tool present, order violated
operational_quality	Path length, error count, retry count	Exceeds budget from spec
diff_shape	Path edit distance, fingerprint distance, Jaccard	Exceeds warn/fail thresholds from spec
sample_sufficiency	Min runs, Wilson lower bound, SPRT	Too few runs or statistical test fails

Verdict semantics:

Verdict

Meaning

What to do

PASS

All checks pass

Ship the candidate

WARN

Soft thresholds exceeded; hard constraints pass

Review — allowed by default, optionally block

FAIL

One or more hard constraints violated

Do not ship. Investigate trace.

INCONCLUSIVE

Insufficient sample for statistical inference

Run more candidate traces

`abw report` — static HTML report

Renders the GateReport JSON to a static HTML file. No JavaScript runtime — plain HTML table rendering the checks, diff metrics, SPRT state, evidence ledger, and per-trace status. Suitable for CI PR comments, offline review, and archiving alongside the trace corpus.

abw gate \
  --baseline traces/baseline \
  --candidate traces/candidate \
  --spec behavior.yaml \
  --out gate-report.json

abw report --input gate-report.json --out report.html

The contract

What a behavior spec actually looks like.

The behavior-spec/v0 YAML is the contract. It has two sections: invariants (hard constraints, always binary pass/fail) and thresholds (soft bounds and statistical configuration).

# behavior-spec/v0 — the refund policy case (OPBR canonical)
schema_version: behavior-spec/v0
id: supportbot-refund-behavior
name: SupportBot refund behavior

invariants:
  # Every candidate run must have used all of these tools
  required_tools:
    - classify
    - lookup_order
    - policy_check
    - refund_tool

  # None of these may appear in any candidate run
  forbidden_tools: []

  # Ordering constraints — THE consequential preconditions
  required_order:
    - before: policy_check        # policy_check MUST come before refund_tool
      after: refund_tool          # a refund committed before policy check → FAIL

  max_path_length: 5
  max_errors: 0
  max_retries: 0

thresholds:
  # Trajectory shape: how far from baseline is "warn" vs "fail"
  path_edit_distance_warn: 0.12
  path_edit_distance_fail: 0.25
  fingerprint_distance_warn: 1
  fingerprint_distance_fail: 2
  tool_jaccard_warn: 0.15
  tool_jaccard_fail: 0.4

  # Statistical requirements
  min_runs: 1
  min_wilson_pass_rate: 0.7       # Wilson lower bound must be ≥ 70%

  # Optional: SPRT (Sequential Probability Ratio Test)
  sprt:
    enabled: false                # enable for distributional eval across many runs
    alpha: 0.05                   # Type-I error bound (false alarm rate)
    beta: 0.20                    # Type-II error bound (missed detection rate)
    p0: 0.95                      # H0: acceptable pass rate
    p1: 0.80                      # H1: degraded pass rate (what we want to detect)

The required_order constraints are the core of the contract. Each constraint says: "in the trajectory of successful tool calls, before must appear before after." If the agent commits a consequential action before satisfying a precondition, the gate FAILs — regardless of what the final output says.

Statistics

Wilson bounds + SPRT: making inference rigorous.

Single-run pass@1 is blind to the distributional variance of stochastic agents. The gate implements two statistical tools to move beyond it:

Wilson lower boundone-sided. we care about the floor, not the point estimate.

A one-sided confidence lower bound on the true pass rate. Given k passes out of n runs, the Wilson lower bound (z = 1.96 for 95% CI) is:

// Wilson lower bound — from gate.ts
function wilsonLowerBound(successes: number, n: number, z = 1.96): number {
  const phat = successes / n;
  const denom = 1 + (z * z) / n;
  const center = phat + (z * z) / (2 * n);
  const margin = z * Math.sqrt((phat * (1 - phat) + (z * z) / (4 * n)) / n);
  return Math.max(0, Math.min(1, (center - margin) / denom));
}
// If Wilson lower bound < min_wilson_pass_rate → FAIL

If the spec sets min_wilson_pass_rate: 0.70 and your 10-run sample has 6 passes (60% observed), the Wilson lower bound may put the true rate below 70% → FAIL. Single-run evaluation would have reported 100% on a lucky run and shipped.

SPRT (Sequential Probability Ratio Test, Wald 1945)

When sprt.enabled: true, the gate runs a sequential hypothesis test that accumulates evidence across runs and stops as soon as Type-I and Type-II error bounds are satisfied. No fixed sample size required; the test returns PASS, FAIL, or INCONCLUSIVE at any stopping point.

// SPRT log-likelihood ratio — from gate.ts
const llr = successes * Math.log(p1 / p0) +
            failures * Math.log((1 - p1) / (1 - p0));

const upper = Math.log((1 - beta) / alpha);     // accept H0 boundary
const lower = Math.log(beta / (1 - alpha));      // reject H0 boundary

// llr >= upper → FAIL  (enough evidence to reject H0: pass rate degraded)
// llr <= lower → PASS  (enough evidence to accept H0: pass rate acceptable)
// otherwise  → INCONCLUSIVE  (collect more runs)

The SPRT-backed gate is what AgentAssay uses to achieve 86% detection power where binary testing achieves 0%. SPRT is not enabled by default in the workbench — it requires choosing p0, p1, alpha, and beta for the specific task family. Enable when running distributional evaluation across many runs.

Python companion

abw-latent: open-model representation trajectories.

The Python package (pip install abw-latent, name abw_latent) is a separate artifact for open-weight models onlyClaude's internals: no access. this is a research stub, not production tooling.. It extracts hidden-state vectors from a local model (default: Qwen/Qwen2.5-0.5B-Instruct) at each step of the agent trajectory, computes trajectory geometry in embedding space, and outputs a representation-trajectory/v0 JSON.

# Run the latent lab demo — requires torch + transformers
cd python && uv sync
abw-latent demo \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --out representation-trajectory.json \
  --step "Agent classifies request as refund_related." \
  --step "Agent looks up the order record." \
  --step "Agent checks refund policy before calling refund tool." \
  --step "Agent issues the refund."

What it computes per step:

Hidden-state vector — the last-token embedding from a specified transformer layer (default: final layer).
Entropy — softmax entropy of the next-token logit distribution. High entropy = model is uncertain. Useful as a pre-action uncertainty signal.
Velocity — L2 distance between consecutive hidden-state vectors. How fast the model's internal representation is changing.
Curvature — rate of change of velocity. Indicates inflection points in the trajectory — where the model "changes direction."
2D projection — PCA(2) for visualization. Lets you plot the trajectory as a curve in embedding space.

Critical scope note (explicit in the source): This is not access to Claude or Codex internals. It is not private chain-of-thought. It only works on open-weight models where you control the inference and can extract hidden states. The evidence ledger in every gate report names open-model representation trajectory as "advisory" — implemented in the Python lab but not wired to the Claude/Codex paths. The workbench explicitly does not use private thinking text.

Five schemas

Every format has a stable version string.

Schema	Format	Who writes it	Who reads it
agent-behavior-trace/v0	JSON	`abw import claude` (or any adapter)	`abw diff`, `abw gate`, `abw report`
behavior-spec/v0	YAML	`abw spec init` (then human edits)	`abw gate`
gate-report/v0	JSON	`abw gate`	`abw report`, GitHub Action, downstream CI
agent-behavior-contract/v0	JSON	Human (formal contract specification)	Workbench evaluator, Paper 1 case files
representation-trajectory/v0	JSON	`abw-latent demo`	Visualization, open-model research

The agent-behavior-contract/v0 schema (distinct from behavior-spec/v0) is the formal contract type from the whitepaper's ABC framework — C = (P, I, G, R). The behavior-spec/v0 YAML is the workbench's operationalization of that contract as a runnable gate configuration. They are related but not identical: spec is runnable; contract is the formal specification.

Python agents

Designed for Claude Code today. Adapter-ready by design.

The only implemented importer is abw import claude — it reads Claude Code JSONL transcripts. The workbench was built specifically for the Claude Code experiment infrastructure that ran Paper 1.

Python-built agents (LangChain, LangGraph, AutoGen, custom Anthropic SDK apps) can use the format directly — write a agent-behavior-trace/v0 JSON file from your Python agent's execution log and pass it to abw diff and abw gate. The gate pipeline is format-agnostic from step 2 onward. What is missing is a Python importer that reads your agent's native trace format and converts it to the schema automatically.

The planned Python library (agent-behavior-workbench on PyPI) will ship:

Adapters for LangChain, LangGraph, OpenAI Agents SDK, custom Anthropic SDK apps — convert native trace formats to agent-behavior-trace/v0.
pytest plugin — pytest-abw — so the gate runs inline in your test suite against live agent executions, not via a CLI hop.
Library import — from agent_behavior_workbench import Contract, Trace, gate — for programmatic use in CI without the CLI.

Same schemas. Same gate logic. Same verdicts. Different language.

What this is not

Three honest exclusions.

Not an LLM-as-judge evaluator. The workbench does not use an LLM to score outputs. Every check is structural — a required tool either appeared or it didn't; an order constraint was either satisfied or violated; a threshold was either exceeded or not. No judge stochasticity. No rubric. The gate-report verdict is deterministic given the same traces and spec.
Not access to internal model states. The workbench sees what the Claude Code transcript reveals: observable tool calls, message order, token counts, status. It does not access chain-of-thought, private thinking, model weights, or logit distributions for hosted models. The Python abw-latent lab does access hidden states — but only for open-weight local models.
Not a general eval framework. The workbench targets one failure mode: output-passing behavioral regression. It is not a replacement for HELM, Inspect, or your LLM-as-judge pipeline. It is a complement — running where output-only evaluation is blind.

Where to go next.

/ workbench

Workbench landing

Quick start, CLI reference, schemas, GitHub Action, license.
/ demo

Interactive demo

The canonical OPBR case in the browser. Drag the slider. Same JSON. Contract flips.
/ paper 1

Paper 1

5 studies, 3,797 runs, all executed via the workbench.
/ paper 2

Paper 2 whitepaper

Why the workbench operationalizes the right reliability target: behavior contracts, not determinism.

How the workbench actually works.

What it solves, precisely.

Yes, proprietary. And intentionally so.

Five commands in dependency order.

abw import claude — JSONL → trace

abw spec init — baseline traces → behavior contract

abw diff — baseline vs candidate aggregate

abw gate — the verdict

abw report — static HTML report