Evaluating the harness

A harness nobody measures is decoration. AgentRig ships three layers of evaluation, each making a bounded claim. Don't over-read what any one of them proves.

All three persist results under .agentrig/eval/results/ via score.mjs. Results are validated on read AND on write (schemaVersion: 2) — never hand-edit JSON; invalid records are quarantined into results/_legacy/.

New here? Generate scenarios for your repo first. Layer B needs scenarios, and the bundled ones are generic JS templates. agentrig eval --scaffold reads the investigation init wrote (.agentrig/context.md) and writes fixture-based scenarios that use your repo's real test runner, package manager, and defect patterns — see Per-repo customization below.

Layer A1 + A2 — static audit#

Deterministic. Dependency-free. Safe in CI.

agentrig eval --static                 # human report, A1 and A2 in separate sections
agentrig eval --static --json          # machine-readable
agentrig eval --static --min 80        # fail CI if Install Completeness < 80%

Or the dep-free script directly (handy in fresh worktrees):

node .agentrig/eval/static-audit.mjs

Sample output:

AgentRig — install completeness audit
  what this proves: every canonical artifact is present and minimally well-formed.
  what this does NOT prove: that those artifacts work, or that agents respect them.

  Layer A1 — structural completeness
  [PASS] P1  Workflow is an explicit, connected state machine (DAG with queued→merged path)
  [PASS] P2  Multiple specialized agent roles installed
  …

  Layer A2 — quality probes
  [PASS] P2  Developer and reviewer use DIFFERENT model families (not just different ids)
  [PASS] P4  Every skill declares description + allowed-tools (not just self-verify)
  [PASS] P6  axes.json has at least one issue code per axis
  [PASS] P12 AGENTS.md has no unfilled {{PLACEHOLDER}} tokens
  …

  Install Completeness: 100%  (33/33 full credit)
  Quality Probes:       100%  (6/6 full credit)

CI gates on Install Completeness with --min. Quality Probes are diagnostic — they surface drift without failing the build.

Layer B — dynamic behavioral eval#

For each scenario in .agentrig/eval/scenarios/<id>/:

1. Seed a throwaway worktree from scenarios/<id>/fixture/.
2. Producer model runs prompt.md in the worktree. For --variant harness the AgentRig harness is staged into the worktree first; for --variant baseline the agent runs bare.
3. Oracle (scenarios/<id>/oracle.yml) deterministically scores hard axes (correctness, tests, scope, regression_risk, …) by running commands and inspecting the diff. No LLM.
4. Judge model — a different family from the producer — runs in a separate provider.startConversation() call in its own cwd containing only prompt.md, diff.patch, transcript.md, oracle.json, and judge_brief.md. It does NOT see the producer worktree or reasoning trace. It writes scores to a JSON file the orchestrator validates against axes.json.

Bare invocation#

agentrig eval --dynamic resolves models from .agentrig/agents/developer.yml (producer) and .agentrig/agents/reviewer.yml (judge) by default — the roster already enforces these come from different model families via the install-completeness audit. So the common path is just:

agentrig eval --dynamic

Override either model explicitly when running a one-off experiment:

agentrig eval --dynamic \
  --producer-model claude-sonnet-4.6 \
  --judge-model gpt-5.5

Or via env vars (handy in CI scripts):

AGENTRIG_PRODUCER_MODEL=claude-sonnet-4.6 \
AGENTRIG_JUDGE_MODEL=gpt-5.5 \
  agentrig eval --dynamic

Resolution chain (highest precedence first):

1. --producer-model / --judge-model CLI flags
2. AGENTRIG_PRODUCER_MODEL / AGENTRIG_JUDGE_MODEL env vars
3. --model (legacy back-compat — only sets the producer)
4. .agentrig/agents/developer.yml model (producer) / .agentrig/agents/reviewer.yml model (judge)
5. Provider default

The chosen model + source is logged at the start of the run and recorded in meta.json.

Useful flags#

Rubric rules (enforced by score.mjs)#

1. Strict 3-tier scores: 0 / 0.5 / 1.0.
2. Issue code required. Any axis < 1.0 with confidence > 0 must carry an issue code from that axis's bounded registry plus a one-line evidence string.
3. Confidence-gated. An axis you couldn't observe is =na (confidence 0) and excluded from rollups.
4. Weighted aggregation. Axes carry an optional weight (default 1) and veto: true. The aggregate is a weighted mean of observed axes.
5. Pass rule: aggregate ≥ passThreshold AND no observed axis at 0 AND no veto axis < 1.0. Veto fails are surfaced in failReason.

Lifecycle types#

Inspect the live rubric in your repo:

agentrig eval --rubric

Statistical lift — does the harness actually help?#

Single-trial deltas are coin flips. The eval requires n ≥ 3 paired trials for any verdict other than INCONCLUSIVE. Pattern:

agentrig eval --dynamic --variant harness  --n 5 \
  --producer-model claude-sonnet-4.6 --judge-model gpt-5.5
agentrig eval --dynamic --variant baseline --n 5 \
  --producer-model claude-sonnet-4.6 --judge-model gpt-5.5

node .agentrig/eval/score.mjs compare --scenario <id> --baseline baseline

compare --baseline pairs trial i of harness with trial i of baseline, computes the median delta and a binomial sign-test p-value, and prints one of three verdicts:

• HELPS — p < 0.05 and median > 0.05
• HURTS — p < 0.05 and median < -0.05
• INCONCLUSIVE — n < 3, p ≥ 0.05, or |median| < 0.05

A HELPS verdict on a real fixture, in a different model family than the judge, is the only thing that justifies the line "AgentRig improved agent behavior here." Anything less is honest inconclusive.

Calibrating the judge#

A judge that returns 1.0 on every axis passes every save validation but tells you nothing. .agentrig/eval/calibration/ ships hand-labeled rubric instances (scenario inputs + transcript

• diff + ground-truth axes). Two-step flow:

# 1. Have your judge model score one of the seed instances and write
#    {"axes":[{"name":"...","score":1.0,"confidence":1}, …]} to /tmp/judge-out.json.
# 2. Score the judge against ground truth.
node .agentrig/eval/score.mjs calibrate \
  --judge gpt-5.5 \
  --instance .agentrig/eval/calibration/run/seed-correct.yml \
  --judge-scores /tmp/judge-out.json

# Roll up across instances.
node .agentrig/eval/score.mjs calibrate --report

agentrig doctor reads the rollup and fails (exit 1) on any judge averaging below 80% agreement within ±0.5 tier of the hand-labeled truth. See .agentrig/eval/calibration/README.md for the instance format and how to grow the calibration set.

Per-repo customization#

Everything under .agentrig/eval/ is editable:

• Scenarios — .agentrig/eval/scenarios/<id>/ (drop in your own fixture + oracle.yml)
• Axes — .agentrig/eval/axes.json (add domain-specific axes like accessibility_compliance)
• Checks — .agentrig/eval/checks.json (tighten or expand the structural audit)
• Calibration — .agentrig/eval/calibration/<type>/*.yml

Bundled vs repo-specific scenarios#

Scenarios that ship with agentrig init (currently add-small-feature, fix-failing-test, review-catches-bug) are marked bundled: true in their scenario.yml. eval --dynamic excludes them from the default run — you want signal about your repo's stack and conventions, not generic JS micro-fixtures.

agentrig eval --dynamic                   # only repo-specific scenarios (default)
agentrig eval --dynamic --include-bundled # also run the bundled template scenarios
agentrig eval --dynamic --scenario fix-failing-test  # explicit ids always run, bundled or not

If you've never run --scaffold, the default run will be empty and the CLI will tell you so:

No repo-specific scenarios found.
Generate some with: agentrig eval --scaffold
Or run the bundled template scenarios anyway: agentrig eval --dynamic --include-bundled

Generating repo-specific scenarios (eval --scaffold)#

The 3 bundled scenarios are language-agnostic JS micro-fixtures — useful as templates, but they don't exercise your repo's test runner, package manager, or common defect patterns. Run:

agentrig eval --scaffold                  # generate 2 repo-tailored scenarios (default)
agentrig eval --scaffold --scaffold-count 3
agentrig eval --scaffold --timeout 10     # raise the per-agent-turn cap (default 45min)

--scaffold reads .agentrig/context.md (the investigation init wrote about your repo) + the 3 generic scenarios as templates, then asks the agent to generate N new scenarios under .agentrig/eval/scenarios/<id>/ that use your repo's actual stack. Each generated scenario is validated against axes.json; invalid output is flagged but not silently deleted, so you can inspect + fix it by hand.

The agent is constrained to:

• Use only the axis names already in axes.json (no inventing axes)
• Use your repo's actual test runner / package manager
• Keep the fixture ≤10 files
• Touch only .agentrig/eval/scenarios/
• Omit bundled: true so the new scenarios run by default

Confirm what's actually measured with agentrig eval --rubric.

When to run what#

CI#

The dynamic eval is too expensive for every PR. Recommended split:

# ci.yml — every PR (cheap, deterministic, no model)
- run: npx -y @doidor/agentrig@latest eval --static --min 100

# agentrig-eval.yml — nightly on main + manual dispatch (real model spend)
on:
  schedule: [{ cron: "17 6 * * *" }]
  workflow_dispatch:

The full nightly workflow ships at .github/workflows/agentrig-eval.yml — it runs both variants with n=5, computes lift, uploads JSON artifacts, and posts/updates a tracking issue with the verdict table.