Evidence of completion for AI coding agents

Evidence of completion is the rule at AR Labs that work by an AI coding agent (an AI agent that writes code, such as Claude Code or Codex) counts as done only when something checkable proves it. Every acceptance criterion (a testable statement of what a piece of work must achieve) is closed by a committed automated test, a committed check script that anyone can re-run, or, where only a person can judge the result, a committed screenshot or recording. Nothing is closed by narrative. An agent reporting that it ran the build, passed the tests or finished the task is not evidence of any of those things; a real diff, a real test run or a committed check that can be run again is. Adopt the rule if your agents ever report work finished that is not.

The problem it prevents

AI coding agents sometimes report work as done when it is not. Research on reward hacking (a model satisfying the letter of its objective while missing its intent) finds that a common exploit is skipping a required verification step and producing a plausible-looking sign of completion instead. The countermeasure is to let work move forward only on artefacts that can be inspected or re-run, and to have them judged by something other than the agent that produced them.

Three tiers of proof

Each criterion declares one proof type, and the strongest tier that applies is preferred.

Most criteria are already covered by the tests written under test-driven development, so the rule adds little work. It only changes what happens to the remainder: whatever tests cannot cover gets checked by a committed script, or recorded for a person to look at, rather than described. A log or text dump is not an observation; a check that reads one is a tier-2 script.

The ladder puts the tiers in order of preference, with what each one commits and the single proof-manifest line every check leaves behind. Narrative sits below the bottom rung, broken.

To find a criterion's rung, its author asks two questions in turn:

Pulling claims up to tier 1

Many claims feel untestable until you set up an adversarial case. "The voice path never waits for the note-taker" sounds like something to eyeball in a log. Register a note-taking handler that sleeps for a fixed time, then assert that dispatch still returns within a bound and the critical path's latency is unchanged, and the claim becomes a test that fails first and then passes. We call these mechanical properties and reaches for them before settling for a recording.

Forbidden conditions

Some criteria say what must not happen: no blocking call on a critical path, no write to a read-only location, no unprompted reply. These are written down explicitly, each with a check that confirms compliance. Often the cheapest check is a static search proving the forbidden call never appears on the path.

Where the evidence lives

Check scripts and screenshots are committed with the change, in a per-task folder. Raw machine output is not: no JSON captures, logs or text dumps. A measurement script the project keeps as a lasting tool can live with the code it measures, and its manifest line names that path. A tier-1 test needs no separate folder, because the test in the suite is the artefact.

build/specs/evidence/app-42.3/
  probe-check.sh            # tier 2: the script is committed, its output is not
  homepage-mobile.png       # tier 3: a person must look at this
scripts/
  latency-trial.sh          # a lasting harness, kept with the code it measures

We used to commit raw output alongside each script. A review of a month of pipeline material found that evidence folders made up a large share of it, mostly raw output, and that none of the bugs those checks caught had needed the output committed: running the script caught them. The rule now commits the script and records the result.

The proof manifest

The spec declares, for each criterion, which kind of proof it needs. The implementer discharges each obligation and records it in a proof manifest: one line per check, giving the criterion, the command, the result and the commit it ran at. The manifest goes in the task's notes and in the pull request body.

S3 | build/specs/evidence/app-42.3/probe-check.sh | PASS | 3c46397
E2 | scripts/latency-trial.sh --runs 50 | PASS (p95 212 ms < 300 ms) | 3c46397
S5 | pytest tests/test_sessions.py::test_returning_visitor_resumes_an_unfinished_session | PASS | 3c46397

Each line is keyed by the criterion's scenario ID where it has one. A decision that rests on a number, such as which of two models to use, carries that number in the result column, so the decision is derived from a recorded result rather than asserted in prose. A criterion with no line, or whose line names a script that does not exist, is not done. The reviewer reads the manifest and may re-run any command at the commit it names.

The builder does not judge its own proof

Whether the outcome happened is decided in a fresh context, by the independent reviewer's evidence axis (see the code review gate) or by a person, looking at the committed tests, scripts, screenshots and manifest lines, never at the implementer's account of them. An agent reviewing its own work in the same session tends to explain away its own gaps.

Matching rigour to risk

The rule is calibrated so it does not turn into ceremony. A slice (one thin piece of a larger feature) that exists to retire a named risk gets the full treatment: every risk gets a tier-1 property test or a tier-2 check script, and its decisions are derived from a check's recorded result. A standard slice leans on its tests and captures only what they cannot cover. A trivial change, such as configuration or documentation, needs one passing test or validation check and no evidence folder.

What the rule cannot prevent

A determined agent could write a manifest line its script would not produce. The defence is reproducibility plus independent judgement: because the check script is committed and the line names the commit it ran at, a reviewer or the CI system can re-run it and catch a forged result. That makes faking cost more than doing the work. It does not make faking impossible, and the rule does not claim it does.