Test-driven development with AI coding agents

At AR Labs every AI coding agent (an AI agent that writes code, such as Claude Code or Codex) implements in red-green-refactor order: write a test from the acceptance criteria, watch it fail for the right reason, write the least code that makes it pass, then tidy. Writing the test first stops an agent from writing tests that confirm whatever its code happens to do. The rule applies to planned work and to quick fixes alike, and it ends with the agent checking by hand that the feature works, not just that the tests pass. The criteria come from the task's spec, and the tests become part of its evidence of completion.

The problem it prevents

Agents left to write code first and tests afterwards tend to write tests that confirm whatever the code does, including when the code is wrong. Simon Willison's guide to agentic engineering patterns recommends red/green test-first development for coding agents because it guards against code that does not work and code that is not needed. Matt Pocock describes the same effect: writing the test first instruments the code before it exists, which makes it much harder for an agent to bend the test to fit broken behaviour.

How it works

1. Run the existing tests first, a step Willison also recommends. Before writing anything, the agent finds and runs the project's suite. That tells it the framework, where tests live and how they are named. It then reads one or two existing test files and matches their style. If the project has no test setup, setting one up is the first task, not something to skip.
2. Red. Write one failing test from an acceptance criterion and run it. A failure caused by an import error or a syntax error is not red; it is broken setup, and it gets fixed first. Red means the test runs cleanly and its assertion fails because the feature does not exist yet. A test that passes immediately is not testing anything new, so it is replaced.
3. Green. Write the least code that makes the test pass. If the test checks that a function returns a list, return a list; do not add pagination, sorting and caching. No refactoring and no error handling for cases the test does not cover.
4. Refactor only if needed, and run the tests again.
5. Regression. After each build step, run the tests that exercise what changed. Once, before handing off to review, run every test the change could affect.
6. Manual verification. Start the service and exercise the feature through a browser, curl or a direct call. Automated tests show the code is correct; manual verification shows the feature is.

Test at the boundary of a deep module

We take our test boundaries from John Ousterhout's idea of the deep module: a module with a small interface and substantial logic behind it. Tests drive the module through its public interface and assert what a caller can observe. A test that asserts on an internal helper breaks whenever the internals change, even though the module still works.

From that follow a few working rules. Test behaviour ("when the user does X, the system does Y"), not implementation ("the mock was called with Z"). Mock everything external, and never call a real third-party API from a test. Do not unit-test display-only components; cover critical user flows end to end instead. Write failure messages in the language of the domain, not the code.

One Gherkin scenario, one test

Gherkin is a plain-language format for acceptance criteria: each scenario states a starting situation (Given), an action (When) and an expected outcome (Then). When acceptance criteria are written as Gherkin scenarios, each scenario becomes exactly one test, named after the scenario. Scenarios are never merged into one test or split across several, so a reviewer can map the spec to the suite line by line.

# Scenario: Returning visitor resumes an unfinished session
#   Given a visitor with an unfinished session from yesterday
#   When they open the app
#   Then they see the session where they left it
 
def test_returning_visitor_resumes_an_unfinished_session(client, seed_session):
    seed_session(user="visitor-1", step=3, finished=False)   # Given
    response = client.get("/", headers={"X-User": "visitor-1"})  # When
    assert response.json()["session"]["step"] == 3              # Then

A ceiling on the regression run

The pre-handoff regression run is a gate with a ceiling of ten minutes of wall time. At the ceiling the run stops, and the agent records which suites passed, which failed and which did not finish, then hands off on that record. A suite that did not finish is named, never dropped. Agents judge a run from the runner's one-line-per-suite summary and open a full log only to diagnose a failure the summary names.

The ceiling came from the pipeline's own test suites, in which every check built a throwaway database and ran the whole system through it, so each suite took 25 to 44 minutes. A fix of about 40 lines cost hours of test runs. Our research found that the only well-sourced time budget for a gating test run is an absolute one of about ten minutes (the Extreme Programming ten-minute build, and DORA's upper limit of about ten minutes), and that large engineering organisations split verification into a fast run of affected tests before a change is accepted and a slower full run afterwards. The ceiling applies that split to agent work.

When tests do not apply

Pure configuration, documentation and infrastructure changes still get validated: run the linter or the build, or check the result by hand, and record what was checked.