Test budgets for AI coding agents: cost proportional to change size

How can the cost of verifying a code change be kept proportional to the size of the change, particularly when an AI coding agent writes small fixes against a slow, database-backed test suite? AR Labs researched this question in October 2026 for the verification step of our coding pipeline (see test-driven development with AI coding agents and evidence of completion). The report reviews published work on test selection and test impact analysis (Google, Meta, Anthropic, Microsoft), test-suite shape, CI time budgets and the timeout behaviour of current AI coding agents, and marks each step that no source states directly as inference.

Summary

Teams that keep test cost proportional to a change do not run less verification overall. They split it into two stages. A fast gate runs only the tests a change can affect, and the slower full suite runs afterwards, post-merge or on a schedule, without blocking the change. Google's TAP, Meta's predictive test selection and Anthropic's 2026 test-impact service all work this way, and Meta reports halving test cost while still catching over 99.9% of faulty changes. The only well-sourced time budget is an absolute one: about ten minutes for the gating stage (the XP/Fowler ten-minute build, DORA's "upper limit of about 10 minutes"). No source found ties test time to implementation time as a ratio. No AI coding agent found bounds verification by change size either; agents have per-command timeouts (typically 120–600 s) and guidance to run single tests. A suite in which every check builds a sandbox database and runs the system end to end is what Google's testing book calls an "ice cream cone", and the slowness belongs to the suite's shape, which supports attributing it to the suite. The remedies, in rising cost, are per-run rather than per-test database setup with transaction rollback, diff-to-test selection, and extracting a pure core that can be tested in milliseconds.

1. Test selection and test-impact analysis: deciding what to run

Large engineering organisations stopped running "everything on every change" years ago. They decide what to run from the change itself, using one of three signals: the build dependency graph, recorded per-test coverage, or a model trained on historical test outcomes. All of them pair the cheap per-change run with a slower, fuller run somewhere else that catches what selection misses.

Google TAP (Test Automation Platform). Memon, Gao, Nguyen, Dhanda, Nickell, Siemborski and Micco, "Taming Google-Scale Continuous Testing", ICSE-SEIP 2017. TAP selects tests by reverse build dependencies on the changed files; a single milestone can reach 4.2 million selected tests. Presubmit (the run before code lands) is explicitly "best-effort": it does not test exhaustively and aims to stop most breakages, with further selection and flakiness mitigation driven by machine learning. Exhaustive coverage is pushed to post-submit runs. paper PDF

Meta predictive test selection. Machalica, Samylkin, Porth and Chandra, "Predictive Test Selection", ICSE-SEIP 2019. A model trained on historical test outcomes picks a subset of the tests that transitively depend on a change. The abstract reports that it "reduces the total infrastructure cost of testing code changes by a factor of two, while guaranteeing that over 95% of individual test failures and over 99.9% of faulty changes are still reported." The design accepts a small miss rate per change in exchange for halving cost, and relies on later runs to catch the rest. arXiv 1810.05286

Anthropic's test-impact analysis under agent load. Sachin Malhotra, "Agentic coding is straining CI. Here's how we scaled test impact analysis at Anthropic", Claude blog, 14 September 2026. Anthropic reports engineers shipping eight times as much code per quarter, with Claude authoring 80% of it, test volume up tenfold and CI jobs up 25x in six months. Their selection service is a "listener" that records per-test results from each CI run and a "selector" that reads that history to decide which tests run on each pull request. When the listener lagged, the selector used stale data and either ran unnecessary tests or missed regressions. Successive patches (bigger machine, sharding by package, daily restarts) each bought less time; the fix was a redesign to stateless workers over a shared store. The relevance here: test-impact analysis is the standard answer to agent-volume CI load, and it becomes critical infrastructure in its own right. Claude blog

Tool-level selection (background knowledge, not re-fetched during this research; verify before citing externally).

• Jest: --onlyChanged, --changedSince=<ref> and --findRelatedTests <files> run only tests whose import graph reaches the changed files. Jest CLI docs
• pytest-testmon: records which lines each test executes and re-runs only tests whose covered code changed. testmon.org
• Bazel: test results are cached and only re-executed when a test's declared inputs change; large targets split across shard_count shards. Bazel's size attribute also attaches a default timeout to each test class (small 60 s, medium 300 s, large 900 s, enormous 3600 s). Bazel common definitions
• Microsoft: Azure Pipelines' Test Impact Analysis runs only tests affected by a commit, plus previously failing and newly added tests. Microsoft Learn
• Academic regression test selection: Ekstazi (Gligoric, Eloussi, Marinov, ISSTA 2015) tracks file-level dynamic dependencies per test class and is the usual baseline in the literature.

Common pattern. Every system above separates "what must run before this change is accepted" (selected, fast, allowed to miss rarely) from "what eventually runs against everything" (full, slow, post-merge or scheduled). None of them makes the per-change run complete. A per-change budget is therefore an established idea; the full suite's job moves elsewhere instead of disappearing.

2. Pyramid, trophy and suite shape

The pyramid-versus-trophy debate concerns how a suite's tests are distributed across scopes. It matters here because a suite where every check builds a sandbox database and runs the system end to end has no cheap layer to select from: test selection (section 1) can skip tests, but it cannot make the remaining ones cheaper.

Google's position. Software Engineering at Google (Winters, Manshreck, Wright, O'Reilly 2020), chapter 11 "Testing Overview", written by Adam Bender, defines tests by size (resources a test may use), not by name. A small test "must run in a single process" and may not "sleep, perform I/O operations, or make any other blocking calls"; a medium test may span processes and call localhost; a large test may span machines. Google targets roughly 80% unit, 15% integration and 5% end-to-end tests by count, noting "every team's mix will be a little different". It names two failure shapes. The ice cream cone (many end-to-end tests, few unit tests) gives slow, unreliable suites; the hourglass (many end-to-end and unit tests, few integration tests) misses failures integration tests would catch earlier. It also states the cost directly: "the slower a test suite, the less frequently it will be run, and the less benefit it provides." SWE Book ch. 11

The counter-positions (background knowledge, not re-fetched during this research).

• Kent C. Dodds' testing trophy (2018, revised 2021) puts most weight on integration tests, arguing they give the best confidence per unit of cost in front-end code. kentcdodds.com
• Spotify's testing honeycomb (André Schaffer, 2018) makes the same argument for microservices, where most logic is wiring. Spotify Engineering
• Google Testing Blog, "Just Say No to More End-to-End Tests" (Mike Wacker, 2015), argues that end-to-end tests are slow, flaky and give poor failure localisation. Google Testing Blog
• Martin Fowler, "On the Diverse And Fantastical Shapes of Testing" (2021), argues that most of the pyramid/trophy disagreement comes from people defining "unit test" differently. What actually separates the camps is whether tests are fast, deterministic and isolated, not what they are called. martinfowler.com

When shape, not size, is the problem (inference from the above). All the camps agree on the property that matters for proportional verification: there must be some layer that runs in seconds and can be scoped to a change. A trophy-shaped suite still runs integration tests against a fast in-process or per-test-isolated database. A suite where every test pays full sandbox-database setup has no such layer. Using the SWE Book's vocabulary, it contains only large tests, which is the ice-cream-cone case. Adding tests to such a suite makes it slower, and selection only shrinks the count. A small fix to pure logic that can only be verified through a long end-to-end run is evidence about the suite, not about the fix.

3. Stated time budgets for CI and the inner loop

The published ceilings converge on ten minutes for the build that gates a change, with slower suites allowed only in a later stage that does not block the developer.

Ten-minute build (XP and Continuous Integration). Martin Fowler's "Continuous Integration" (martinfowler.com, revised 18 January 2024) adopts the XP guideline of a ten-minute build and advises acting "as soon as builds start going slower than the ten minute rule." Where the full suite cannot meet that, Fowler prescribes a deployment pipeline (staged build). The commit build runs first, aims to finish in about ten minutes, and uses test doubles to stay fast. Secondary builds run the slower tests against real databases and end-to-end scenarios and "may take hours". A failure in a secondary build lacks the "stop everything" quality of a commit-build failure, but the team fixes it quickly. When a later stage catches a bug, the team strengthens the commit-stage tests so the cheaper stage catches that class of bug next time. martinfowler.com

DORA. The DORA capability page on continuous integration says each commit should trigger automated tests "that provide feedback in a few minutes" and that tests "should not take more than a few minutes to run, with an upper limit of about 10 minutes according to DORA's research." This is the only source found with a ceiling attributed to empirical research and not to practitioner convention. dora.dev

Google. The SWE Book (ch. 11) gives no minute ceiling for presubmit, but it ties test size to permitted resources, and Bazel enforces default timeouts per size class (small 60 s, medium 300 s, large 900 s, enormous 3600 s; see section 1, background knowledge). The TAP paper describes presubmit as best-effort for latency reasons. SWE Book ch. 11, TAP paper

Agent-era data point. Linear (engineer Mufeez Amjad, 21 September 2026, reported via a secondary write-up) says agentic coding turned CI into a bottleneck. The test suite grew about fourfold since January 2026 and pull-request wait times exceeded six minutes. Linear brought the wait back to about five minutes despite the growth, using faster runners, a native TypeScript compiler, cheaper lint rules, faster change detection (26 s to 8 s median), Vitest shards raised from 4 to 8, and isolate: false for modules safe to share. Linear treated roughly five to six minutes as the threshold worth engineering against. Only the secondary source was read; the primary Linear post was not fetched. explainx.ai summary

Ratios. No source found states a ratio of test time to implementation time. Every published budget is an absolute ceiling (about 10 minutes for the gating stage) combined with staging. A ratio tied to implementation time would be a new rule, not an established one (inference).

4. AI coding agents and test-running budgets (2025–2026)

Main finding: no agent system found bounds verification effort relative to the size of the change. The bounds that exist are per-command wall-clock timeouts, which protect the harness from hangs. They are not a judgement about how much testing a change deserves. The guidance that does exist tells the agent to run narrowly.

Claude Code. Anthropic's official best-practices page recommends documenting in CLAUDE.md that the agent should "prefer running single tests, and not the whole test suite, for performance", and grounds the advice in context-window cost: every command's output fills the context and degrades later work. This is a scoping instruction, not a budget. Claude Code best practices (page could not be fetched directly during this research; wording taken from the search-result extract.) Observed in the harness used for this research: the Claude Code Bash tool defaults to a 120-second command timeout with a 600-second maximum. A suite that runs for tens of minutes therefore cannot run as a single foreground command and must be backgrounded. That is a hard ceiling on one command, not on verification as a whole.

OpenHands. Per-command execution timeout is configurable, but it only applies when the model's command action supplies an explicit timeout. Commands that omit one fall back to the sandbox-level timeout, and a 2026 pull request in NVIDIA NeMo Gym exports both variables so the defaults agree. An older OpenHands issue (#1637) asks for better and more flexible sandbox-timeout handling. NeMo Gym PR #3578, OpenHands issue #1637

Open SWE (LangChain). A pull request enforces the documented 300-second default for sandbox commands when callers omit a timeout. open-swe PR #2649

OpenAI Codex. The Codex repository's AGENTS.md gives hooks a 600-second default timeout. A third-party issue documents a sandbox-policy rejection that never surfaced to the launcher, so the run sat until a 300-second hard kill. It is a concrete case of a timeout serving as the only failure signal, which costs the full budget instead of failing fast. codex AGENTS.md, larch issue #5533

Agents burning time in test loops. Teemu Piirainen, "Your AI Coding Agents Are Slow Because Your Tools Talk Too Much" (DEV Community, 7 February, year not stated, 2026 by context) measured a validator agent taking 608 s to report on a frontend suite that ran in 96 s (6.3x), and 224 s on a 24 s backend suite (9.3x). The cause was 419 KB of human-formatted Vitest output: the agent's context truncated the five numbers it needed, so it re-ran commands. Switching to --reporter=json, discarding noise and extracting results with jq cut tool calls from 12 to 1 (backend) and 28 to 1 (frontend), brought agent time to 42 s and 66 s, and stopped false failure reports. The lesson for a budget rule is that agent verification time is suite runtime multiplied by re-runs, and output format drives the re-runs. dev.to

CI-side strain from agent volume. Anthropic (section 1) and Linear (section 3) both report agent-driven change volume overwhelming CI in 2026, and both responded with selection, sharding and infrastructure changes, not with fewer tests. Vendor posts (e.g. getautonoma.com (link no longer available), minitap.ai) describe a "test suite graveyard" pattern in which tests break faster than teams fix them and are skipped, then disabled. These are marketing sources with unsourced figures; treat them as anecdote.

Gaps. Devin, Cursor, Aider and SWE-agent were not researched in depth for this report. Nothing was found showing any of them ties test effort to change size, but absence was not exhaustively checked.

5. Making database-backed suites fast, and the case against elaborate infrastructure

Sourced during this research: Fowler's commit build stays fast by using test doubles and leaves real-database tests to secondary stages (martinfowler.com). Linear cut its critical shard time from about 300–379 s to about 195 s partly by turning off per-file isolation (isolate: false) for modules that could safely share setup (explainx.ai). The SWE Book names "booting system portions" as a common cause of slow tests (ch. 11).

The techniques below are standard practice from background knowledge and were not re-fetched during this research. The links are canonical, but verify them before citing externally.

• Build the database once, isolate per test with a transaction. Django's TestCase and Rails' transactional tests wrap each test in a transaction and roll it back, so schema setup happens once per run instead of once per test. Django testing docs
• Clone instead of rebuilding. PostgreSQL's CREATE DATABASE ... TEMPLATE copies a pre-migrated database far faster than re-running migrations. It suits suites that need a fresh database per worker or per test file. PostgreSQL docs
• Share one container per run. Testcontainers supports a single container shared across a test session (the "singleton container" pattern) in place of one per test. Testcontainers
• Functional core, imperative shell. Gary Bernhardt, "Boundaries" (2012): put decisions in pure functions tested with plain values and keep I/O in a thin shell covered by a few integration tests. A small fix to decision logic is then verifiable in milliseconds. destroyallsoftware.com
• Snapshot or approval tests of the pure core. Record the output for a set of inputs once, then diff against the recording on later runs (Jest snapshots; approval testing). This is cheap when the core is pure and brittle when it is not.
• Test doubles at the persistence boundary. A repository interface with an in-memory fake lets domain tests skip the database entirely. Fowler's "solitary versus sociable" unit-test distinction is the vocabulary. martinfowler.com/bliki/UnitTest

The case against elaborate test infrastructure for one project (background knowledge). David Heinemeier Hansson, "Test-induced design damage" (2014), argued that restructuring code around mocks and layers to make tests fast can damage the design more than slow tests cost. His preferred remedy is real-database tests made fast by transactional rollback, not abstraction. dhh.dk Fowler's 2021 test-shapes essay makes the related point that teams should judge tests by speed and determinism, not by an ideal shape. Inference: for a single project, the cheapest fixes usually come in this order. First, make the database setup shared and roll back each test, which changes fixtures only. Second, add a selection mechanism such as file-to-test mapping, testmon or --findRelatedTests. Third, extract a pure core where fixes concentrate. Building a bespoke snapshot/replay harness or a fake persistence layer is justified only if the project is long-lived and changed often.

What this suggests for a verification-budget rule

Options are listed with their basis. "Inference" marks a step that no source states directly.

Measuring change size.

• Lines and files changed in the diff. This is cheap to compute. No source uses implementation time as the measure. Inference: implementation time is a noisy proxy, because a three-minute fix can be high-risk, so diff size plus touched-module criticality is the better basis.
• Blast radius via reverse dependencies: the tests that transitively depend on the changed files, as TAP does. This measures what the change can break, not how big it is, and it is the measure every selection system in section 1 actually uses.

Candidate ceilings.

• An absolute ceiling of about ten minutes for the gating verification, from the XP/Fowler ten-minute build and DORA's "upper limit of about 10 minutes". It is the best-sourced number available.
• Tiered ceilings by test size, borrowed from Bazel's defaults (60 s / 300 s / 900 s / 3600 s, background knowledge). A small change gets small and medium tests in the gate; large tests go to a later stage.
• A ratio such as "testing no more than N times implementation time" has no source behind it. Inference: if used, apply it as a lower trigger beneath the absolute ceiling, never as the only bound, because a 30-second fix should not get a 5-minute verification budget for a module whose fastest honest check takes 8 minutes.

What to do at the ceiling.

• Stage, don't drop: run the selected, fast subset as the gate and queue the full suite as a post-merge or scheduled secondary run, as Fowler's pipeline, TAP's post-submit and Meta's accepted miss rate all do. The change can be accepted on the gate. A secondary-stage failure reopens the change but does not block other work.
• Select before running: map the diff to tests, by import graph, coverage or a hand-written module-to-test map, before a testing agent starts. Meta's results show the miss rate this permits is small (over 99.9% of faulty changes still reported at half the cost).
• Make output machine-readable (JSON reporters) so agent time approaches suite runtime and does not reach 6–9x of it (Piirainen).
• Fail fast when the harness rejects a command, so a timeout is not the only failure signal (the Codex sandbox issue).

Attributing a slow suite.

• A rule that says "a suite that cannot finish inside the budget is a finding against the suite" matches Fowler's practice of treating a slow build as something to act on, and of strengthening commit-stage tests whenever a later stage catches a bug. It also matches the SWE Book's ice-cream-cone diagnosis.
• Inference: record the finding with its shape evidence. Give the share of checks that build a sandbox database, the per-check setup cost, and whether any check exercises the changed code without full-system setup. That turns "slow" into one of three fixable defects: no fast layer (section 2), no selection (section 1), or per-test setup that should be per-run (section 5).
• Inference: keep the change's verdict and the suite's finding separate. The change passes or fails on the checks run inside the budget, plus the queued full run. The suite finding goes to whoever owns the suite, not to the agent that wrote the fix.

Sources

• Memon et al., "Taming Google-Scale Continuous Testing", ICSE-SEIP 2017, https://research.google.com/pubs/archive/45861.pdf
• Machalica, Samylkin, Porth, Chandra, "Predictive Test Selection", ICSE-SEIP 2019, https://arxiv.org/abs/1810.05286
• Sachin Malhotra, "Agentic coding is straining CI…", Claude blog, 2026-09-14, https://claude.com/blog/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic
• Adam Bender, "Testing Overview", Software Engineering at Google ch. 11, 2020, https://abseil.io/resources/swe-book/html/ch11.html
• Martin Fowler, "Continuous Integration", revised 2024-01-18, https://martinfowler.com/articles/continuousIntegration.html
• DORA, "Continuous integration" capability, https://dora.dev/capabilities/continuous-integration/
• Linear CI rework (Mufeez Amjad, 2026-09-21), secondary summary, https://explainx.ai/blog/linear-ci-rework-agentic-coding-bottleneck-2026
• Teemu Piirainen, "Your AI Coding Agents Are Slow Because Your Tools Talk Too Much", DEV, https://dev.to/teppana88/your-ai-coding-agents-are-slow-because-your-tools-talk-too-much-24h6
• Claude Code best practices, https://code.claude.com/docs/en/best-practices
• NVIDIA NeMo Gym PR #3578, https://github.com/NVIDIA-NeMo/Gym/pull/3578 ; OpenHands issue #1637, https://github.com/OpenHands/OpenHands/issues/1637 ; open-swe PR #2649, https://github.com/langchain-ai/open-swe/pull/2649
• openai/codex AGENTS.md, https://github.com/openai/codex/blob/5c18cc0acc3734f0e78e422a7fd94ea4a2be652e/AGENTS.md ; larch issue #5533, https://github.com/zhupanov/larch/issues/5533
• Background-knowledge links (not re-fetched during this research): Jest CLI, pytest-testmon, Bazel common definitions, Azure Pipelines TIA, Kent C. Dodds testing trophy, Spotify honeycomb, Google Testing Blog 2015, Fowler test-shapes 2021, Django TestCase, PostgreSQL template databases, Testcontainers, Bernhardt "Boundaries", Fowler UnitTest bliki, DHH "Test-induced design damage" (URLs inline above).