Testing a browser voice agent: what counts as evidence

This page describes how AR Labs tests a browser voice agent, and how we decide what a test result entitles us to claim. The example throughout is Salesbot, our public voice agent for sales discovery, built on Pipecat, an open-source Python framework for real-time voice agents. A voice agent is an AI agent (a language model run in a loop with tools) with speech recognition on the way in and speech synthesis on the way out. Most of what it does can be tested without anyone speaking; the parts a visitor hears cannot, and a report that blurs the two overstates what is known. The page is for anyone building or buying a voice agent.

Two kinds of evidence

A voice agent has two layers that fail in different ways. The conversation layer decides what to ask, what to record and when to move on; it works on text. The audio layer turns speech into text and text back into speech, in time for the reply to feel natural. The conversation layer can be driven by typed lines and checked by code. The audio layer has to be heard.

flowchart TD
  T["Typed or replayed lines"] --> C["Conversation layer: stages, recording, closing"]
  C --> AT["Automated tests and scored replays"]
  M["Microphone and speakers"] --> AU["Audio layer: recognition, timing, voice"]
  AU --> L["A person listening"]
  AT --> R["What may be claimed"]
  L --> R

What typed sessions and automated tests can prove

In Salesbot, a typed line enters the pipeline exactly where a finished transcript of speech would arrive, so the agent treats it as speech. That one design choice makes most of the agent testable without audio. The current release has 747 automated test functions in 35 files for the voice server and 17 test files for the web page. They cover the behaviour a visitor depends on:

Behaviour	How it is tested
Stage progression	A test walks the conversation graph the server actually builds, and others check that a stage completes only when code finds every required answer recorded
Structured extraction	The handler that records answers rejects blank, placeholder and unknown values before storing anything
Clarifying missing answers	After each recording the agent is told which answers are still missing; a test replays a real call that motivated this, with no model involved
No answer before the visitor speaks	Nothing is recorded for a stage until the visitor has replied in it; the test uses the agent's own recorded failures as fixtures
Confirmation and closing	Tests check that the end of the call is driven by code after the visitor confirms, so it does not depend on the model remembering to hang up
Safety limits	A stage that runs too long moves on, and answers already given are recovered
Persistence	Each completed stage is written to the database, and the full record when the call closes
The page and server agreeing	Contract tests keep the web page's list of stages and limits identical to the server's

Two techniques do most of the work. Replaying recorded failures without a model: a real call that went wrong becomes a test fixture, so the fix is checked against the exact case that exposed the problem. Silent replay: saved lines are fed to the running agent as though spoken, with no microphone, so a conversation bug can be reproduced on demand. The replay enters at the same point as speech, because a shortcut past that point would hide the turn-ordering bugs that live there.

Scored replays against the real model

Unit tests check our code. The language model's behaviour has to be measured, because no test can make a model reliable. We replay the visitor's side of saved conversations through the whole agent on a chosen model and record every answer the model writes down. A scorer then sets each recorded value beside what the visitor actually said in that stage. Values that match the visitor's words are marked automatically; a person judges the rest as a fair paraphrase or as something the visitor never said. The same run records each reply's time to first output.

These replays, with a person's listening calls, informed the choice of model for the agent. On the same replays, Claude Haiku 4.5 produced its first words in a median of 611 ms against 1,101 ms for GPT-4o-mini, and at the 95th percentile in 880 ms against 3,345 ms. On a reproduction of a known failure, GPT-4o-mini stored answers the visitor never gave, while on three replays of it Haiku stored none and asked for the missing point instead. These are small samples, recorded as single observations rather than rates. A fixed script has a further limit: a model that asks more questions can run out of scripted lines before a stage completes.

What needs a person listening

Some qualities exist only as sound, and no typed session reaches them:

• speech recognition accuracy, including names, numbers and accents;
• latency as the visitor hears it, from the end of their speech to the start of the agent's audio;
• voice quality and pronunciation;
• interruption: whether the agent stops cleanly when the visitor talks over it;
• behaviour under silence, background noise and overlapping speech;
• microphone permission and device selection in real browsers.

People have listened to Salesbot calls, and listening found problems the tests had not, such as a model that asked two questions in a row. Each listening call is still one observation. A repeatable check, recorded and committed, is the third tier of our evidence of completion rule: a screenshot or recording for what only a person can judge.

A latency budget as a design tool

Before measuring, we set a budget for one turn: a typical time for each step from the visitor's audio reaching the server to the agent's audio reaching the visitor, against a total target of under 1.5 seconds. The model's figure was measured on replays; the others are typical published figures. The budget is useful before any measurement exists because it shows where time can go. In Salesbot's case most of the spread is network distance between the server and the speech and model services, which pointed to moving the voice server nearer to them rather than tuning code. The Salesbot page gives the full table. A budget is a design estimate; it is not a measurement of what a visitor hears.

What has and has not been verified

In a controlled session on 2 November 2026, run against the live demo with typed replies because the test browser had no usable microphone, the following were observed: the call started; the agent greeted the visitor; typed replies were accepted; answers were captured and shown during the call; the first stage completed and the second opened without intervention; a long reply filled several answers at once; the agent asked a specific follow-up for the one missing answer rather than inventing it; it read the answers back and asked whether they were right; it closed the call after confirmation; the closing screen showed the summary; and the database held the completed conversation with both stages recorded, alongside the conversation log and per-turn trace.

That session did not test speech recognition, interruption, latency as heard, voice quality or the follow-up email. None of the audio qualities listed in the previous section has yet been checked with a recorded, repeatable test.

Stating what was verified

Three rules keep a voice agent's test report true. Name the layer: "the conversation completes its stages from typed input" is a different claim from "the agent understands speech". Name the sample: one session, three replays, or a rate over many runs. And date the claim, because a release changes what was verified. Where a claim rests on a person's judgement, say so and keep the recording or the judgement file beside the code, so someone else can check it.