Skip to main content

Under constructionStatus: Building. Being built now. Parts may run; the whole is not finished.

Every project’s status →

01.03Voice sessions

VOX Presenter

AR Labs' VOX Presenter runs voice sessions about documents: a fast voice agent discusses a document aloud while a second agent writes analysis on screen.

On this page
  1. How it works
  2. What exists today
  3. Decisions
  4. Related projects

VOX Presenter is AR Labs' voice interface for talking through documents. It runs on VOX, the lab's working system: the AI agents (language models run in a loop with tools), skills and records we run our own work on. VOX Presenter lets a person open a voice session about any document in VOX, such as a design document, a research report or a build log, and discuss it aloud. While they talk, a second agent writes a fuller analysis on screen and passes what it finds back into the conversation, so the spoken discussion gets sharper as it goes.

It is an early prototype for our own use. The longer aim is a voice-controlled presentation assistant that explains material both by voice and visually, but the project started with the conversation because a discussion is a collaboration and a slideshow is a recording.

How it works

A session starts from a command in our workspace for Claude Code, Anthropic's coding agent (an AI agent that writes code). The command gathers the document and its context into a session file, starts the voice backend, and opens a browser page. The backend is a Pipecat voice pipeline: Deepgram turns speech into text, a fast conversational model writes the reply, and Cartesia speaks it. The browser connects to the backend over a direct WebRTC connection on the same machine.

Two agents take part, each in its own Pipecat worker, coordinated through Pipecat's multi-worker message bus:

  • The voice agent holds the spoken conversation. It needs a fast time to first word, so it runs on a small, quick model.
  • The scribe runs on Claude Opus 4.6, outside the spoken path. After each exchange it receives the question and answer, writes a structured analysis that appears in a panel beside the transcript, and returns a short summary that is added to the voice agent's context.

Everything runs on one machine: a launcher skill, a browser page, and one backend process that holds both agents on a local message bus, with a separate Claude Code process for the voice grill described below.

C4 container diagram: launcher skill, browser page, backend with voice and scribe workers, workspace files and the grill fork; Deepgram, GPT-4o-mini, Cartesia and Claude Opus 4.6 outside. A person starts a session with the voice command in a Claude Code workspace. The launcher (a Claude Code skill) writes the session context, starts the voice session backend and opens the browser frontend (Vite and React). Browser and backend talk over a local WebRTC connection: microphone audio and typed lines go in; the agent's voice and the scribe's notes come back. The backend is one Python process running Pipecat with two workers on a message bus: a voice worker that uses Deepgram for speech-to-text, GPT-4o-mini for replies and Cartesia for speech, and a scribe worker on Claude Opus 4.6 that writes the analysis and feeds a short summary back. The backend reads documents in the workspace and appends the transcript and summary as the session runs. In voice grill mode it drives a read-only Claude Code discussion fork over stream JSON, and a one-shot finish step writes the confirmed brief. VOX Presenter: containers C4 LEVEL 2 · A VOICE SESSION ABOUT A DOCUMENT · ONE MACHINE, TWO AGENTS ON A MESSAGE BUS FIGURE VOX PRESENTER CONTAINERS DRAWN BY AR LABS SHEET 1 / 1 Person [Person] Talks a document through VOX PRESENTER [Software system, runs locally] Launcher [Container: Claude Code skill] Writes the session context, starts the backend, opens the page Browser frontend [Container: Vite + React] Transcript, scribe panel, typed input, voice grill screen Voice session backend [Container: Python + Pipecat] Voice worker: speech in, reply, speech out; read-only file tool Scribe worker: analysis after each exchange Both on a local message bus Workspace files [Files on disk] Documents; transcript and summary appended per turn Grill fork + finish [Container: Claude Code CLI] Read-only discussion in a snapshot; finish writes the brief Deepgram [External: speech-to-text] GPT-4o-mini [External: voice model] Cartesia [External: text-to-speech] Claude Opus 4.6 [External: scribe model] KEY Container the lab built Files on disk Person External service or app System boundary Call or data flow, labelled types the voice command speaks, listens, reads opens starts mic audio, typed lines voice, scribe notes [local WebRTC] calls [HTTPS, WS] reads, appends drives turns [stream JSON] writes the brief
A voice session about a document: one backend process, two agents on a message bus, everything on one machine.

The voice agent can also read other files during a session through one tool, read_file, which is confined to the workspace and cannot write. The transcript and the scribe's analysis are appended to two files in the workspace after every exchange, so a session that stops unexpectedly keeps everything said up to that point. A session ends when the person says so: the agent speaks a summary and the person confirms it.

The voice grill

A second mode, the voice grill, runs a structured design interview by voice. A read-only copy of a Claude Code session prepares the questions about a design. The voice agent presents each question, discusses it, and drafts an answer from what was said. The person confirms or edits the draft before it is sent back. The screen shows the question inside nested frames for the part of the design it belongs to, and the frame being discussed lights up in time with the voice: the agent writes silent markers into its reply, which are kept out of the speech and released at the moment the matching words are played. Nothing is written to the project until the interview finishes.

What exists today

The voice session pipeline is built and runs locally: document-fed conversation, typed input alongside speech, the scribe and its feedback into the conversation, workspace-confined file reading, explicit close with confirmation, reconnection handling, and session output appended to disk as the session runs. The voice grill is built, including its question flow, draft and confirmation, synchronised frames, and filtering of every result before it reaches the screen or a file.

The first slice of a third mode, the voice explainer, is built: a canvas in the browser lays out a set of items, the voice walks through them one per turn and lights each as it speaks, and the person can queue any item to hear about next. Drawing the relations between items, side answers, saved transcripts and reconnection are not in it yet.

It is a personal prototype, around version 0.1, and has not been tested with other users. It is not deployed and runs on one machine.

Planned and not built: the rest of the voice explainer; remote and mobile access; and the original visual layer of generated slides and diagrams with narration.

Decisions

Conversation before slides. The first research described a full presentation system with slide schemas and rendering. During design review the value was placed in the spoken discussion, so the slides moved to a later phase. If the conversation proves useful, visuals are an addition to a working foundation; if it does not, little was spent on rendering.

Two agents with opposite profiles. A spoken reply has to start quickly, while good analysis takes time. Splitting the work lets the voice model be chosen for speed and the scribe for depth, and keeps the slow model out of the path the listener hears.

A message bus as the boundary between agents. The first design used the Claude Agent SDK for the intelligence layer. Research found its subprocess start-up, about 12 seconds, and its large fixed token overhead unsuitable for a voice session. Pipecat's multi-worker framework gives a typed message bus between workers instead, so either agent's backend can change without touching the voice path.

Model choice tested in use. The design first chose a Llama model served by Groq for the voice agent, for its very fast first word. In live testing the hosted model was retired without notice and the free tier's rate limits rejected the size of a document-fed prompt, so the default moved to OpenAI's GPT-4o-mini, with Groq kept as a setting.

A browser, not local audio. An early design used the computer's microphone and speaker with no screen. It was dropped because the scribe's analysis needs somewhere to appear while the conversation runs.

Read freely, write only at the end. In the voice grill, the session preparing questions can read the project but cannot change it during the discussion. Writing happens once, at the finish, after the person has confirmed each answer.

VOX Facilitator is our design for a voice agent that runs a whole planning session over a folder of documents, built on this voice loop. Salesbot is the public browser voice agent that runs a structured sales discovery conversation, and Voice UI is the generic voice interface taken from it.