Skip to main content

05.03Voice AI

Pipecat voice agent architecture: pipeline, STT, TTS and latency

How to build a real-time voice agent on Pipecat with Daily WebRTC, Deepgram, ElevenLabs and LangGraph: configuration, latency budgets and known pitfalls.

Plate: a waveform passes through three discs, speech-to-text, model and text-to-speech, above a segmented latency budget bar.A waveform enters a pipeline of three discs, speech to text, language model and text to speech, joined by a bar with frames on it, and leaves as a waveform. A segmented bar below shows each stage's share of the latency budget.STTLLMTTSLATENCY BUDGET, ONE TURNR-01VOICE AIMOTIF: WAVEFORMS2026 · 03 · PLATE 01 / 30FINDINGEVERY STAGE SPENDS THE LATENCY BUDGET
On this page
  1. Summary
  2. 1. Pipecat pipeline architecture
  3. 2. Deepgram STT integration
  4. 3. ElevenLabs TTS integration
  5. 4. LLM processing and LangGraph integration
  6. 5. Latency optimisation practice
  7. 6. Configuration gotchas and common pitfalls
  8. 7. Recommended tech stack for a lead-qualification voice agent
  9. Limitations
  10. Sources

How should a voice agent that qualifies sales leads with the SPIN selling question sequence be built on the open-source Pipecat framework? AR Labs researched this question in March 2026, as groundwork for Salesbot, our Pipecat voice agent for sales discovery, and the reusable Voice UI taken from it. The report draws on official Pipecat documentation, API references, GitHub issues and practitioner write-ups to cover pipeline architecture, speech-to-text and text-to-speech configuration, LangGraph integration, latency optimisation and common pitfalls.

Method: nine parallel web searches, 40+ sources (official docs, GitHub, tutorials), with a confidence rating per section.

Summary

Pipecat is an open-source Python framework by Daily for building real-time voice and multimodal conversational AI agents. It provides frame-based pipeline architecture with built-in support for WebRTC (via Daily.co), STT (Deepgram, etc.), TTS (ElevenLabs, etc.), and LLM processing (OpenAI, Anthropic, custom LangGraph agents). Pipecat aims for 500 to 1,000 ms from end of speech to reply.

1. Pipecat pipeline architecture

Frame-based architecture (high confidence)

Core concept:

  • Frames are data containers that flow through the pipeline like packages on a conveyor belt
  • Each frame contains specific typed data (audio, text, images, system control)
  • Frames receive unique identifiers (e.g., TranscriptionFrame#1) for debugging and tracking

Frame types:

# Common frame types
- AudioFrame          # Raw audio data (from user or TTS)
- TextFrame           # Transcriptions, LLM responses
- TranscriptionFrame  # STT output
- ImageFrame          # Visual data for multimodal
- SystemFrame         # Control signals (immediate processing, bypass queues)
- DataFrame           # Standard data (queued, ordered processing)
- ControlFrame        # Pipeline control (queued, ordered processing)

Processing priority:

  • SystemFrame: Processed immediately, bypassing queues (interruptions, errors)
  • DataFrame and ControlFrame: Queued and processed in order

Sources:

Pipeline composition patterns (high confidence)

Basic pipeline:

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineTask
 
# Pipelines are lists of frame processors linked together
pipeline = Pipeline([
    transport.input_processor(),   # Input from Daily.co/WebSocket
    stt_service,                   # Speech-to-text (Deepgram)
    llm_service,                   # LLM processing (OpenAI/LangGraph)
    tts_service,                   # Text-to-speech (ElevenLabs)
    transport.output_processor(),  # Output to Daily.co/WebSocket
])
 
# PipelineTask orchestrates lifecycle
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
)

The basic pipeline as a diagram, with the services this report covers:

FIG. 01
flowchart LR
  IN["Transport input (Daily WebRTC)"] --> STT["STT (Deepgram)"]
  STT --> LLM["LLM (OpenAI, Anthropic or LangGraph)"]
  LLM --> TTS["TTS (ElevenLabs)"]
  TTS --> OUT["Transport output (Daily WebRTC)"]

Directional flow:

  • DOWNSTREAM: Source → Sink (normal processing flow)
  • UPSTREAM: Sink → Source (feedback, events, control signals)

Nested pipelines:

  • A Pipeline is itself a FrameProcessor, enabling composition
  • Create complex processing graphs by nesting pipelines

Parallel processing: ParallelPipeline (from pipecat.pipeline.parallel_pipeline) creates branches, and each branch receives all upstream frames.

Sources:

Daily.co WebRTC integration (high confidence)

Setup pattern:

from pipecat.transports.services.daily import DailyTransport, DailyParams
 
transport = DailyTransport(
    room_url="https://your-daily-domain.daily.co/room-name",
    token=daily_room_token,  # From Daily REST API
    bot_name="Sales Bot",
    params=DailyParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        vad_enabled=True,  # Voice Activity Detection
        vad_analyzer=SileroVADAnalyzer(),  # Built-in VAD
    ),
)

Room creation (backend): a typical /start_bot endpoint creates a private Daily room through the Daily REST API (https://api.daily.co/v1/rooms), optionally generates a meeting token, spawns a bot process for that room, and returns the room URL to the frontend. In the browser, the Daily SDK handles the WebRTC connection automatically.

Built-in features:

  • Echo cancellation and noise reduction
  • Device management
  • Event abstractions
  • WebRTC media stream handling

Transport event handlers (on_joined, on_participant_joined, on_participant_left, on_error) drive the session lifecycle: greet on join, end the pipeline when the participant leaves. For custom clients, WebsocketServerTransport offers the same audio and VAD parameters over a plain WebSocket.

Sources:

2. Deepgram STT integration

Deepgram streaming configuration (high confidence)

Two service options:

  1. DeepgramSTTService (Standard):
from pipecat.services.deepgram import DeepgramSTTService
 
stt = DeepgramSTTService(
    api_key="YOUR_DEEPGRAM_API_KEY",
    url="wss://api.deepgram.com/v1/listen",  # WebSocket endpoint
    live_options={
        "language": "en-US",
        "model": "nova-2",           # Latest Deepgram model
        "smart_format": True,        # Automatic formatting
        "interim_results": True,     # Real-time partial transcripts
        "endpointing": 300,          # ms of silence to finalize (CRITICAL)
        "vad_events": False,         # DEPRECATED - use Silero VAD instead
    },
)
  1. DeepgramFluxSTTService (Advanced Conversational):
from pipecat.services.deepgram import DeepgramFluxSTTService
 
stt_flux = DeepgramFluxSTTService(
    api_key="YOUR_DEEPGRAM_API_KEY",
    eager_eot_threshold=0.5,         # Eager end-of-turn detection
    eot_threshold=0.8,               # Standard end-of-turn threshold
    keyterms=["Pipecat", "Deepgram"], # Boost recognition of specific terms
)

Sources:

Interim results and endpointing (high confidence)

Pipecat handles interim and final transcripts automatically. Interim results are partial transcripts while the user is speaking; final results are triggered by VAD silence detection plus endpointing. The endpointing value (for example 300ms of silence marks the end of an utterance) is the main trade-off: lower is faster but may cut the user off mid-sentence, higher is safer but slower to respond.

VAD integration: Pipecat recommends Silero VAD (SileroVADAnalyzer, set on the transport) over Deepgram's built-in VAD. When VAD detects the user stopped speaking, Pipecat sends a finalise request to Deepgram, Deepgram returns the final transcript faster, and the pipeline proceeds to LLM processing.

Turn detection with Flux: DeepgramFluxSTTService provides its own turn detection (StartOfTurn/EndOfTurn events) and broadcasts UserStartedSpeakingFrame/UserStoppedSpeakingFrame. Use ExternalUserTurnStrategies.EXTERNAL_ONLY so VAD-based turn management does not conflict with it (see Pitfall 3 below).

Sources:

Latency practice (high confidence)

Critical latency settings:

# OPTIMAL DEEPGRAM CONFIG FOR LOW LATENCY
live_options = {
    "model": "nova-2",              # Fastest accurate model
    "interim_results": True,        # Enable streaming
    "endpointing": 200,             # AGGRESSIVE: 200ms silence (vs 300ms default)
    "smart_format": True,           # Minimal processing
    "punctuate": True,              # Helps LLM understand intent
}

The full voice-to-voice latency breakdown is in section 5.

Sources:

3. ElevenLabs TTS integration

ElevenLabsTTSService configuration (high confidence)

Two service options:

  1. ElevenLabsTTSService (WebSocket, RECOMMENDED):
from pipecat.services.elevenlabs import ElevenLabsTTSService
 
tts = ElevenLabsTTSService(
    api_key="YOUR_ELEVENLABS_API_KEY",
    voice_id="21m00Tcm4TlvDq8ikWAM",  # Rachel voice (example)
    model="eleven_turbo_v2_5",        # DEFAULT: Latest turbo model
                                      # eleven_turbo_v2 still supported
    params=InputParams(
        language="en",                # Language code
        stability=0.5,                # 0.0-1.0 (lower = more variable/expressive)
        similarity_boost=0.75,        # 0.0-1.0 (higher = closer to voice sample)
        style=0.0,                    # 0.0-1.0 (style exaggeration)
        use_speaker_boost=True,       # Enhance clarity
        speed=1.0,                    # 0.7-1.2 (playback speed)
    ),
)
  1. ElevenLabsHttpTTSService (HTTP, simpler, no interruptions): same api_key, voice_id and model arguments, but the HTTP service lacks word-level timestamps and interruption handling.

Sources:

Voice ID configuration (high confidence)

Voice IDs come from the ElevenLabs Voice Library (https://elevenlabs.io/voice-library) or, for cloned voices, from the https://api.elevenlabs.io/v1/voices endpoint. Common pre-made voice IDs: Rachel 21m00Tcm4TlvDq8ikWAM, Domi AZnzlk1XvdvUeBnXmlld, Bella EXAVITQu4vr4xnSDxMaL.

Voice quality parameters: stability runs from 0.0 (very variable, expressive) to 1.0 (very stable, monotone); similarity_boost from 0.0 (less like the original voice) to 1.0 (very close to it); style from 0.0 (neutral delivery) to 1.0 (exaggerated style and emotion); use_speaker_boost enhances clarity and similarity.

Sources:

eleven_turbo_v2 vs eleven_turbo_v2_5 (high confidence)

eleven_turbo_v2 (older) is fast, low latency, good quality and still supported. eleven_turbo_v2_5 is the current default: even faster than v2, with improved quality and naturalness and better handling of edge cases, and is recommended for new projects. Configuration is identical apart from the model string.

Sources:

WebSocket vs HTTP trade-offs (high confidence)

WebSocket (ElevenLabsTTSService) HTTP (ElevenLabsHttpTTSService)
Word-level timestamps (critical for interruptions) Yes No
Real-time streaming audio Yes No (full audio generated before playback)
Interruption handling (user can interrupt bot mid-sentence) Yes No
Setup Slightly more complex Simpler, easier debugging

Recommendation: Use the WebSocket service for a sales voice agent, where interruptions are critical.

Sources:

4. LLM processing and LangGraph integration

Custom processor pattern (medium confidence)

Creating custom LLM processors:

from pipecat.processors.frame_processor import FrameProcessor, FrameDirection
from pipecat.frames.frames import Frame, TextFrame, LLMMessagesFrame
 
class CustomLLMProcessor(FrameProcessor):
    """Custom processor for LangGraph integration"""
 
    async def process_frame(self, frame: Frame, direction: FrameDirection):
        if isinstance(frame, LLMMessagesFrame):
            user_message = frame.messages[-1]["content"]
            # Call your LangGraph agent
            response = await self.langgraph_agent.process(user_message)
            # Emit response as TextFrame
            await self.push_frame(TextFrame(response), direction)
        else:
            # Pass through other frame types
            await self.push_frame(frame, direction)

Sources:

LangGraph integration (medium confidence)

Example: policy-aware voice agent with LangGraph (pattern from the WebRTC Ventures guide, January 2026):

from langgraph.graph import StateGraph
from pipecat.processors.frameworks.langchain import LangchainProcessor
 
class ConversationState(TypedDict):
    messages: list
    current_stage: str  # e.g., "situation", "problem", "implication", "need"
    user_context: dict
 
workflow = StateGraph(ConversationState)
 
# Add nodes for SPIN stages
workflow.add_node("situation", situation_questions_node)
workflow.add_node("problem", problem_questions_node)
workflow.add_node("implication", implication_questions_node)
workflow.add_node("need_payoff", need_payoff_questions_node)
 
workflow.add_edge("situation", "problem")
workflow.add_edge("problem", "implication")
# ... etc
 
app = workflow.compile()
 
# A compiled LangGraph graph is a LangChain runnable
langchain_processor = LangchainProcessor(chain=app)
 
pipeline = Pipeline([
    transport.input_processor(),
    stt_service,
    langchain_processor,  # LangGraph integration
    tts_service,
    transport.output_processor(),
])

Architecture separation:

  • LLM layer: Interprets user intent from transcripts
  • Decision layer (LangGraph): Evaluates policy, routes conversation flow
  • Action layer: Executes approved actions (e.g., database queries, API calls)

Sources:

State passing between frames (medium confidence)

LangGraph state is maintained within the graph's checkpointer; Pipecat passes messages via LLMMessagesFrame. The flow is: user speaks, STT emits a TranscriptionFrame, Pipecat creates an LLMMessagesFrame with conversation history, LangchainProcessor extracts the messages, LangGraph processes them with its internal state, and the response travels as a TextFrame to TTS. Inside the graph, LangGraph's InjectedState gives a tool read access to the current graph state, and returning a Command(update={"current_stage": "problem"}, goto="problem_node") updates state and routes in one step.

Sources:

SPIN state machine shape

The worked example in the source research models the SPIN conversation as a linear LangGraph state machine with six nodes: greeting, situation, problem, implication, need-payoff and close. Each node calls the LLM with a stage-specific system prompt asking two or three questions of that type, and a simplified transition rule advances the stage after a fixed number of assistant turns (the source notes that production code should use the LLM to classify readiness instead).

def create_spin_graph():
    workflow = StateGraph(SPINState)
    for name, fn in [("greeting", greeting_node), ("situation", situation_node),
                     ("problem", problem_node), ("implication", implication_node),
                     ("need_payoff", need_payoff_node), ("close", close_node)]:
        workflow.add_node(name, fn)
    # Linear SPIN flow
    workflow.set_entry_point("greeting")
    workflow.add_edge("greeting", "situation")
    workflow.add_edge("situation", "problem")
    workflow.add_edge("problem", "implication")
    workflow.add_edge("implication", "need_payoff")
    workflow.add_edge("need_payoff", "close")
    workflow.add_edge("close", END)
    return workflow.compile()

LangSmith tracing integration (high confidence)

Observability for Pipecat plus LangGraph comes from attaching a PipecatLangsmithTracer (from pipecat.tracing.langsmith) to the PipelineTask as an observer, with enable_metrics=True. Langfuse offers an equivalent integration.

Sources:

5. Latency optimisation practice

Target metrics (high confidence)

Voice-to-voice latency breakdown (user stops speaking until bot starts speaking):

Step Latency
1. VAD detects end of speech 10-50ms
2. STT finalises transcript 50-150ms (with endpointing=200)
3. LLM processes and generates 200-800ms (model-dependent)
4. TTS generates first audio chunk 100-300ms (streaming)
Total 360-1300ms

Target: 800ms median for natural conversation. Acceptable: 1500ms for a proof of concept or MVP.

Sources:

Optimisation strategies (high confidence)

1. Use latency-optimised models. STT: Deepgram Nova-2 with endpointing: 200 (aggressive; default 300). LLM: fast models over quality for real-time use (OpenAI gpt-4o-mini or gpt-3.5-turbo, Anthropic claude-3-haiku, or local Llama 3.1 8B if a GPU is available). TTS: ElevenLabs eleven_turbo_v2_5.

2. Optimise VAD settings:

from pipecat.vad.silero import SileroVADAnalyzer, VADParams
 
vad = SileroVADAnalyzer(
    sample_rate=16000,
    params=VADParams(
        threshold=0.5,                # Lower = more sensitive (faster detection)
        min_speech_duration_ms=100,   # Minimum speech to trigger
        min_silence_duration_ms=200,  # Minimum silence to end turn
    ),
)

3. Enable preemptive generation (advanced). Start LLM processing before the turn officially ends, which saves 200-500ms. Pipecat and LiveKit handle this automatically with optimised settings. With custom processors, monitor interim STT results, detect likely end-of-turn patterns, start LLM processing early, and cancel if the user continues speaking.

Sources:

4. Shorten LLM context. Don't send the full conversation history every time; summarise or truncate older messages, for example keeping the system message plus the last ten messages (five exchanges).

Sources:

5. Use word-level timestamps for interruptions. ElevenLabs, Cartesia and Rime support word-level timestamps, which are required for natural interruption handling; the ElevenLabs WebSocket service provides them automatically. Pipecat uses them to align conversation context with what the user actually heard and to resume gracefully after an interruption.

Sources:

6. Host models in the same region or cluster. Co-locating STT, LLM and TTS in the same data centre (for example a Modal deployment on one GPU cluster) reduces inter-service latency by 50-200ms.

Sources:

6. Configuration gotchas and common pitfalls

Critical issues (high confidence)

1. Python-only server architecture. Pipecat is fundamentally a client/server system. Bot logic runs in a Python server process, is not embeddable in mobile apps (no native iOS/Android support), and must run on a backend (AWS, Modal, Fly.io, etc.). Solution: deploy the bot to a cloud server, connect the frontend via WebRTC (Daily.co) or WebSocket, and let mobile apps connect through the Daily iOS/Android SDKs.

Sources:

2. VAD configuration conflicts. Using both Deepgram VAD and Silero VAD causes conflicts, and Deepgram's vad_events is deprecated:

# WRONG:
stt = DeepgramSTTService(live_options={"vad_events": True})  # DEPRECATED
 
# CORRECT: Silero VAD in the transport, Deepgram VAD off
transport = DailyTransport(
    params=DailyParams(vad_enabled=True, vad_analyzer=SileroVADAnalyzer()),
)
stt = DeepgramSTTService(live_options={"vad_events": False})  # or omit

Sources:

3. Turn detection strategy conflicts. DeepgramFluxSTTService has its own turn detection, which conflicts with VAD-based turn management:

from pipecat.pipeline.strategies import ExternalUserTurnStrategies
 
pipeline = Pipeline(
    processors=[...],
    user_turn_strategy=ExternalUserTurnStrategies.EXTERNAL_ONLY,
    # Prevents VAD turn events from conflicting with Flux turn events
)

Sources:

4. Sample rate mismatch bug. Known bug: Pipecat 0.0.47 with XTTS produces a "cartoonic voice" because of a sample rate mismatch. Workaround: set sample_rate=24000 explicitly on the XTTS service, or downgrade to 0.0.46 until fixed.

Sources:

5. Global Interpreter Lock (GIL) on heavy workloads. The Python GIL limits parallel execution. Mitigation: use async/await throughout (Pipecat is async-first), offload heavy compute to C extensions (numpy, torch), and run multiple bot processes (one per conversation).

Sources:

6. STT transcript not visible in pipeline. STT output is handled "under the hood" by Pipecat and is hard to intercept for custom processing such as encryption. Workaround: a custom processor inserted after STT:

class TranscriptInterceptor(FrameProcessor):
    async def process_frame(self, frame, direction):
        if isinstance(frame, TranscriptionFrame):
            transcript = frame.text
            # Custom processing (e.g., encrypt, log, analyze)
        await self.push_frame(frame, direction)
 
pipeline = Pipeline([
    transport.input_processor(),
    stt,
    TranscriptInterceptor(),  # <-- Custom interception
    llm_processor,
    # ...
])

Sources:

7. Cross-cloud latency. AI models on Azure with the bot backend on AWS means high latency; network hops add 100-500ms. Solution: host all services in the same cloud region, or use Modal/Replicate for co-located GPU inference.

Sources:

Layer Choice Pros Cons
Transport Daily.co WebRTC Managed WebRTC, low latency, mobile SDKs Paid service ($0.004/min after free tier)
STT Deepgram Nova-2 Fast, accurate, streaming, affordable Requires API key ($0.0043/min)
TTS ElevenLabs Turbo v2.5 Natural voices, low latency, word timestamps Higher cost ($0.18/1K chars)
LLM Anthropic Claude 3 Haiku (or OpenAI GPT-4o-mini) Fast, good reasoning, affordable API dependency
Orchestration LangGraph Deterministic SPIN flow, easy debugging Learning curve
Deployment Modal or Fly.io Auto-scaling, GPU support, low latency Vendor lock-in
Monitoring LangSmith Built-in tracing for LangGraph Paid (has free tier)

Alternative stack (lower cost)

Layer Choice Pros Cons
Transport WebSocket (custom) No vendor lock-in, full control More dev work, handle WebRTC yourself
STT AssemblyAI or Whisper (local) Whisper is free if self-hosted, accurate Higher latency than Deepgram
TTS Azure TTS or Coqui XTTS (local) Lower cost, decent quality Coqui requires GPU, less natural than ElevenLabs
LLM Llama 3.1 8B (local via Modal) Free inference after GPU costs, fast Requires GPU ($0.001/sec), lower quality than Claude/GPT-4
Orchestration LangGraph (same)
Deployment AWS Lambda + ECS Pay-per-use, familiar tooling Cold starts, more config

Limitations

  • LangGraph integration examples are limited (newer integration, fewer public examples).
  • Some package versions may have been updated since the research date (March 2026); the source recorded pipecat-ai 0.0.47 as the latest stable release.
  • Production deployment costs are estimates; actual costs vary by usage.

Sources

Official Pipecat resources:

Tutorials and guides:

Service providers: