Skip to main content

05.04Voice AI

Sesame Conversational Speech Model (CSM) and open-source voice AI

How Sesame's CSM produces natural conversational speech, and which open-source frameworks, speech-to-speech and TTS models can build comparable voice agents.

Plate: two waveform ribbons braid like two speakers taking turns, meeting at a target of concentric discs.Two thick waveform ribbons, one violet and one ink, braid across the plate like two speakers taking turns, meeting at a target of concentric discs in the centre.R-02VOICE AIMOTIF: WAVEFORMS2026 · 02 · PLATE 02 / 30FINDINGSPEECH SHAPED BY THE WHOLE CONVERSATION
On this page
  1. Summary
  2. Part 1: how Sesame works
  3. Part 2: open-source voice AI in 2025
  4. Part 3: building voice AI of Sesame's quality
  5. Evidence gaps and limitations
  6. Methodology
  7. Key sources by category

How does Sesame's Conversational Speech Model achieve near-human conversational voice, and what do open-source models and frameworks offer for building voice agents of similar quality? AR Labs researched these questions in February 2026, as background for our voice agent work (see Salesbot and the Pipecat architecture report). The report analyses Sesame's published research and code alongside orchestration frameworks, speech-to-speech and text-to-speech models, and voice architecture patterns, with a confidence rating on each finding.

Method: standard depth, two search passes (breadth, then depth) plus direct reading of primary sources; 30+ sources with CRAAP scores; 18 high, 9 moderate and 3 low confidence findings.

Summary

Sesame AI's Conversational Speech Model (CSM) treats speech generation as one end-to-end multimodal task rather than the usual two-stage text-to-speech synthesis. The company released its 1B-parameter base model under the Apache 2.0 licence in March 2025, with sub-200 ms latency and near-human naturalness, which it attributes to three architectural choices: a dual-transformer design, split-RVQ audio tokenisation (the Mimi codec) and conversation-aware context processing.

On the open-source side in 2025, orchestration frameworks (Pipecat, LiveKit, TEN) and leading models (Qwen3-Omni, Moshi, VibeVoice, Kokoro-82M, Fish-Speech, CosyVoice) together let developers build voice agents of comparable quality. The main architectural developments are speech-to-speech (S2S) models that bypass cascading pipelines, neural audio codecs using residual vector quantisation, and orchestration frameworks that stream multimodal audio in real time with under 200 ms end-to-end latency.

Main finding: in 2025 open-source models came to match or exceed proprietary alternatives in naturalness, latency and multilingual capability, while running on consumer-grade hardware.

Part 1: how Sesame works

Core idea: the Conversational Speech Model (CSM)

Confidence: HIGH (3+ authoritative sources: Sesame Research, TechCrunch, GitHub)

Sesame's CSM fundamentally differs from traditional text-to-speech (TTS) systems by treating speech generation as a single-stage end-to-end multimodal learning task using transformers. This contrasts sharply with conventional cascading pipelines that first generate text representations, then synthesise audio separately.

Traditional TTS pipeline:

Text → Linguistic Analysis → Phoneme Sequence → Prosody Prediction → Waveform Synthesis

Sesame CSM approach:

Text + Audio Context → Unified Transformer Processing → Direct Audio Token Generation

The CSM uses 2 minutes of conversational history to adjust speech dynamically, enabling context-aware prosody, emotional intelligence, and natural speech patterns including micro-pauses, tonal variations, filled pauses ("um", "uh"), breath sounds, and mid-utterance tone modulation.

Dual-transformer architecture

Confidence: HIGH (Sesame Research, Towards Data Science, DigitalOcean)

The architecture employs two autoregressive transformers with a critical division at the "zeroth codebook" level:

  1. Backbone transformer (Llama-based):

    • Processes interleaved text and audio tokens with full conversational context
    • Trained on every audio frame to capture comprehensive conversation dynamics
    • Predicts the first (semantic) audio codebook

  2. Audio decoder transformer (smaller):

    • Reconstructs remaining acoustic codebooks (N-1 layers)
    • Trained on random 1/16 subset of frames (compute amortisation)
    • Dramatically reduces memory requirements without quality degradation

Model variants:

  • Tiny: 1B backbone + 100M decoder
  • Small: 3B backbone + 250M decoder
  • Medium: 8B backbone + 300M decoder

The publicly released CSM-1B uses the Tiny configuration under Apache 2.0 licence, available on GitHub and Hugging Face.

Mimi codec: split-RVQ audio tokenisation

Confidence: HIGH (Kyutai Codec Explainer, Sesame Research, Speechmatics)

Sesame uses the Mimi neural audio codec, a streaming codec developed by Kyutai for the Moshi speech-to-speech model. Mimi has since become the de facto standard tokeniser for conversational speech models including Sesame CSM, VoXtream, and LFM2-Audio.

Technical pipeline:

  1. Input processing:

    • Raw speech waveform at 24kHz
    • Series of strided convolution layers (stride factors: 4, 5, 6, 8, 2)
    • Downsample to 12.5 Hz frame rate
    • 4 convolutional blocks + final 1D convolution → dimension 512 latent representation

  2. Split-RVQ mechanism:

    • Semantic codebook (1st): Captures linguistic content, meaning, phonetic information
    • Acoustic codebooks (N-1): Encode speaker identity, prosody, emotion, fine-grained audio quality
    • Produces "one semantic codebook and N-1 acoustic codebooks per frame at 12.5 Hz"

  3. Residual Vector Quantisation (RVQ):

    • Multi-stage quantisation process
    • Each codebook iteratively quantises residual error from previous stage
    • High inductive bias: first quantiser captures essential content, subsequent quantisers capture progressively finer-grained features
    • Enables compact audio representation while preserving quality

Semantic training approach:

  • Uses knowledge distillation with WavLM as semantic teacher model
  • Additional loss function minimises cosine distance between semantic RVQ code and WavLM-generated embedding
  • Ensures semantic codebook captures linguistically meaningful representations

Key advantage: By splitting semantic and acoustic information into parallel quantisation paths, Mimi enables the dual-transformer architecture to process meaning and style separately, improving both efficiency and controllability.

Training method

Confidence: HIGH (TechCrunch, Sesame Research)

Training data:

  • Volume: ~1 million hours of English audio data
  • Sequence length: 2048 tokens (~2 minutes of audio)
  • Epochs: 5 full passes through dataset
  • Hardware: High-performance GPU clusters (specific configuration not publicly disclosed)

Data structure:

  • Training samples structured as alternating interleaved patterns of text and audio
  • Speaker identity encoded directly in text representation
  • Enables single-model multi-speaker synthesis without explicit speaker embeddings

Compute amortisation:

  • Backbone transformer trained on every frame (100% coverage)
  • Audio decoder trained on random 1/16 subset (6.25% frame coverage)
  • Maintains audio quality while reducing memory requirements by ~85%
  • Critical for scaling to billion-parameter models with 2-minute context windows

Performance characteristics

Confidence: HIGH (Sesame Research, eWeek, Cerebrium Deployment Guide)

Latency: <200ms end-to-end in real-time interactions. The streaming architecture enables immediate audio playback as tokens are generated, which suits interactive conversational applications.

Quality metrics:

  1. Traditional benchmarks (saturated): Word Error Rate and speaker similarity are both near-human, but these metrics show limited discrimination for the strongest current models.

  2. Contextual benchmarks (novel):

    • Homograph disambiguation: Correctly pronounces context-dependent words (e.g., "lead" as /lɛd/ for metal vs. /liːd/ for verb)
    • Pronunciation consistency: Maintains speaker-specific pronunciation variants across multi-turn conversations

  3. Subjective evaluation:

    • Without context: Listeners showed no clear preference between CSM-generated and real speech
    • With context: Evaluators favoured original recordings when judging conversational appropriateness
    • Interpretation: CSM achieves segment-level naturalness but prosodic gaps remain in long-form conversation flow

Emotional intelligence:

  • Analyses conversation context to respond with appropriate emotional tones
  • Recognises user mood (excitement, tiredness, uncertainty)
  • Adjusts tone, pace, and prosody dynamically based on conversational cues
  • During testing, appropriately responded to user emotional states (e.g., asking "is everything okay?" when detecting tiredness)

Key advantages over traditional TTS

Confidence: HIGH (Multiple sources corroborate architectural benefits)

  1. No semantic bottleneck: Operating directly on RVQ tokens avoids information loss from semantic compression
  2. Context-aware prosody: Backbone observes full conversation history when predicting acoustic tokens
  3. Improved efficiency: Single-stage design reduces computational overhead compared to cascading pipelines
  4. Enhanced expressivity: Direct multimodal learning captures subtle prosodic patterns that rule-based systems miss
  5. Natural disfluency: Learns to produce human-like hesitations, repairs, and filled pauses from data rather than explicit programming

Integration requirements

Confidence: MODERATE (Learn Prompting, Opus Research)

CSM is one component in a complete voice AI stack. Required complementary systems include:

  • Automatic speech recognition (ASR): Convert user speech to text (e.g., Whisper, Deepgram, AssemblyAI)
  • Large language model (LLM): Understand intent and generate responses (e.g., GPT-4, Claude, Llama)
  • Orchestration framework: Manage pipeline flow, state, and latency optimisation

The architecture positions CSM as a TTS replacement rather than a complete conversational AI solution.

Part 2: open-source voice AI in 2025

Orchestration frameworks

These frameworks coordinate multiple AI services (ASR, LLM, TTS) in real-time conversational pipelines.

1. Pipecat

Confidence: HIGH (GitHub: pipecat-ai/pipecat)

Overview: Python framework for building real-time voice and multimodal conversational agents with 10.1k GitHub stars and 205+ contributors. For a configuration-level study, see Pipecat voice agent architecture.

Architecture: Modular pipeline composition with discrete, reusable components; streaming-first design for low latency; WebSocket and WebRTC transport support; voice-first design (speech treated as the primary concern).

Supported integrations (70+ services): STT (AssemblyAI, Deepgram, OpenAI Whisper, Google Cloud, Azure, AWS and 12+ others), LLMs (Anthropic, OpenAI, Gemini, Groq, Mistral, DeepSeek and 13+ others), TTS (ElevenLabs, OpenAI, Google, Deepgram and 20+ providers), S2S (OpenAI Realtime API, Gemini Multimodal Live, AWS Nova Sonic), vision, video avatars, and audio processing (Silero VAD, Krisp, Koala noise filtering).

Related tools: Pipecat Flows (structured conversation state management), Voice UI Kit (React components), Pipecat CLI, Whisker (real-time debugging dashboard), and client SDKs for JavaScript, React, React Native, Swift, Kotlin, C++ and ESP32.

Licence: BSD-2-Clause (permissive open source)

2. LiveKit

Confidence: HIGH (LiveKit.io, WebRTC Ventures Comparison)

Overview: Fully open-source platform for building real-time media applications, with a specialised agent framework on top. Powers ChatGPT's Advanced Voice Mode for millions of users globally.

Architecture:

  • Media server: Handles voice and video streaming with optimised codec support
  • Inference gateway: Unified access to TTS, LLMs, and STT services
  • Agent framework (LiveKit Agents): Tools for building AI agents on the real-time media foundation
  • Cloud platform: Deployment and scaling infrastructure

Key features: WebRTC infrastructure run at large scale, phone number and SIP integration for telephony, codec and bitrate optimisation for browser compatibility, full-stack observability, and session management for concurrent agent interactions. Supports both cloud-hosted and self-hosted deployments.

Use cases: Large-scale voice applications requiring reliability, telephony-integrated voice agents, video plus voice multimodal agents, physical AI agents with real-time communication.

Distinguishing feature: Tested at large scale through the ChatGPT deployment, which provides production reliability evidence.

3. TEN Framework

Confidence: MODERATE (GitHub: TEN-framework/ten-framework)

Overview: Open-source framework for real-time multimodal conversational AI with extension-based architecture.

Architecture:

  • Plugin system: Discrete extensions for specialised tasks (STT, LLM, TTS, avatar rendering)
  • Multi-language support: Python, TypeScript, Go extensions communicate through standardised interfaces
  • Real-time transport: RTC (Real-Time Communication) and WebSocket connections
  • State management: Efficient conversation state tracking across multimodal streams

Multimodal capabilities: Speech recognition and synthesis with provider flexibility; Live2D, HeyGen and Tavus avatar integration with lip-sync; concurrent text and voice channels; real-time speaker diarisation.

Supported integrations: STT: Deepgram. LLM: OpenAI. TTS: ElevenLabs, custom providers. Platforms: Agora RTC, Twilio SIP. Avatars: MotionSync, Trulience, HeyGen.

Development and hardware: TMAN Designer (a visual graph editor), Docker and GitHub Codespaces support; demonstrated ESP32-S3 deployment for edge agents; minimum requirements of a 2-core CPU and 4GB RAM.

Distinguishing feature: Native avatar support with synchronised lip-sync distinguishes TEN from pipeline-focused frameworks.

Comparison with alternatives:

  • Pipecat: Pipeline-first, more service integrations, stronger Python tooling
  • LiveKit: Production scale, proven in the ChatGPT deployment, media infrastructure focus
  • TEN: Multimodal-first, visual integration native, hardware/edge deployment support

Open-source speech models (leading models in 2025)

Speech-to-speech (S2S) models

These models bypass traditional cascading pipelines by directly processing speech input to speech output.

Qwen3-Omni (Alibaba Cloud)

Confidence: HIGH (GitHub: QwenLM/Qwen3-Omni, arXiv Technical Report, Hugging Face)

Overview: Natively end-to-end multilingual omni-modal foundation model supporting text, images, audio, and video inputs with real-time streaming speech and text outputs. Released 22 September 2025.

Performance:

  • Latency: 234 ms theoretical end-to-end first-packet latency in cold-start settings (technical report)
  • Benchmarks: SOTA on 22 of 36 audio/video benchmarks, open-source SOTA on 32 of 36
  • ASR & Voice: Comparable to Gemini 2.5 Pro

Language support: 119 text languages, 19 speech input languages, 10 speech output languages.

Model variants: Qwen3-Omni-30B-A3B (base instruction-tuned), -Thinking (extended reasoning), -Captioner (audio/video description).

Licence: Apache 2.0 (fully permissive commercial use)

Architecture note: Uses audio-text interleaving similar to Sesame CSM but extends to full omni-modal processing (vision + audio + text).

Moshi (Kyutai)

Confidence: HIGH (GitHub: kyutai-labs/moshi, arXiv Paper, Kyutai.org)

Overview: First real-time full-duplex spoken large language model using Mimi codec for streaming audio. Pioneered the neural audio codec approach adopted by Sesame and others.

Performance:

  • Theoretical latency: 160ms
  • Practical latency: 200ms
  • Full-Duplex: Processes user speech and generates responses simultaneously (no turn-taking required)

Architecture:

  • Casts spoken dialogue as speech-to-speech generation
  • Generates speech as tokens from the residual quantiser of the Mimi neural audio codec
  • Models user speech and agent speech as parallel streams
  • Removes explicit speaker turns, enabling natural interruptions and overlapping speech

Key innovation: Modelling arbitrary conversational dynamics (interruptions, backchanneling, overlapping speech) rather than rigid turn-taking.

Licence: CC-BY 4.0 (attribution required, commercial use allowed). Python implementation (primary) and Swift implementation for iOS.

Text-to-speech (TTS) models

VibeVoice (Microsoft)

Confidence: MODERATE (Code removed from repository after misuse concerns, but technical details well-documented: GitHub Community Fork, Hugging Face, Microsoft Blog)

Overview: Framework for generating expressive, long-form, multi-speaker conversational audio addressing scalability, speaker consistency, and natural turn-taking challenges.

Capabilities:

  • Long-form synthesis: Up to 90 minutes continuous speech
  • Multi-speaker: Supports 4 distinct speakers (most models limited to 1-2)
  • Latency: ~300ms first audible speech (hardware dependent)
  • Real-time variant: VibeVoice-Realtime-0.5B supports streaming text input

Performance: 182ms TTFB (Time To First Byte), MOS 4.69, 41 languages.

Architecture: Expressivity focus with prosody modelling for multi-speaker conversations; long-form generation that avoids the usual degradation after 30-60 seconds; natural turn-taking synthesis for conversational audio.

Licensing note: Microsoft removed the official code repository after discovering misuse inconsistent with research intent. Community forks exist but lack official support.

Comparison note: Choose VibeVoice for English-only, long-form multi-speaker synthesis with excellent timbre capture (requires 12-20GB VRAM).

Kokoro-82M

Confidence: HIGH (Hugging Face: hexgrad/Kokoro-82M, GitHub, Analytics Vidhya, Medium)

Overview: Compact open-weight TTS model achieving SOTA quality with only 82 million parameters.

Performance:

  • TTS Arena ranking: #1 (beating models 5-15x larger)
  • Latency: ~200ms TTFB
  • Efficiency: 1:5 real-time factor on consumer GPUs
  • Quality: Comparable to 467M-1.2B parameter models

Training:

  • Data: <100 hours of audio (significantly less than competitors using 100k-1M hours)
  • Compute: 500 GPU hours on A100 80GB ($400 total cost)
  • Hardware: Runs efficiently on consumer-grade GPUs

Architecture: Decoder-only architecture based on StyleTTS 2 and ISTFTNet, with no diffusion or encoder components. (StyleTTS2 uses diffusion models to describe speech styles as latent random variables.)

Distinguishing feature: Smallest parameter count with highest quality, enabling deployment in resource-constrained environments.

Fish-Speech (Fish Audio)

Confidence: HIGH (GitHub: fishaudio/fish-speech, arXiv Paper, Fish Audio Blog)

Overview: Open-source multilingual TTS with LLM-based architecture and large-scale training data.

Performance:

  • Latency: <150ms TTFB (with KV-cache optimisation)
  • Real-time factor: 1:5 on RTX 4060 mobile, 1:15 on RTX 4090
  • Quality:
    • Resemblyzer similarity: 0.914 (ground truth: 0.921), a 0.76% gap
    • SpeechBrain similarity: 0.762 (ground truth: 0.770), a 1.04% gap

Training: 720,000 hours of multilingual audio; 13 languages supported; uses a large language model architecture to learn diverse linguistic patterns.

TTS-Arena ranking: #2 as "Anonymous Sparkle"

Architecture: LLM-based synthesis approach, KV-cache optimisation for interactive applications, advanced multilingual pronunciation modelling.

Distinguishing feature: Closest to human ground truth performance in similarity benchmarks, indicating strong voice cloning and naturalness.

CosyVoice (Alibaba Cloud)

Confidence: HIGH (GitHub: FunAudioLLM/CosyVoice, arXiv Paper, Official Site)

Overview: Multi-lingual large voice generation model providing inference, training, and deployment full-stack capability. Latest version (3.0) released December 2025 with 0.5B parameters.

Performance (CosyVoice 3.0):

  • Latency: 150ms (streaming mode)
  • Parameters: 0.5B (smallest production-quality model)
  • English word error rate: 1.68% on the test-en set (the reinforcement-learning variant, Fun-CosyVoice3-0.5B-2512_RL)
  • Chinese similarity: 78.0% (human baseline: 75.5%), exceeding human performance
  • Pronunciation error reduction: 30-50% improvement over CosyVoice 1.0
  • Character error rate: Lowest on Seed-TTS hard test set

Language support: 9 languages (Chinese, English, Japanese, Korean, etc.) and 18 dialects.

Distinguishing feature: Exceeds human similarity baseline in Chinese TTS.

Comparison with Qwen3-TTS:

  • Choose CosyVoice: For non-Chinese languages, slightly better timbre in voice cloning
  • Choose Qwen3-TTS: For Chinese applications, faster voice cloning (3s vs 5s), lower VRAM usage

Voice AI architecture patterns (2025)

Cascading pipeline (traditional)

Confidence: HIGH (AssemblyAI Voice Stack, Video SDK, Softcery Architecture Comparison)

Flow:

FIG. 01
flowchart LR
  A["User speaks"] --> B["VAD detects end"] --> C["STT completes"] --> D["LLM processes"] --> E["TTS generates"] --> F["Response plays"]

Components:

  1. VAD (Voice activity detection): Detects speech presence and boundaries
  2. STT (Speech-to-Text): Converts audio to text
  3. LLM (Large Language Model): Understands intent, generates response
  4. TTS (Text-to-Speech): Synthesises audio from text

Latency breakdown:

  • VAD: 200-800ms (waiting for sustained silence confirmation)
  • STT: 100-500ms (streaming vs. batch)
  • LLM: 200-2000ms (model size, prompt complexity)
  • TTS: 200-800ms (streaming vs. full synthesis)
  • Total: 700-4100ms typical, 500-1000ms optimised

Advantages: Simple to implement and debug; clear separation of concerns; easy to swap components; well-understood failure modes.

Disadvantages: Cumulative latency makes conversation unnatural; cannot handle interruptions gracefully; no overlapping speech support; rigid turn-taking structure.

Best for: Transactional interactions (customer service, information retrieval) where latency tolerance is higher.

Speech-to-speech (real time)

Confidence: HIGH (Gustavo Garcia Medium, AssemblyAI)

Flow:

User speech (streaming) → S2S model → Agent speech (streaming) ← Parallel processing

Architecture: End-to-end speech processing without intermediate text representation; streaming input and output processed concurrently; enables interruptions, backchanneling and overlapping speech.

Latency: Sub-300ms end-to-end with edge deployment; real-time factor <1 (faster than playback speed).

Key models:

  • Moshi (Kyutai): 160-200ms, full-duplex
  • Qwen3-Omni (Alibaba): 234 ms theoretical first-packet latency
  • OpenAI Realtime API (proprietary)

Advantages: Natural conversational flow; rapid turn-taking support; preserves prosodic information; interruption handling.

Disadvantages: More complex to implement; harder to debug (no intermediate text); fewer supporting tools; limited transparency in decision-making.

Best for: Natural conversations, therapy/coaching applications, collaborative tasks requiring rapid back-and-forth.

Hybrid or supervisor architecture

Confidence: MODERATE (Gustavo Garcia Medium, Introl Blog)

Concept: Use fast S2S models for conversation flow while delegating complex tasks to larger text-based LLMs.

Flow:

FIG. 02
flowchart LR
  Q1["Simple queries"] --> S1["S2S model (fast response)"]
  Q2["Complex queries"] --> S2["S2S"] --> L["Text LLM"] --> T["TTS (high quality reasoning)"]

Example (OpenAI "Chat-Supervisor" pattern): Conversational flow handled by the Realtime API (S2S), with complex reasoning and tool use delegated to GPT-4 (text), combining speed and capability.

Advantages: Balances latency and capability; uses the strengths of both architectures; graceful degradation (can fall back to text mode).

Disadvantages: Increased architectural complexity; routing logic requires tuning; potential mode-switching jarring to users.

Best for: Applications requiring both natural conversation and complex reasoning (virtual assistants, productivity tools).

Voice activity detection (VAD) in depth

Confidence: HIGH (2025 Voice AI Guide, Introl Blog)

Purpose: Detect when users start and stop speaking to trigger appropriate pipeline actions.

State machine (four states):

  1. QUIET: No meaningful audio detected. Monitors the audio stream for speech onset and ignores background noise below threshold.
  2. STARTING: Speech beginning detected. Waits ~200ms for confirmation (avoids false triggers), accumulates an audio buffer, and transitions to SPEAKING if speech continues.
  3. SPEAKING: Active speech confirmed. Streams audio to STT until sustained silence.
  4. STOPPING: Speech ending detected. Requires ~800ms sustained silence (prevents premature cutoff), finalises the audio buffer, and triggers STT completion and LLM processing.

Implementation options:

  • Silero VAD: Lightweight neural network, good balance of accuracy/speed
  • WebRTC VAD: Classic signal processing approach, very fast
  • Krisp: ML-based with noise filtering
  • Deepgram/AssemblyAI: Cloud-based with integrated STT

Tuning parameters: Silence threshold (responsiveness vs. false triggers); confirmation delay (reduce false positives from transient noise); cutoff delay (avoid interrupting natural pauses in speech).

Why it matters: Poor VAD implementation causes most user frustration in voice agents (premature cutoffs, slow responses, missed utterances).

Part 3: building voice AI of Sesame's quality

Confidence: MODERATE (synthesised from this survey)

Cascading pipeline: Pipecat or LiveKit for orchestration; Silero VAD; Deepgram, OpenAI Whisper or AssemblyAI for STT; GPT-4o, Claude 3.5 Sonnet or Llama 3.3 70B as the LLM; Fish-Speech 1.5, Kokoro-82M or CosyVoice 3.0 for TTS.

Speech-to-speech: Qwen3-Omni (234 ms theoretical first-packet latency, omni-modal) or Moshi (200ms, full-duplex) as the model, with the TEN Framework (native S2S support, multimodal) for orchestration and the same LLM choices for a hybrid.

Architecture decision guide

If you need Choose
Natural, human-like conversations with interruptions Speech-to-Speech (Moshi or Qwen3-Omni)
Multi-speaker long-form synthesis (podcasts, audiobooks) VibeVoice (if available) or CosyVoice
Lowest latency + smallest resource footprint Kokoro-82M TTS + cascading pipeline
Multilingual support (10+ languages) Qwen3-Omni (S2S) or Fish-Speech (TTS)
Production reliability at scale LiveKit orchestration + commercial STT/TTS
Maximum control and customisation Pipecat + open-source models (Fish-Speech/Kokoro/CosyVoice)
Matching Sesame's emotional intelligence 2-minute conversation context window + fine-tuning on emotionally annotated data; no open-source model fully replicates CSM's emotion detection

Key gaps compared with Sesame

Confidence: MODERATE (Based on technical comparison)

  1. Conversational context processing: Sesame CSM has a native 2-minute context window with explicit conversation history modelling. Open-source stacks must implement context management in the orchestration layer. Gap: most TTS models lack built-in conversation awareness.
  2. Emotional intelligence: Sesame demonstrated mood detection and appropriate emotional responses. Open-source models require explicit emotion annotation and fine-tuning. Gap: limited out-of-box emotional understanding.
  3. Prosodic coherence: Sesame's own subjective testing revealed remaining gaps in long-form conversational prosody. Open-source models face similar challenges, though Kokoro-82M and Fish-Speech show strong segment-level quality. Gap: multi-turn prosodic consistency is challenging for all current models.
  4. Integration simplicity: Sesame is a single model with a unified interface; open-source requires orchestrating multiple components. Gap: increased complexity but greater flexibility.

Evidence gaps and limitations

What is not known about Sesame

  1. Exact training pipeline: Specific data preprocessing, augmentation, filtering criteria not publicly disclosed
  2. Hardware requirements: Inference GPU requirements for real-time deployment not specified
  3. Fine-tuning method: Best practices for adapting CSM to new domains/languages unclear
  4. Production deployment: Scaling characteristics, concurrent user limits, cost per inference not documented
  5. Emotional detection mechanism: How mood analysis works technically (separate model, integrated in CSM, or rule-based)

Targeted searches on CSM inference GPU requirements, production deployment costs, fine-tuning and the emotional intelligence architecture did not resolve these: most technical blogs focus on architecture overview, the GitHub README lacks deployment guides, and the original research paper emphasises training and evaluation over production concerns.

Open-source unknowns

  1. Long-term model support: Sustainability of community-maintained models (e.g., VibeVoice removal raises questions)
  2. Comparative benchmarking: Limited head-to-head testing across models with identical evaluation protocols
  3. Edge deployment performance: Real-world latency on mobile/embedded devices varies significantly by optimisation
  4. Safety and misuse mitigation: Voice cloning ethical safeguards underdocumented across models

Methodology

Two passes of 5 searches each (breadth, then depth on the Mimi codec, S2S models, TTS benchmarks and architecture patterns), plus full reading of 5 primary sources. The 30+ sources comprise 3 arXiv papers, 8 official documentation sources, 12 technical blogs and 7 news and analysis pieces. High confidence means 3+ independent, recent, authoritative sources; moderate means 2 sources or one authoritative source; low means a single, speculative or dated source. Architectural claims were triangulated across GitHub, papers and blogs, and performance metrics cross-checked between model documentation and independent testing.

Key sources by category

Sesame AI architecture

Audio codecs

Orchestration frameworks

Speech-to-speech models

TTS models

Architecture patterns