The Anatomy of a High-Conversion Voice Agent

Latency, interruption handling, and natural prosody. The key components of AI voice agents that actually convert leads.

AI Voice Agents are transforming customer support and inbound lead generation. However, if a voice agent has a 3-second delay, misinterprets when a user interrupts, or sounds robotic, users will quickly hang up. Building conversational voice systems requires managing audio streaming pipelines and latency budgets.

1. The Conversational Latency Budget

Human conversation averages a response gap of 200ms to 400ms. If an AI agent exceeds 1.2 seconds of delay, the interaction feels awkward and disjointed. To achieve sub-second latency, we optimize every step of the pipeline:

This includes using fast Speech-to-Text (STT) models like Deepgram to process audio streams, running low-latency LLMs like Gemini Flash on high-performance inference APIs, and utilizing streaming Text-to-Speech (TTS) engines like Cartesia or ElevenLabs.

2. Setting Up Real-Time Audio Connections

Below is a TypeScript implementation illustrating how to structure a WebSocket connection that routes real-time user audio to an LLM, processes output streams, and manages interruptions using Voice Activity Detection (VAD).

// Establish WebSocket connections for duplex audio transmission
export function initializeAudioStream(url: string, onAudioData: (data: ArrayBuffer) => void) {
  const ws = new WebSocket(url);
  ws.binaryType = "arraybuffer";

  ws.onmessage = (event) => {
    if (event.data instanceof ArrayBuffer) {
      onAudioData(event.data);
    } else {
      const msg = JSON.parse(event.data);
      if (msg.type === "USER_INTERRUPTED") {
        // Clear audio cache to stop bot speech instantly
        stopAudioPlayback();
      }
    }
  };

  return ws;
}

3. Calibrating Voice Activity Detection (VAD)

Managing interruptions is a key challenge for voice bots. If a user interrupts the bot while it is speaking, the bot must immediately stop its output audio stream and clear its generation queue. This requires configuring Voice Activity Detection (VAD) parameters.

If the VAD threshold is too low, background noise can trigger false interruptions, causing the bot to stop speaking. If the threshold is too high, the bot will ignore the user's voice, leading to overlapping audio. We calibrate VAD inputs dynamically based on line volume to keep interactions natural.

4. Low-Latency Voice Optimizations

• Fine-Tune Interruption Margins: Set Voice Activity Detection timeouts to 300ms. Shorter values risk cutting off users who pause briefly; longer values delay response times.
• Use Chunk-Based Audio Streaming: Stream audio in small 20ms chunks rather than waiting for full sentences to render.
• Store Session State: Persist call details in database records so the agent can reference context from previous calls.
• Gracefully Handle Errors: If an API timeout occurs, play a natural fallback transition (e.g. 'Let me pull that up for you...') to mask the delay.

Calibrating latency budgets and interruption handlers results in conversational AI voice agents that handle support queries, book appointments, and capture leads naturally.