Shipping sub-600ms voice AI: a production latency architecture

Why "fast" voice agents still feel broken, and the streaming, native-multimodal architecture we use to hold end-to-end conversation latency under 600 milliseconds.

By , Head of AI and Growth · · 12 min read

Why does latency make or break a voice agent?

Human conversation tolerates roughly 300 milliseconds of dead air before it feels broken. Past that, callers talk over the agent, repeat themselves, or hang up. Latency is not a performance metric for voice AI — it is the product.

Legacy pipelines chain three serial hops — speech-to-text, then the LLM, then text-to-speech — and each adds budget. Stack them and you are at 2–3 seconds per turn. That is why most "AI receptionists" feel like a 2010 IVR.

  • 2.8s Legacy turn latency
  • <600ms Target latency
  • −94% Dropped-call reduction

How do you get under 600ms?

Four moves, in order of impact:

  • Stream everything. Never wait for a complete transcript or a complete response. Partial ASR feeds the model; the model streams tokens to TTS as it generates.
  • Collapse the hops. A native multimodal model that takes audio in and emits audio out removes two serialization boundaries entirely.
  • Predict the endpoint. Voice-activity detection plus an endpoint predictor starts generating before the caller fully stops speaking.
  • Barge-in. Let the caller interrupt; cancel synthesis instantly. Nothing signals "machine" like an agent that can't be cut off.
// Stream partial transcripts straight into generation — no full-utterance wait.
asr.on("partial", (text) => {
  if (endpoint.predictsTurnEnd(text)) {
    const tokens = llm.stream(text);          // begin early
    tts.streamFrom(tokens);                   // speak as tokens arrive
  }
});

// Barge-in: caller speech cancels synthesis mid-sentence.
vad.on("speechStart", () => tts.cancel());

The takeaway

You do not hit sub-600ms by buying a faster model — you hit it by removing serialization. Stream, collapse hops, predict the endpoint, and allow interruption. Do those four and the conversation stops feeling like a transaction and starts feeling like a person.

Book a 20-minute AI Opportunity Call · Get a free AI opportunity map

iqdigital