Latency Engineering
Why Are Voice AI Startups Adding Latency Back Into Their Agents On Purpose?
5 mins
Aryan Kushwaha
Why Are Voice AI Startups Adding Latency Back Into Their Agents On Purpose?
For two years, every voice AI pitch deck had the same headline number: latency, and how low it's gotten. Sub-300ms. Sub-100ms. Sub-90ms. The assumption was simple shave milliseconds, win the category. The overlooked reality is that once you cross the threshold where a response feels instant, going faster stops helping and can start hurting. An agent that fires back in 150ms with zero breath, zero hesitation, and perfectly even pacing doesn't read as fast. It reads as wrong. That's why some of the same teams that spent 2024 and 2025 racing latency benchmarks are now spending part of that saved time putting pauses, breath sounds, and filler words back in.
The short answer: raw speed and perceived naturalness are two different metrics, and past a certain point they start competing for the same milliseconds. Startups are learning to spend some of their latency budget on imperfection breathing, "um," a half-second hesitation before a hard answer because a flawlessly fast agent breaks the conversational illusion just as badly as a slow one does.
The core thesis of this piece: the voice AI industry's latency race has quietly split into two races lowest raw response time, and lowest perceived response time and the startups winning on caller trust are the ones deliberately re-spending milliseconds on human imperfection rather than chasing the smallest number on a benchmark chart.
Why Doesn't Faster Automatically Mean More Natural?
Trillet, a voice AI platform for agencies, made this trade-off explicit rather than accidental. <cite index="25-1">The company's core response time averages around 400 milliseconds, sitting comfortably below the point where human conversation starts to feel like it has an awkward gap</cite>. Rather than push that number lower, Trillet's engineering spend went elsewhere. Ming Xu, Trillet's co-founder and CIO, frames it as a deliberate architectural choice: <cite index="25-1">the platform adds processing for natural speech patterns filler words, conversational acknowledgments, and pacing that matches human dialogue while raw API platforms optimize purely for speed and leave the "does this sound like a person" problem to developers</cite>. That added processing costs roughly 100–300ms per response, a cost Xu's team treats as worth paying because <cite index="25-1">an agent that answers in 900ms but sounds natural beats one that answers in 600ms but feels robotic, since caller perception not the millisecond count is what actually matters</cite>.
The research backing this isn't just internal. <cite index="25-1">A 2021 review of conversational turn-taking research found that human response gaps under roughly 500ms can feel interruptive, while gaps beyond 1,200–1,500ms start to feel unresponsive</cite> which means the "natural" window isn't zero, it's a range. Getting to the bottom of that range and stopping is different from racing past it.
How Are Startups Technically Injecting Breath and Hesitation Into the Stream?
Two different approaches have emerged, and they sit at opposite ends of the stack.
The first is TTS-side markup giving the voice model explicit tags to insert non-lexical sound. ElevenLabs' expressive conversational mode ships with built-in audio tags for breathing sounds, natural inhale/exhale pauses, and conversational filler words, on top of a base response time the company has pushed toward roughly 187ms. The idea is that the imperfection is authored, not accidental: a developer can mark exactly where a breath or hesitation belongs in a script.
The second approach is emergent rather than scripted. Rime, a linguistics-first voice AI lab, built its Arcana model on a proprietary dataset of full-duplex, spontaneous conversation including the interruptions, laughter, and disfluencies that a studio voice-actor recording strips out. Lily Clifford, Rime's CEO and co-founder, has described the resulting model as one that <cite index="26-1">infers emotion from context and produces laughs, sighs, hums, audible breaths, and disfluencies like "um" on its own, with emergent behaviors the team is still discovering</cite>. In other words: rather than hand-place a breath tag, the model learned when humans breathe from the data itself. Rime's models run at roughly 200–250ms time-to-first-audio, which the company treats as fast enough that spending some of it on realism doesn't cost the conversation its rhythm.
Underneath both approaches sits infrastructure most people never think about: jitter buffers. These are small, deliberate delays typically 40-100ms that voice platforms use to smooth out uneven network packet arrival so audio doesn't sound choppy. It's a second, unglamorous place where "imperfection" (a buffer, a tiny wait) is engineered in on purpose to make the final output sound cleaner, not despite the delay but because of it.
What Happens When You Get the Balance Wrong?
This isn't a one-directional trade. Overcorrecting toward "sound human" carries its own failure mode, and there's now data on exactly what that failure looks like.
A recent benchmark of full-duplex voice agents (Full-Duplex-Bench-v3) tested several production voice models under real-world disfluency conditions. One model, Ultravox, almost always inserted a filler phrase like "let me check on that" before calling any backend tool an <cite index="20-1">88% filler rate</cite>. That felt more human in isolation. But because the filler speech ran while the tool call was still pending, it frequently overlapped with what the caller was saying next, driving a <cite index="20-1">47.9% interruption rate and pushing total task-completion latency to 8.4 seconds</cite> slower, not faster, than agents with no filler behavior at all.
There's a second, more mechanical failure mode. Breathing and filler sounds can confuse the system meant to detect when a caller has finished talking. Voice activity detection models can mistake a caller's own "um" or audible breath for continued speech, adding unnecessary wait time before the agent responds the opposite of the effect the fillers were supposed to create. Getting endpointing thresholds wrong in either direction (too short and the agent cuts callers off; too long and every turn drags) is one of the most common tuning mistakes in production voice stacks.
Where Is the Latency Budget Actually Going?
Every response is really five stages stacked together, and "add naturalness" isn't a separate step — it's milliseconds pulled out of this same budget.
Stage | Typical range | What eats the time |
|---|---|---|
Speech-to-text | 100-300ms | Transcription, accent handling, background noise |
LLM inference | 200-800ms | Model size, reasoning depth, context length |
Text-to-speech | 100-400ms | Voice quality, streaming vs. full-generation |
Network round trip | 50-200ms | Distance to servers, codec transcoding |
Jitter buffer | 40-100ms | Smoothing packet arrival variance |
Naturalness layer (fillers, pacing) | 100-300ms | Deliberately inserted, not a system cost |
Stack the fast end of every range and you're near 600ms with zero naturalness spend. Stack the slow end and you're past 2 seconds before a caller feels frustrated. The naturalness layer is the only line item on this list that's optional and self-imposed which is exactly why it's become the place startups compete on judgment rather than raw engineering.
Where This Goes Next
The industry's public benchmark obsession whoever posts the lowest millisecond number is likely to keep losing relevance as more platforms cluster under the 500-800ms range where humans stop perceiving further gains. The next competitive line is shifting from raw latency to perceived latency: how a platform spends the milliseconds it has, not how few it uses. Expect emergent, data-trained disfluency (the Rime approach) to spread faster than hand-authored breath tags, since it scales across languages and voices without a developer scripting every pause. And expect endpointing and filler-injection to become per-use-case dials rather than fixed settings a fast transactional agent checking an order status tuned differently from a slower, higher-stakes appointment-booking or triage agent where a beat of hesitation actually builds trust.
[
FAQ
]
Frequently Asked Questions
[
Browse Articles
]
Browse More Articles
Explore content across the voice AI stack — from infrastructure to real-world applications
