Latency Engineering
How To Solve The Voice AI Feedback Loop Problem In Real-Time Production
5mins
Aryan Kushwaha
How To Solve The Voice AI Feedback Loop Problem In Real-Time Production
Most teams building voice agents assume the hard part is the model: get a good LLM, wire up a text-to-speech voice, ship it. The part that actually breaks in production is the audio pipeline itself. The agent starts talking, the user's speaker leaks into their mic, the agent hears its own voice, and the whole conversation stutters or loops. This is the feedback loop problem, and it shows up the moment you go from a demo to a real phone call or browser session with real background noise and real network jitter.
Here's the direct answer: you fix it by treating the system as full-duplex from the start, not as two half-duplex turns stitched together. That means server-side acoustic echo cancellation (AEC) tuned to your own audio path, VAD thresholds set per environment instead of a single global default, and a websocket protocol that treats "user started talking" as an interrupt signal, not a queued event. Standard noise suppression libraries tuned for phone calls do not fix this, because they assume the two ends of a call are on separate devices with separate acoustic paths. A voice agent talking and listening on the same device breaks that assumption entirely.
The thesis of this piece: real-time voice AI reliability is a plumbing problem before it's a model problem, and the plumbing has to be solved at the protocol and signal-processing layer, not patched with a bigger buffer or a longer silence timeout.
Why Does Audio Bleed-Back Happen In The First Place?
Acoustic echo happens when the agent's own TTS output, played through a speaker, gets picked up again by the same device's microphone and sent back into the pipeline as if it were new user speech. On a phone call this is a known, decades-old problem with mature solutions. In a browser-based or app-based voice agent, it gets worse for three reasons:
The playback and capture devices are often the same physical unit (laptop speakers into laptop mic), so the echo path is short and loud.
The agent's TTS stream and the user's mic stream travel over the same websocket connection, so timing drift between them is easy to introduce and hard to detect.
Most teams reach for a generic noise suppression model (built for suppressing background hum or crosstalk) instead of a proper echo canceller, which is a different algorithm solving a different problem.
The result is what people call the "dreaded audio bleed-back": the agent hears a faint, delayed copy of its own sentence, treats it as a barge-in, and either restarts its response or garbles the next turn.
Where Does Cascading Latency Actually Come From?
Latency in a voice pipeline is rarely one big delay. It's several small delays that stack: speech-to-text processing time, LLM time-to-first-token, TTS synthesis time, network round trip, and jitter buffer smoothing on both legs. Each one is defensible in isolation. Added together, they push the perceived response gap past the point where a human conversation feels natural, which research on turn-taking generally puts well under a second.
The fix is not to shave milliseconds off each component in isolation. It's to stream every stage instead of batching it:
Send partial transcripts to the LLM as soon as VAD confirms end-of-utterance, rather than waiting for a fixed silence window.
Start TTS synthesis on the first sentence of the LLM's response while the rest is still generating.
Stream TTS audio to the client in chunks over the same websocket connection, rather than waiting for the full clip.
Chaining these three streaming stages is what actually collapses cascading latency, because each stage's tail latency overlaps with the next stage's head latency instead of stacking end to end.
Why Doesn't Standard Noise Suppression Solve This?
Off-the-shelf noise suppression models are trained to remove stationary background noise: fans, traffic, keyboard clatter. They are not designed to track a known reference signal (the agent's own outgoing audio) and subtract a delayed, distorted copy of it from the incoming mic signal. That second task is what acoustic echo cancellation does, and it needs the reference signal, not just the noisy microphone feed, to work.
This is why bolting a generic denoiser onto a voice agent pipeline reduces bleed-back a little but never eliminates it. Proper server-side AEC needs:
Access to the exact audio samples sent to the client for playback, timestamped.
An adaptive filter that estimates the echo path delay, which changes with network conditions and device hardware.
A double-talk detector so the canceller doesn't wrongly suppress genuine user speech that happens to overlap with agent playback.
What Should VAD Thresholds Actually Look Like In Production?
A single global VAD threshold, tuned once in a quiet office, is one of the most common causes of both false interruptions and missed barge-ins in production voice agents. The right approach is closer to a small decision table than a single number:
Environment signal | Typical failure with fixed threshold | Adjustment needed |
|---|---|---|
Quiet room, good mic | Agent interrupted by breathing or lip smacks | Raise energy threshold slightly |
Noisy environment (car, street) | Agent never detects barge-in over background noise | Combine energy VAD with a speech classifier, not energy alone |
Echo-prone device (laptop speakers) | Agent's own voice re-triggers VAD as a false barge-in | Gate VAD input through AEC output, not raw mic signal |
High network jitter | Barge-in detected late, agent talks over user | Shorten the confirmation window, accept slightly more false positives |
The pattern across all four rows: VAD should never run on the raw microphone signal alone. It should run on the AEC-cleaned signal, with thresholds that adapt to a rolling noise floor estimate rather than a fixed constant set at deploy time.
Where This Is Heading
The teams getting this right are moving away from client-side echo cancellation (which varies wildly by browser and device) and centralizing AEC, VAD, and interrupt handling on the server, where the reference signal, the mic signal, and the LLM state all live in one place and can be reasoned about together. Expect the next round of voice AI infrastructure to standardize this as a single "conversation state" service sitting between the websocket transport and the model layer, rather than leaving each team to re-solve echo cancellation and barge-in detection on their own. As that layer matures, the teams still hand-rolling AEC per project will be the ones stuck debugging audio bleed-back in production instead of shipping features.
[
FAQ
]
Frequently Asked Questions
[
Browse Articles
]
Browse More Articles
Explore content across the voice AI stack — from infrastructure to real-world applications
