Model Training
Interaction Behavioralism
Why Is Low ASR Confidence The Hardest Problem In Voice AI?
5 mins
Aryan Kushwaha
Why Is Low ASR Confidence The Hardest Problem In Voice AI?
Everyone building voice agents assumes the hard part is getting the transcript right. Get a good speech to text model, stream the audio, done. The part nobody talks about is what happens the moment the model isn't sure, when a word comes back with a low confidence score and your system has to decide, in real time, whether the speaker actually said something unexpected or the recognizer just slipped. That decision runs silently inside almost every voice product shipping today, and it's the same decision whether you're building a telephony agent or, as one Unio community member found while building something completely different, a teleprompter.
The short answer: a low confidence token is ambiguous by nature, it could mean the speaker went off script or the model misheard them, and the only real fix is giving the system something to check the token against, a known script, a dialogue state, or a constrained vocabulary, rather than trying to make the confidence score itself more trustworthy.
What Does A Low Confidence Score Actually Mean?
Less than most builders assume. In Speechmatics' realtime API, for example, the confidence field on a partial transcript has no meaning at all and isn't meant to be relied on, it only becomes meaningful once a segment is finalized. Telnyx's streaming docs say the same thing about interim results across engines: partials come back with a placeholder confidence value, and the real number only shows up on the final transcript. That's a real constraint, not a bug. The whole point of a partial is speed, you're trading certainty for getting words on screen (or into your agent's logic) in under half a second, and certainty is exactly what you don't get to keep.
So when a builder says a low confidence token showed up mid-stream, what they're actually looking at is the system admitting it doesn't know yet, not a diagnosis of what went wrong. The recognizer isn't telling you "the speaker went off script." It's telling you "I can't tell you anything useful about this word yet." Deciding what that silence means is left entirely to the application.
Why Does This Problem Look Different With A Script Than Without One?
This is the interesting part of what the teleprompter builder ran into. A teleprompter that follows a speaker's voice instead of scrolling on a timer has one advantage a telephony voice agent almost never gets, it knows the script in advance. Feeding proper nouns and jargon into the session as additional vocabulary before it starts, then aligning partial transcripts against that known token stream to move the cursor, means every low confidence word has something to be checked against. If the recognizer returns a shaky token but the surrounding words still line up with the next line of the script, the system can reasonably assume the recognizer slipped and hold position. If the words stop matching the script entirely, that's a real signal the speaker went off book, and the cursor can park until they return.
A telephony voice agent almost never has that luxury. There's no script to check the transcript against, because the whole premise of the call is that the caller can say anything. That's exactly why voice agent teams end up building an entirely different apparatus to handle the same underlying problem, deciding what a shaky signal means, without the one thing that makes the teleprompter's version tractable, ground truth.
How Do Production Voice Agents Actually Solve This Without A Script?
Mostly by triangulating multiple weak signals instead of trusting any single one, and by accepting a nonzero error rate as the cost of running in real time. Barge-in detection, deciding whether a caller trying to interrupt a talking agent is really trying to interrupt it, is the clearest example. One published analysis of a real deployment found that a simple rule based policy (streaming ASR plus a confidence threshold) flagged interruptions correctly only 11 percent of the time, the other 89 percent were false triggers caused by background noise, backchannel words like "uh-huh," the agent hearing an echo of its own voice, or the agent and caller talking over each other by timing accident.
The fix teams have converged on looks a lot like what the teleprompter did, just without a script to lean on. Instead of one confidence number, production barge-in systems stack several: Voice Activity Detection to confirm there's real speech and not just noise, a minimum duration threshold so a cough doesn't trigger a false stop, and a second classification pass after the fact to check whether the audio was actually directed at the agent. Vendors report tuning VAD confidence thresholds in small increments, around 0.5 to 0.6 as a starting point, and treating a false barge in rate above roughly 8 percent as a sign the system is too trigger happy. None of this makes any single token more trustworthy. It just gives the system more context to weigh a bad signal against, the same job the script does for the teleprompter, rebuilt out of timing, noise classification, and conversation state instead of a known transcript.
What Can Telephony Builders Actually Borrow From The Script Based Approach?
The honest answer is: constraints wherever you can get them, even partial ones. A fully open ended caller can't be scripted, but most of what they say in a given turn usually isn't fully open ended either. A few patterns already do this:
Expected vocabulary at known points in the call. The same additional vocabulary trick the teleprompter used, feeding in jargon and proper nouns ahead of time, works just as well for a voice agent that knows it's about to ask for an account number or a product name.
Dialogue state as a soft script. If the agent just asked "was that Tuesday or Wednesday," the set of plausible next utterances is small even though the caller could technically say anything, and a low confidence token can be checked against that narrow set instead of the full language.
Confirmation loops for anything downstream of a shaky token. Rather than trusting a low confidence transcript of a date, amount, or spelling, production systems increasingly read it back and ask the caller to confirm, effectively creating a tiny piece of ground truth on the fly.
A second pass after the decision, not just before it. The post-interruption validation pattern from barge-in systems, re-checking the audio that triggered a decision after the fact, generalizes well beyond interruptions to any place a system acted on a low confidence signal.
None of these turn an open call into a scripted one. But they all do the same thing the teleprompter's script does: give a shaky token something real to be compared against, instead of asking the confidence score to be more certain than it can be.
Where Ground Truth Comes From, By Use Case
Product | Is there a script to check against? | What actually substitutes for ground truth |
|---|---|---|
Voice-following teleprompter | Yes, the full script is known in advance | Alignment of partial transcripts against the known token stream |
Live captioning / subtitles | Partially, for pre-recorded content | Higher max-delay settings (2 to 4 seconds) traded for accuracy over speed |
Telephony voice agent (structured step) | Narrow, dialogue state limits likely answers | Confirmation loops, constrained vocabulary at that step |
Telephony voice agent (open conversation) | No | Multi-signal barge-in detection: VAD, duration threshold, post-hoc reclassification |
Where This Goes Next
The gap between the teleprompter's world and the open telephony call is going to keep narrowing, not because ASR confidence scores get better, but because voice agents are getting better at generating their own scripts on the fly. Dialogue state tracking, structured slot filling, and next-turn prediction are all, functionally, ways of building a temporary, narrow script for whatever the agent expects to hear in the next few seconds. The teams that treat a low confidence token as a request for more context, not a broken measurement, are the ones already building toward that. The ones still trying to raise the confidence threshold until the noise goes away are solving the wrong problem.
[
FAQ
]
Frequently Asked Questions
[
Browse Articles
]
Browse More Articles
Explore content across the voice AI stack — from infrastructure to real-world applications
