Concurrency Logic
How Do You Scale Voice AI To Hundreds Of Simultaneous Speakers Without Killing Latency?
5 mins
Aryan Kushwaha
How Do You Scale Voice AI To Hundreds Of Simultaneous Speakers Without Killing Latency?
Most voice AI demos work because they cheat. One caller, one agent, a clean line, nobody talking over anybody. The moment you put a hundred or more live participants in the same audio room, most of that architecture falls apart. Mixing gets expensive, diarization gets confused, and the 200ms latency budget that felt comfortable in a two person call disappears into buffering and re-encoding.
The short answer: nobody mixes a crowded room the way they mix a call between two people. Teams that scale multi-party live audio, whether it's a debate room, a town hall, or a large moderated discussion, switch from mesh or MCU based mixing to a Selective Forwarding Unit (SFU) architecture, run overlap-aware diarization models built specifically for simultaneous speech, and treat the latency budget as a series of small, individually optimized slices rather than one number to hit.
Why Does Audio Mixing Break Down Once The Room Gets Crowded?
There are three ways to architect a group audio call, and only one of them survives at scale. In a mesh network, every participant connects directly to every other participant. That works for five people. Past that, the number of connections grows fast enough that both bandwidth and CPU load become unmanageable, since each device has to send its own stream to everyone else individually.
The second option is a Multipoint Control Unit (MCU), which decodes every incoming stream, mixes them into one composite feed, and sends that single stream back out. Clients love this because they only have to handle one stream. Servers hate it, because mixing means decoding and re-encoding every voice at once, and that processing adds real latency before a single word even goes out.
The option that actually scales is a Selective Forwarding Unit (SFU). Instead of mixing audio on the server, an SFU just forwards each participant's stream to whoever needs to hear it, without decoding or re-encoding along the way. LiveKit, one of the more widely used open infrastructure layers for this kind of build, positions its server as exactly this: an SFU that routes media between participants without requiring direct peer to peer connections at scale. For a debate room with hundreds of listeners and a rotating handful of active speakers, that difference in server load is the difference between a system that holds up and one that falls over during the exact moment it's supposed to shine.
How Do You Keep Diarization Accurate When People Talk Over Each Other?
Debate rooms aren't polite. People interrupt, talk over each other, and jump in mid sentence, which is exactly the scenario that breaks most speaker diarization systems. Standard clustering based diarization, the kind used by many general purpose transcription tools, degrades badly the moment two voices overlap.
The fix is overlap aware diarization, models trained specifically to detect and separate simultaneous speech rather than assume one voice at a time. Academic work on this, including research on end to end local segmentation for online diarization, shows latency in these systems can be tuned anywhere from around 500 milliseconds up to five seconds depending on how much accuracy you're willing to trade for speed. Commercial systems have caught up to a similar range: AssemblyAI's newer streaming models label up to ten speakers inline as audio arrives and revise earlier labels as more context comes in, so a speaker tag assigned in the first half second of someone talking can still get corrected once the model hears more of the sentence.
The practical lesson for a debate room build: don't treat diarization as a single pass. Treat it as a rolling, revisable process that trades a small amount of certainty for a large amount of speed, and accept that the first label on any given utterance might not be the final one.
Where Does The Extra Latency Actually Come From?
"Sub-200ms" sounds like one number, but it's really a budget split across several stages, and each stage has its own failure points. According to infrastructure firm RTC League, a realistic latency budget for a well built WebRTC voice pipeline breaks down roughly like this within a single geographic region:
Stage | Typical latency contribution |
|---|---|
Network propagation | Varies by distance; kept low by staying within the region |
TURN relay processing | 10 to 30 ms |
SFU forwarding | 5 to 20 ms |
Jitter buffer at the receiver | 40 to 80 ms |
Overall target, same region | Under 150 ms |
Overall target, intercontinental | Under 300 ms |
Notice that the jitter buffer, the part of the pipeline that smooths out irregular packet arrival so audio doesn't sound choppy, is often the single biggest contributor. That's the trade off nobody advertises: the buffering that makes a crowded room sound clean is also the thing eating most of your latency budget. RTC League's engineering team, led by founder and CEO Muhammad Aatif Bashir, argues scalability has to be a day one design decision rather than something bolted on once usage grows, and the latency budget above is exactly why: there's very little slack left to fix later once a hundred participants are already in the room.
The other lever teams pull is geography. Keeping infrastructure within roughly 30 milliseconds of network round trip time from the user base means running regional deployments instead of a single global server, which is a real infrastructure cost, not just a configuration setting.
What Does Noise Suppression Actually Require At Hundreds Of Participants?
Mixing and diarization get most of the attention, but noise suppression is what keeps a room usable once you have more than a handful of open mics. Every laptop fan, every keyboard click, every bit of room echo from a hundred different environments adds up, and it has to be cleaned before it reaches the mixer or the transcription pipeline, not after.
This is also where the SFU architecture pays off twice. Because an SFU doesn't mix streams centrally, noise suppression and voice activity detection can run per-stream, closer to the source, before a track is even forwarded to other participants or to the diarization model. That keeps the cleanup workload proportional to how many people are actually speaking, not how many people are just sitting in the room. LiveKit's own agent framework, for example, bundles an audio mixer that handles sample rate alignment and frame timing when multiple remote tracks need to be combined before being sent to a single downstream STT session, which is exactly the kind of glue code a debate room build needs when a moderator agent has to listen to several tracks at once.
The engineering discipline that separates a demo from a production system: treat noise suppression, voice activity detection, and diarization as three separate, swappable components, not one bundled black box, so each can be tuned or replaced independently as the room size and speaker overlap patterns change.
Where This Goes Next
The next wave of improvement here isn't going to come from a faster mixer, it's going to come from pushing intelligence closer to the edge and away from a central mixing point entirely. Expect more systems to run lightweight voice activity and noise suppression directly on the client before audio ever leaves the device, shrinking what the server has to process per stream. On the diarization side, expect the gap between streaming and batch accuracy to keep closing, since the overlap aware models research groups have been publishing since 2021 are only now showing up in production APIs. For anyone building a debate room, town hall, or large moderated voice product, the winning architecture in two years probably looks less like one big SFU doing everything, and more like a mesh of small, specialized components, each handling one job, stitched together by an orchestration layer that's willing to trade a little certainty for a lot of speed.
[
FAQ
]
Frequently Asked Questions
[
Browse Articles
]
Browse More Articles
Explore content across the voice AI stack — from infrastructure to real-world applications
