Compute Stack
What Actually Powers Sub 200ms Multilingual Voice AI? It Is Not Just GPUs
5 mins
Aryan Kushwaha
What Actually Powers Sub 200ms Multilingual Voice AI? It Is Not Just GPUs
Everyone assumes that if you throw enough H100s at a voice model, low latency just shows up. Buy the cluster, load the weights, ship the demo. The overlooked reality is that raw compute barely matters if the software sitting between the GPU and the model is generic. The startups actually hitting sub 200ms round trips across many languages are the ones rewriting the inference stack itself, not the ones with the biggest GPU order.
Here is the direct answer: multilingual voice models are not slow because of weak hardware. They are slow because most serving stacks are built for text, and text serving assumes you can wait. Voice cannot wait. Getting to sub 200ms means custom kernels, careful memory bandwidth management, and architecture choices made specifically for streaming audio, layered on top of whatever GPUs you already have.
The thesis of this piece: latency in multilingual voice AI is won or lost in the inference engine, not the GPU spec sheet.
Why Do Standard Transformer Serving Stacks Fail At Voice Latency?
Most inference infrastructure in the industry, from vLLM to TensorRT-LLM defaults, was built to serve large language models answering text prompts. That workload tolerates a second or two of first token latency because nobody notices the delay while reading. Voice is different. A caller on the phone notices a 400ms pause immediately, and it reads as the system being broken.
Cartesia ran into this directly while building its Sonic voice model. The company found that most hosted serving paths are heavily optimized for massive transformer workloads, so it needed to build a custom inference engine tailored specifically to its architecture, which required deeper control over the serving environment than a standard hosted endpoint abstraction provides. That is the core trade off in this piece: you either accept the abstraction layer everyone else uses and inherit its latency floor, or you go build your own and own the complexity that comes with it.
Cartesia chose to build its own stack around a state space model architecture instead of a standard transformer, specifically because SSM architectures handle memory and sequence state differently, which lets the company run performant models while maintaining state of the art quality. The payoff shows up in the numbers: running on Together AI's GPU infrastructure, Cartesia reports a 90ms model latency for voice generation while serving millions of audio minutes daily on PCI compliant infrastructure.
What Does It Actually Take To Get Sub 200ms Round Trip Across Multiple Languages?
Multilingual adds a second layer of difficulty on top of raw speed. A model that is fast in English can fall apart in a tonal or agglutinative language if the tokenizer, the streaming decoder, or the prosody model was not built with that language in mind.
Sarvam AI's work building sovereign multilingual models for India is a useful case study here because the constraint was not just speed, it was speed across roughly two dozen languages at once. Sarvam scaled training to more than 4,096 NVIDIA H100 GPUs and cut production time to first inference from weeks to minutes while building models meant to serve real time voice AI in Indian languages. That is cluster scale solving the training side of the problem.
The harder part was inference, and this is where hardware level orchestration actually shows up. Working with NVIDIA on its Sovereign 30B model, Sarvam achieved a 4x speedup in inference performance on NVIDIA Blackwell over baseline H100 GPUs, made up of a 2x speedup from kernel and scheduling optimizations on H100 SXM GPUs, combined with a further 2x speedup from Blackwell's compute capabilities and NVFP4 weight quantization. In other words, half of that total speedup came from software work on hardware Sarvam already had, before a single new chip was involved.
This is the pattern worth pulling out: teams do not get from generic latency to sub 200ms by buying newer GPUs. They get there by rewriting the kernel and scheduling layer first, and only then layering newer silicon on top for a second multiplier.
Is Memory Bandwidth Really The Bottleneck, Not Compute?
Yes, for the specific case of streaming voice generation, memory bandwidth is usually the binding constraint before raw FLOPs are. Voice models generate audio token by token in a tight loop, and every step has to read the full set of model weights from GPU memory. When that loop is memory bound rather than compute bound, having more raw compute sitting idle does nothing for your latency.
This is part of why Sarvam highlighted specific gains under tighter compute and memory bandwidth constraints, where optimized kernels and scheduling strategies delivered 1.5x to 3x throughput improvements at typical operating points, with the improvements growing larger at longer input and output sequence lengths. Longer conversational turns are exactly where a badly tuned memory access pattern shows up as a laggy voice agent.
Cartesia's SSM based approach is a different answer to the same bandwidth problem. Instead of optimizing a transformer's memory access pattern, the company changed the underlying math so there is less state to move per step in the first place, which is part of why it can run low latency real time streaming voice generation on device, not just in a data center.
How Are Teams Actually Trading Off Cluster Scale Against Custom Kernels?
Here is the practical breakdown builders are working with right now:
Lever | What it buys you | What it costs |
|---|---|---|
More H100/Blackwell GPUs | Higher training throughput, more concurrent streams served | Capital and ongoing compute spend, does not fix per request latency alone |
Custom inference kernels | Lower per request latency, better tokens per second per GPU | Engineering headcount, ongoing maintenance as models change |
Weight quantization (e.g. NVFP4) | Additional speedup layered on top of kernel work | Some risk to output quality if not tuned per language |
Architecture change (SSM vs transformer) | Structurally lower memory movement per step | Full model retraining, smaller ecosystem of tooling |
On device / edge deployment | Removes network round trip entirely | Limited to smaller models, harder to update centrally |
Sarvam's numbers back up why kernel work sits above raw cluster size on this list. On H100 class infrastructure, its 30B model reportedly delivers 3x to 6x higher throughput per GPU at equivalent tokens per second per user, compared to a Qwen3 baseline. That is not new hardware, that is the same GPU doing more work per second because of how the serving code was written.
Where Is This Headed Next?
The next round of gains will not come from bigger clusters, they will come from teams treating kernel design, quantization, and scheduling as one system instead of three separate optimization projects tackled in sequence. Sarvam's own framing of its NVIDIA collaboration makes this point directly: the gains came from treating model design, kernel engineering, scheduling strategy, quantization, and GPU architecture as a single system rather than isolated components. Expect more voice AI teams, especially ones serving multilingual markets where no off the shelf kernel is tuned for the language mix, to pull inference in house the way Cartesia and Sarvam already have, rather than renting a generic endpoint and hoping latency holds.
[
FAQ
]
Frequently Asked Questions
[
Browse Articles
]
Browse More Articles
Explore content across the voice AI stack — from infrastructure to real-world applications
