Before you blame the model for a slow voice agent, look at everything around it. Most of the lag comes from waiting to be sure the caller is done, and from tiny gaps nobody measures. Here’s a time budget for each stage and the fixes to try first.
Key Takeaways
- People usually reply to each other within about 200 ms. A phone bot won’t match that, but getting under 800 ms end to end is doable and worth the effort.
- Your biggest controllable delay is often just waiting to be sure the caller has stopped talking. Endpointing that reads the words, not only the silence, cuts that wait down fast.
- Stream at every stage boundary. Start speaking on the first complete clause, not the full response.
- Measure each stage separately at p95, using real phone audio instead of clean lab recordings.
You reply faster than you think. Most of us answer about 200 milliseconds after someone stops talking. A 2009 study of ten languages found the gap was short in all of them.
Now call a freshly built voice agent and count.
One. Two. Sometimes it’s still thinking.
That pause is where callers lose patience. Some hang up. Others talk over the bot or say “hello?” into the silence. And developers won’t love this part: the model usually isn’t the problem.
Stop Blaming the LLM
A voice agent is a relay race. The caller’s voice comes in over the phone line and gets turned into text. Then the model writes a reply, and a speech engine turns it into sound for the trip back. Every handoff eats a bit of time.
Profile a first build and the result tends to surprise people. The language model is only part of the wait. The rest slips through the cracks between steps.
Say your pipeline runs speech-to-text, then the LLM, then text-to-speech. Here’s roughly what each step can spend before a call feels off:
- Getting the caller’s audio to your servers: 50 to 100 ms
- Endpointing, or figuring out the caller is really done talking: 200 to 300 ms if you tune it well, much longer on default settings
- Final transcript: about 100 ms after the endpoint fires
- LLM time to first token: 200 to 350 ms
- TTS time to first audio byte: 50 to 150 ms
- Network and playback out: 50 to 100 ms
Add those up and a good call lands around 650 ms. A bad one drifts past a full second. The best platforms in production say they stay under 800 ms end to end. Aim there.
Not 200 ms. Nobody is hitting human parity over a phone line yet, and trying will eat your sprint.
Voice agents also raise the stakes on the transparency problem DevX flagged in its look at how AI agents are set to transform software development. If you can’t see what each stage did and how long it took, you can’t fix it.
Your Silence Detector Is the Biggest Lever
Voice activity detection tells you when someone is speaking. It does not tell you when they’re done.
“My account number is… uh… 4417” contains pauses longer than plenty of silence thresholds. So teams crank the threshold up to avoid cutting people off, and now every single turn carries a half-second tax before any real work begins.
There are better options:
- Semantic endpointing. Have a small, quick model read the live transcript and judge if the caller’s thought is complete. “I’d like to book” leaves you hanging. “I’d like to book for Tuesday” probably doesn’t. “I’d like to book for Tuesday” probably is.
- Context-aware timeouts. People pause halfway through reading a phone number, so give them extra time. A yes-or-no answer needs barely any wait at all.
- Speculative generation. Start the LLM call on the partial transcript and discard the result if the caller keeps talking. You burn some tokens and buy back hundreds of milliseconds.
- Per-language tuning. Speech rhythm varies, so a threshold tuned on one group of callers can feel rude to another.
This is also where production systems differ most. An AI phone Agent built for live calls doesn’t wait on a fixed silence timer. It runs its own turn-taking model. So before you tune thresholds by hand, find out what yours uses.
Stream Everything or Don’t Bother
The most common architecture mistake in early voice builds is waiting for complete outputs. Waiting for the full LLM response before calling TTS. Waiting for the whole audio clip before playing it.
Stream at every boundary instead. The moment the model finishes a sentence, or even half of one, send it to TTS. The caller should hear it right away.
The setup is simple: a small buffer that holds tokens until a sentence is done, then passes it along to TTS. The model keeps going while the voice talks. Run playback on its own task, too, so one slow piece can’t hold up the next.
Don’t slice too thin, though. Break at every comma, and the agent sounds like a jumpy GPS. Waiting for whole paragraphs throws the latency win away.
Here’s a cheap win. Let the model say a few words first, for example, “Sure, let me check that.” That one line covers the gap while the full answer is ready. Good support reps do this without thinking.
Tool Calls Are Where Milliseconds Go to Die
Useful agents do things. They check an order, move a booking. Each one is a request to your backend. The request goes to the backend. The backend sends back data. The model then takes that data. Turn it into something the model can say.
A few habits keep this in check:
- Prefetch on connect. Caller ID can trigger a CRM lookup before anyone says a word.
- Keep tool payloads tiny. The model doesn’t need 40 fields to confirm a delivery date.
- Put a hard timeout on every tool, with a fallback line ready. “I’m still pulling that up” beats dead air.
- Use templated filler while a tool runs, not generated filler. It’s instant and predictable.
- Colocate your services. An LLM endpoint in one region and a CRM on another continent will cost you on every lookup.
Measure Per Stage, at p95
Averages hide the bad calls. Log a timestamp each time audio passes from one stage to the next, from arrival up to playback.
Check your p95 and p99 too. One three-second silence sticks with a caller longer than a hundred quick replies.
And test over real phone lines. The audio comes in thin, at 8 kHz, with dropouts, noise, and every accent imaginable.
Build, Buy, or Split the Difference
Building it all yourself gives you control. It also makes the silence thresholds and SIP quirks your problem, along with the pager at 3 a.m.
Most teams split it. They keep the parts that are truly theirs: the prompts and the tools the agent can call. The real-time voice layer gets rented. A managed voice platform handles phone lines and speech streaming behind one API, and your engineers get time back for work customers actually notice.
Either way, judge vendors by the same per-stage numbers you’d want from your own code.
Your caller has no idea which model you picked. They only hear the pause.
So open your logs tonight, find the slowest stage, and fix that one first.
Photo by Charanjeet Dhiman: Unsplash
Rashan is a seasoned technology journalist and visionary leader serving as the Editor-in-Chief of DevX.com, a leading online publication focused on software development, programming languages, and emerging technologies. With his deep expertise in the tech industry and her passion for empowering developers, Rashan has transformed DevX.com into a vibrant hub of knowledge and innovation. Reach out to Rashan at [email protected]
