Getting an AI to reply in chat is a solved problem. Getting an AI to answer a phone call — to speak, listen, be interrupted, respond in real time without sounding robotic — is a completely different game. It's genuine real-time engineering, and most people underestimate how many pieces have to work together within milliseconds. This is the map of the voice stack for AI agents, from someone who builds it.
Why voice is hard (and text isn't)
In text, latency is tolerable. The customer sends a message, you process it calmly, you respond three seconds later — nobody complains. In voice, three seconds of silence kill the conversation. The human ear is unforgiving: high latency is instantly perceived as "this service is bad."
And it's not just speed. On a call, the customer interrupts, changes the subject mid-sentence, talks over you, hesitates. A serious voice agent has to handle all of this naturally — which demands an orchestration that text simply doesn't need. I covered part of this in the architecture of an AI call center; here I go deep into the pieces.
The layers of the stack
1. Telephony (SIP)
The call arrives through the world of traditional telephony — the SIP protocol, carrier trunks, audio codecs. You need a robust edge (components like Kamailio live here) that bridges that world and the rest of the system: trunk registration, call routing, converting the telephony audio into something the pipeline understands. It's the least glamorous layer and the one that breaks the most if you get it wrong.
2. Real-time media transport
With the call established, the audio needs to flow both ways with minimal latency. This is where a media layer from the WebRTC/streaming world comes in (LiveKit is a strong example) — the "pipe" optimized for real time through which the voice travels.
3. STT — real-time transcription
The customer's audio becomes text, in streaming, while they're still speaking. Providers like Deepgram are the reference here. What matters:
- Streaming, not batch. The STT starts transcribing before the customer finishes the sentence.
- End-of-speech detection (endpointing). Knowing when the customer is done is decisive — cutting in early is annoying, waiting too long sounds slow.
- Language and accent. In Portuguese, STT quality with regional accents makes all the difference.
4. LLM — the brain
The transcribed text goes to the language model, which understands the intent, decides the response and — crucially — calls tools when it needs to act (query a system, generate an invoice). In voice, two requirements are non-negotiable:
- Very low latency, because the customer is waiting live.
- Fallback across providers: on a call, there's no "try again." If the primary LLM chokes, it falls to the secondary in milliseconds.
5. TTS — the voice back
The LLM's response is synthesized into speech. Providers like ElevenLabs and Cartesia offer neural voices that sound human. Here, quality defines the customer's perception:
- Natural voice. A robotic voice destroys trust instantly.
- Streaming. The TTS starts speaking as soon as the first words come out of the LLM — it doesn't wait for the whole response.
- TTS fallback too, because no provider has 100% uptime.
The orchestration: where the magic (and the difficulty) lives
Having the five pieces isn't enough. What makes the conversation sound natural is the streaming orchestration, with the stages overlapped instead of sequential:
Customer speaks ─→ STT transcribes (streaming)
└─→ LLM already starts processing
└─→ TTS already starts speaking
└─→ customer hears — and can interrupt (barge-in)
Barge-in — the customer interrupting the AI mid-sentence — is not optional. Without it, the agent feels like a dumb IVR. And this whole dance has to happen with total latency below the threshold of human discomfort. Voice-agent orchestration frameworks (the ecosystem around LiveKit Agents, for example) exist precisely to manage this pipeline. Platforms like Retell package much of this into a product, trading flexibility for speed of implementation.
How to choose the pieces
There's no "best" stack — there's the right one for your case. The decision axes:
- Latency: each provider adds milliseconds. The total latency budget is tight.
- Language: STT/TTS quality in Portuguese (with accents) varies a lot between providers.
- Cost: STT and TTS charge per minute; add that to the cost of LLM and telephony.
- Voice quality: what sounds natural to your audience.
- Build vs. buy: assembling the pipeline piece by piece (more control and optimization) or using a platform that packages it (faster to get started).
The takeaway
A voice agent is 20% language model and 80% real-time engineering: telephony that doesn't drop, STT and TTS in streaming, orchestration with barge-in and fallback, all within a brutal latency budget. The LLM is the commoditized part; the hard part — and where the value is — is making the five layers dance together in milliseconds.
That's why most of the "voice AI agents" promised out there don't deliver: they stopped at the text prototype and never faced the real time of voice. Facing that is exactly what I do.
I build voice AI agents in production — from SIP to TTS, with the orchestration that holds up on a real call. If you want an AI answering the phone, let's talk.