--- title: "The architecture of an AI call center for internet providers" description: "How to build voice and WhatsApp support that talks to the customer 24/7: SIP telephony, real-time STT/TTS, LLM with fallback, and integration with the ISP's ERP." slug: arquitetura-call-center-ia-provedores lang: en date: 2026-07-22 updated: 2026-07-22 author: Lucas Silva category: deep-dive tags: [AI Call Center, Telephony, SIP, LiveKit, ISP, Voice, STT, TTS] reading_time: 13 featured: true --- An internet provider (ISP) lives on two things: the customer being connected and the customer being served. The first is network; the second, historically, is people on the phone. And this is where the math stops working: the volume of "my internet is down," "where's my invoice," "I need a duplicate copy" calls is huge, repetitive, and grows right along with the subscriber base. Hiring agents at the same pace the base grows is unsustainable. That's the problem that led me to build **ConectaAI** — an AI call center that handles support over voice and WhatsApp, 24 hours a day, integrated with the provider's system. This post is the architecture behind it. It's not theory: it's what has to exist for an AI to actually answer a phone call and solve the customer's problem. ## The real problem: real-time voice is hard Everyone knows how to build a text chatbot. Voice is a different game. On a call, the customer speaks, waits for a response, interrupts, changes the subject — and every millisecond of latency is perceived as "this service is bad." If the AI takes 3 seconds to respond, the conversation dies. So the core challenge is: **turn a phone call into a conversation with an AI, with latency low enough to sound human.** That breaks down into several pieces that all have to work together in real time. ## The system's layers ### 1. The telephony edge (SIP) The call arrives from the world of traditional telephony — SIP protocol, carrier trunks, all of it. This edge needs a robust component that can handle real traffic and bridge telephony with the real-time media world. It's the least glamorous, most tedious layer, and the one that breaks the most if you get it wrong. It handles trunk registration, call routing, and converting telephony audio (codecs, RTP) into something the rest of the system understands. ### 2. Real-time media transport With the call established, the audio needs to flow with minimal latency between the customer and the AI agent. This is where a real-time media layer (from the WebRTC/streaming world) comes in, carrying the audio in both directions. Think of it as the "pipe" the voice travels through, optimized for real time, not studio quality. ### 3. The agent's voice pipeline This is the heart of it. In a continuous loop, for each of the customer's speaking turns: ``` Customer audio → STT (speech-to-text) transcribes in real time, with end-of-speech detection → LLM understands the intent, decides the response and the tools → TTS (text-to-speech) synthesizes the response into a natural voice → Audio back to the customer ``` Every stage has to be *streaming*. The STT doesn't wait for the customer to finish the sentence to start transcribing. The TTS starts speaking as soon as the first words of the response come out of the LLM. It's that overlap that brings down the perceived latency. A few things I learned: - **End-of-speech detection (endpointing)** is decisive. Cutting the customer off too early is irritating; waiting too long sounds slow. It's constant tuning. - **Barge-in** (the customer interrupting the AI mid-sentence) is not optional. Without it, the AI feels like a dumb IVR menu. - **The voice matters.** A robotic synthetic voice destroys trust. Good neural voices (with fallback between TTS providers) completely change the customer's perception. ### 4. The brain: LLM with context and tools The LLM doesn't converse in a vacuum. It receives: - **The history** of the call (and of the customer, if they've called before). - **The customer's data**, pulled from the provider's ERP at the start of the call — plan, financial status, connection state. - **The tools** it can trigger during the conversation. And it's in the tools that the business magic happens. The agent doesn't *describe* how to solve the problem — it solves it: - Checks the connection status (is the customer's ONU online?) and runs the diagnosis. - Generates a **duplicate invoice** and sends the PIX on the spot. - Opens a **service order** and schedules the technician's visit. - Verifies payments and unblocks access. This requires direct integration with the provider's ERP (in the ISP world, systems like IXC dominate). Each tool is a validated API call, with confirmation on the irreversible steps and a log of everything. ### 5. Multi-LLM fallback I already covered this in the [post about WhatsApp agents](https://www.lucassilva.io/blog/agente-ia-whatsapp-producao), but in voice it's even more critical: on a live call, you don't have the luxury of "try again." If the primary LLM chokes, the system drops to the secondary in milliseconds, without the customer noticing. No single point of failure in the response. ## Two channels, one brain ConectaAI handles support over voice **and** WhatsApp. The temptation is to build two systems. The mistake is to build two systems. The channel (phone or WhatsApp) is just the input/output layer — the brain, the tools, and the business logic are shared. That means the same agent that solves it over voice solves it over text, with the same ERP integration and the same rules. A customer can start on WhatsApp and call later, and the context follows. Architecting it this way from the start saves months and keeps the two channels from diverging in behavior. ## Why it's not "just plugging in a chatbot" If you take one lesson from here, take this: **a real AI call center is 20% language model and 80% real-time systems engineering and integration.** The LLM is the easy, commoditized part. The hard part — and where the value is — is: - Keeping SIP telephony from falling over. - Keeping voice latency below the threshold of discomfort. - Actually integrating with the provider's legacy ERP. - Ensuring resilience: queue, retry, fallback, observability. - Knowing when to hand off to a human, with the full context. That's infrastructure software, not an API wrapper. And that's exactly why most of the "AI call centers" being promised out there don't deliver: they stopped at the text prototype and never faced voice in production. ## The result that matters The provider doesn't buy "AI." They buy: support that never sleeps, a queue that doesn't overflow at peak hours, a duplicate invoice resolved at 2 a.m. without waking anyone up, and a human team freed to handle what's genuinely complex. The AI absorbs the repetitive stuff — which is most of it — and the human handles the exception. That's the architecture of an AI call center that works. Not because the model is clever, but because the entire system was built to withstand the real world of a provider with thousands of customers calling in. --- *I build AI products in production — including voice and WhatsApp support for internet providers. If you have an ISP drowning in repetitive calls, [let's talk](https://www.lucassilva.io).*