Every AI in production has an uncomfortable secret: it depends on an LLM provider that isn't under your control. And providers go down. They get slow on a Tuesday morning. They return a 529 overload error at the worst possible moment. If your agent depends on a single provider, then your product's availability is actually the availability of the weakest link in the chain — and you've outsourced your uptime to a company that owes you nothing.
In serious production, that's unacceptable. The solution is LLM fallback, and it's one of the first things I stand up in any agent.
The problem, concretely
Imagine an agent handling a conversation on WhatsApp or a live call. The customer sends a message, the agent calls the primary LLM and... timeout. Or a 500 error. Or 15 seconds of latency. Without fallback, what the customer sees is: silence. The conversation died halfway through. On a voice call, it's even worse — there's no "try again in a bit," the conversation is now.
The cause isn't always a total outage. Often it's degradation: the provider is up, but slow or unstable. From the customer's point of view, too slow is the same as down.
The architecture: cascading fallback
The principle is simple: no single point of failure in the response. In practice, I set up a cascade of models:
Request
→ primary LLM (fast, good, cost-effective)
failed/timeout? ↓
→ secondary LLM (another provider, equivalent capability)
failed/timeout? ↓
→ tertiary LLM (last resort, just so it doesn't die)
Rules that make it work:
Different providers at each level. If the primary and the secondary are from the same provider, one outage takes down both. Resilience comes from diversity — models from different companies, different infrastructures.
Aggressive timeout before giving up. Don't wait 30 seconds for the primary to respond. If it hasn't come in X seconds, consider it a failure and fall to the next. High latency is failure.
Retry with common sense. A quick retry on the same provider can fix a momentary hiccup. But don't keep hitting the broken link — fall to the next one early.
Total transparency for the customer. They should never know the primary went down. The switch happens in milliseconds, behind the scenes. Success is the customer noticing nothing.
Bonus: fallback is also cost optimization
Once you have the infrastructure to route between models, it serves more than resilience. The same layer lets you pick the right model per message type: the cheap model for simple intents ("what's my invoice?"), the expensive model only when the conversation demands reasoning. Most messages are simple — and that choice cuts the cost per conversation drastically.
In other words: the architecture that saves you when a provider goes down is the same one that saves money every day. Two problems, one solution.
What to watch (or you won't know it's broken)
Silent fallback is great for the customer and dangerous for you: if the primary is failing all the time and the secondary is holding, everything looks fine — until the secondary goes down too. So measure: fallback rate per provider, latency at each level, how many requests reached the last resort. If your primary started failing 30% of the time, you want to know before it fails 100%.
The principle
Resilience isn't a feature you add later. It's an architecture decision you make at the start: my product can't have the availability of third parties I don't control. LLM fallback is how you take that control back — turning the inevitable outage of a provider into a non-event nobody notices.
An agent without fallback works beautifully in the demo. In production, it's a bomb waiting for the day the provider has a bad day. And the provider always does.
I build resilient AI for production — with fallback, observability, and the engineering that holds up in the real world. If you have an AI that can't go down, let's talk.