Every team starts with one model. It's the sensible choice — one API key, one SDK, one bill. You build fast. It works in the demo. Then you hit production and discover that "works" was doing a lot of heavy lifting.
The single-API trap
The trap has three jaws. The first is availability — Claude, GPT-4, Gemini: all have had outages in the last 12 months, some lasting hours. If your product calls a single provider, your product is down when they are.
The second is cost. You're routing a $0.015/1K token model for tasks that a local $0.0001/1K token model handles at equivalent quality. Nobody audits this until the invoice lands.
The third is privacy. Your terms of service may not allow sending customer PII to an external API, but your code does it anyway because there's no place to enforce that constraint at the call site.
"The router isn't interesting. The policy it enforces is."
What we built
A ModelRouter class that sits in front of every LLM call in the stack. It knows about four things: the workload type (reasoning, code gen, embedding, extraction), the privacy classification of the input (public, internal, PII, regulated), the current health of each provider, and the per-workload cost budget.
The routing policy
The routing logic resolves in this order:
- If input is PII or regulated → local model only. No exceptions. No overrides at the call site.
- If provider health check failed in the last 60s → skip that provider entirely.
- If workload is code generation → Qwen 2.5 Coder first, fallback to Claude.
- If workload is multi-step reasoning → Claude first, fallback to DeepSeek R1.
- Default → cheapest provider that meets latency SLA for this workload type.
The fallback chain
Fallbacks aren't just "try the next model if the first fails." A fallback chain is a policy decision: what's the acceptable quality degradation under failure, and who owns that decision? We encode it explicitly — per workload, per environment, with logging so the team can see when fallbacks fired and why.
# Example: reasoning workload policy
ROUTING_POLICY = {
"reasoning": {
"primary": "claude-3-5-sonnet",
"fallback_chain": ["deepseek-r1-32b", "llama3.3-70b"],
"privacy_override": "local_only",
"max_latency_ms": 8000,
"log_fallbacks": True,
}
}Production results
The router runs at 8–12ms overhead in production. In six months of operation it's rerouted around three Claude outages (total: ~4 hours), blocked 2,400+ PII-containing requests from hitting external APIs, and reduced model spend by 34% by routing simpler tasks to local models.
It's the most unglamorous piece of infrastructure on the stack and probably the most consequential. The architecture conversation that should happen in week one, not month six.
If you're building something similar or have questions about the implementation, write to me directly. I reply to everything.