Model Routing Is Simple. Until It Isn’t.
Model routing isn't a classification problem—it's a systems optimization challenge: real costs depend on cache hit rates, not just token pricing, and task difficulty is often invisible at routing time.
- Sticker prices are misleading: GPT-4.1 had lower token costs but Sonnet cost half as much thanks to high cache hit rates.
- 智能体 workloads are cache-friendly: multi-step reasoning reuses context, making cache-read pricing dominate total cost.
- Task difficulty can't be reliably estimated upfront: simple prompts snowball into complex tool use, while technical queries may be handled cheaply by small models.
- Enterprise routing demands balancing cost, latency, compliance, and more—far beyond 'hard tasks go to big models.'
As AI agents become mainstream, 'model routing' sounds like an obvious win: use a classifier to send simple requests to cheap models and reserve expensive ones for hard tasks. On paper, costs drop while performance holds.
But IBM Research's engineering note reveals a stark reality: do it naively, and your bill could double instead of shrink.
Why now? In 2025, agents are moving from demos to production, exposing real infrastructure challenges. Routing appears to be a classification problem, but inside an agent's multi-step loop, it quickly morphs into a systems optimization headache. The team ran 417 tasks on the AppWorld benchmark using the same CodeAct agent with GPT-4.1 and Claude Sonnet 4.6. The shock: Sonnet cost $79 total vs. GPT-4.1's $155 — nearly double — despite GPT-4.1 having lower per-token pricing and Sonnet needing three times more reasoning steps.
The culprit? Two hidden dimensions.
First, cost is not just base pricing. Caching dominates actual expenditure. 智能体 workloads reuse huge chunks of context across steps — system prompts, tool manuals, history. High cache hit rates mean the effective input cost plummets, and Sonnet's lower cache-read pricing gave it a massive advantage. It's like picking a cheaper truck but ignoring toll fees: the total haul cost can be higher.
Second, task complexity is often invisible at routing time. 'Summarize this contract' looks easy, but might trigger retrieval, compliance checks, tool calls, and multiple refinement rounds. Conversely, a technical query might be handled cheaply by a small specialized model. Enterprise settings add more variables: data residency, privacy rules, model allowlists. Routing becomes a multi-objective optimization (cost, latency, reliability, compliance), not a simple difficulty sorter.
This reveals a broader trend: AI is shifting from calling a single API to orchestrating a system. Model routing must evolve into a continuous, dynamic control plane that senses context reuse, latency windows, compliance costs, and regional model availability. The mental model moves from 'model selection' to resource scheduling, where models are compute units evaluated on real-time performance and placement, not static leaderboards.
What can developers do now?
- Stop trusting sticker prices. Measure your own workload's cache hit patterns. If your agent reuses long system prompts or tool descriptions, bake cache efficiency into your model evaluations.
- Add runtime feedback to routing. Don't rely solely on an initial classifier; let the agent re-assess complexity after the first step and hand off if needed.
- Incorporate governance early. If your system may enter regulated industries, treat model allowlists and data locality as first-class routing variables, not afterthoughts.
The counterintuitive takeaway: cheaper per-token pricing and shorter reasoning chains can still cost you more. In AI engineering, context reuse efficiency is becoming a more powerful economic lever than raw model intelligence. And because task difficulty is so hard to predict upfront, future routers may need agents that 'test the waters' before deciding where to commit.
This note doesn't offer a silver bullet, but a clear starting point: model routing is not a simple choice — it's the beginning of a systems engineering journey.
Analysis by BitByAI · Read original