Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents

You think routing is a simple problem, but it's redefining itself

In the narrative of AI infrastructure, "routing" has never been the star. It hides behind load balancers, inside model gateways, like a dutiful traffic cop: look at what this message looks like, route it to the appropriate model. Cheap small models for simple questions, expensive large models for complex tasks—this logic sounds perfectly reasonable, until agents show up.

The core finding from the vLLM Semantic Router team can be summarized in one sentence: the optimal decision for a single message is often the optimal disaster for the entire session.

From "where should this message go" to "can this session be moved"

Imagine a typical coding agent workflow: the user says "refactor this module and run tests," the model generates a tool call, the tool returns results, the user adds "fix the failing case," then maybe goes idle for a few hours before sending "continue."

A traditional prompt router sees five separate text segments. It might think the "tool result" segment is short and cheap to throw at a small model; seeing "continue," it reruns the selection logic; or because the current message is brief, it abandons a prefix cache that's been warming up for a dozen turns and sends the request to another backend.

Every one of these "optimizations" is a disaster. Tool results sent to models that didn't initiate the call, continuation IDs pointing to non-existent physical backends, hot caches replaced by cold starts—these aren't bugs, they're architectural blind spots.

SAAR's solution is straightforward: let the router itself have session memory. Not by dumping memory into the model, but by having the router maintain its own session state. It needs to know whether this session is currently stuck in a tool loop, trapped in non-portable provider state, or whether the "sunk cost" of prefix cache has grown too high to justify switching.

A counterintuitive cost formula

Most people calculate model switching costs by looking at token price differences. The SAAR team introduces a more systematic perspective: prefix-cache-aware switch pricing. Prefix cache isn't a nice-to-have; in long-horizon agents, it's essential. A coding session that has run for 20 turns—the KV cache from the first 19 turns is what makes the 20th turn's "fix it" respond instantly. Switching models means all of this goes to zero, and traditional routing completely misses this accounting.

The test data is hard: 21,600 deterministic turns, 79.29% fewer switches, 3,836 unsafe switches intercepted, estimated physical model cost reduced by 78.71%. More crucially, across 2,896 live AMD ROCm requests, zero observed session continuity violations. This isn't a lab toy; these are production-grade metrics.

This reveals a deeper trend: infrastructure itself is becoming "agentized"

SAAR's ambition goes beyond a single feature. It signals that the design unit of AI infrastructure is shifting from "request" to "session." This shift is as fundamental as the move from "process" to "thread," or from "stateless HTTP" to "WebSocket persistent connections."

As agents become the primary interaction pattern, the entire stack needs rethinking: load balancers need to understand session affinity, cache layers need to understand trajectory locality, and even billing models may need to shift from "per-token" to "per-session duration." The evolution of vLLM Semantic Router from prompt routing to session routing is a microcosm of this larger trend.

What this means for you

If you're building agent gateways, model schedulers, or any multi-model orchestration system, you should ask yourself three questions now:

First, does your routing decision have session context, or are you "guessing" the intent of each message every time?

Second, do you explicitly manage "cannot switch" as a state, rather than relying on post-hoc error reporting?

Third, does your cost model account for the hidden costs of cache invalidation and state migration?

SAAR isn't the only answer, but it defines the new boundary of the problem. In the next six months, we'll see more "session-aware" infrastructure components emerge—not because it's trendy, but because it's inevitable for the agent era.

One final surprise

The most easily overlooked detail in this blog: AMD is in the author list, and testing heavily relies on ROCm. This means session-aware routing isn't just a vLLM community experiment; it's a direction that chip vendors are also betting on. When AMD starts caring about "how agent sessions flow through GPU clusters," the industry signal is clear enough.