Not all requests deserve the same treatment. A refund question, a bug report and a “tell me a joke” shouldn’t hit the same prompt, the same tools or even the same model. Routing puts a classifier in front, then dispatches each request to a path designed for it.

The analogy

The hospital reception desk. The receptionist doesn’t treat anyone — they ask two questions and send you to the right department: a sprain goes to radiology, a rash to dermatology. Ten seconds of triage saves an hour of misdirected care.

The principle

flowchart LR
    IN([request]) --> R["LLM: classify"]
    R -->|billing| A["path A: refunds"]
    R -->|technical| B["path B: tech support"]
    R -->|other| C["path C: small talk"]
  • The router is one cheap, fast call (or even a non-LLM classifier): “which category is this?”
  • Each path is optimized separately: its own prompt, its own tools, its own model — a small model for chit-chat, a strong one for legal questions.
  • Paths don’t pollute each other: improving the refund prompt can’t break tech support.

A concrete example

A customer-support entry point:

router(msg) → "billing" | "technical" | "other"

billing   → prompt with refund policy + invoice tools
technical → prompt with product docs + diagnostic tools
other     → tiny model, friendly canned behavior

Each branch stays short and sharp, instead of one prompt trying to be three experts at once.

When to use it

  • Inputs fall into distinct categories that are handled genuinely differently.
  • You want cost control: send the easy 80% to a cheap model, reserve the expensive one for the hard 20%.
  • Separation of concerns matters — separate prompts beat one prompt with fifteen “if the user asks about X” clauses.

When to avoid it

  • The categories blur together or a request needs several skills at once → look at orchestrator–workers.
  • Only one real path exists: a router in front of a single branch is just added latency.

The classic trap

The silent misroute. If the classifier is wrong, the specialist confidently answers the wrong question. Give the router an “unsure” exit that falls back to a general path or a human — a receptionist who admits “I don’t know, let me check” beats one who guesses.