What are the parts of LangChain's router?

A base prompt that tells the classifier to pick the least expensive model likely to complete the task, plain-language criteria for each of three tiers (fast, balanced, performance), and a classifier model. It runs once, on the first human message of a thread.

What do you count when you run the classifier against the labeled set?

Two errors, separately. Routed too low is the quality risk. Routed too high is cost you did not need to spend. Fix the criteria until the first number is one you can defend, then look at the second.

Who should label the 150 threads?

Whoever reads the most model output, usually the PM and one engineer. One column for the cheapest tier you would trust with the thread, and one for why in a few words. The second column becomes the tier criteria.

How often did threads need the top model in LangChain's test?

10% of routed threads went to the performance tier, 56% to balanced, and 34% to fast. Median thread cost was $0.097, $1.50, and $2.88 by tier. That split is for Open SWE's task mix and should not be assumed for another agent.

How to Build a Model Router for an AI Agent

How do you build a model router for an AI agent?

THE SHORT ANSWER

Put the router in the agent's harness, where the task context already lives, and build a labeled set before it sees live traffic. LangChain's router for Open SWE (2026-10-01) picks one of three model tiers on a thread's first message and cut median cost per thread from $2.61 to $0.94, 64% less, with merged PRs flat across 973 threads. They validated it with a live A/B on their own engineers. For a customer-facing agent, pull 150 real threads, label each with the cheapest tier you would trust, write the tier criteria from the labels, run the classifier against the set, and rerun it on every model swap. At Smartcat, 150 labeled translation requests gave engineering a working router in three days.

LangChain's post is the build guide: understand the tasks from traces, pick models along the cost and intelligence curve, put the router in the harness, and track task outcomes. Label Before You Route adds the step that makes it safe when the users are customers and not your own engineers, a labeled set of 150 real threads that the classifier has to pass before live traffic.

The method is the one in The Eval Is The Spec, where 150 labeled translation requests at Smartcat replaced a routing PRD, and the artifact it feeds is the routing table in Gross Margin Is Your Job Now: model, fallback, and eval threshold per surface, reviewed quarterly. Both belong to the Enterprise AI Agents argument about what each successful outcome costs.

Last reviewed 2026-10-02 · 1 min read

How do you build a model router for an AI agent?

SOURCES

THE LONG VERSION

RELATED ANSWERS