What should the comparison be?

Two numbers per branch: what the calls cost, and what last month's wrong picks cost at the measured error rate for the worst slice. On most cheap-to-reverse branches a decision model wins easily. On the rest, you know what you are buying.

Which branches should move to a decision model first?

The ones you can undo on Tuesday: tagging, topic labels, first-pass screening where a person or a second model looks at what was flagged. Branches that gate a command, message a customer, or touch money wait until a labeled set exists for that branch.

Does a confidence score solve this?

Not on its own. The Amplitude analysis cites a 77-case pilot with 87.4% mean confidence against 75.3% actual accuracy, and a 791-decision study where the decision model agreed with a frontier model 6 to 8 points more often than it matched the labels. A threshold catches uncertain answers. Confident wrong ones pass.

Why does a slice matter?

Because the average hides it. My sentiment classifier's 89% was measured on an English-heavy set. In production, customers writing in other languages were scored as angry regardless of content. I now run language-segment evals every week.

Is Price Per Call the Right Test for Jev?

Is price per call the right way to compare a decision model like Jev?

THE SHORT ANSWER

No. Price per call tells you what you pay when the pick is right. Jev, TypeSafe's decision model, is far cheaper per call: Basia Kubicka's October 6, 2026 explainer puts input at $0.042 per million tokens with free output. But the cost of a decision model sits in its wrong picks, and a cheap call puts a pick at every branch. Amplitude's analysis cites a pilot with 87.4% mean confidence against 75.3% accuracy and notes that escalating uncertain answers does not catch confident wrong ones. A ticket-routing sentiment classifier I shipped scored 89% on held-out data and 61% in production within six weeks, and it cost an 18% slower response on critical tickets for two months. Compare per branch: what a wrong pick does downstream, whether you can undo it, a labeled set by slice, and who re-runs it in week six.

No. It is the number that is easiest to paste into a pull request, and it measures the wrong thing.

Price per call is what a decision costs when the pick is right. Jev is very cheap on that measure. Basia Kubicka's explainer gives $0.042 per million input tokens with free output, and LangChain's Winston Huynh reports one test where a full set of eval judgments cost $0.34 on Jev against $28.17 on Claude Sonnet 4.6.

The cost that decides whether a step should move is in the wrong picks. I shipped a decision model before the name existed: a sentiment classifier that routed angry support tickets to senior reps. It scored 89% on a held-out set and 61% in production within six weeks, because the set was English-heavy and customers writing in other languages were scored as angry. Response time on critical English-language tickets went up 18% for two months before I caught it. Nothing in that story was about what a call cost.

Vinay Goel and Ram Soma at Amplitude published the accuracy side: a pilot with 87.4% mean confidence against 75.3% accuracy, and the point that a cascade catches uncertain answers and not confident wrong ones.

So compare per branch. What does a wrong pick do downstream? Can you undo it on Tuesday? Is there a labeled set for this branch, split by the slices you care about? Who re-runs it in week six? Move the cheap-to-reverse branches first.

The full argument is in Jev Is 400x Cheaper Per Call. Price the Wrong Pick., and the reversibility triage is from The Cost of Being Wrong Is the Only Number That Matters Now.

Last reviewed 2026-10-06 · 2 min read

Is price per call the right way to compare a decision model like Jev?

SOURCES

THE LONG VERSION

RELATED ANSWERS