The Threshold Card: Tune the Judge Before You Trust the Score, and Date the Re-Tune

Hamel Husain and Antaripa Saha published a comparison this morning that I had wanted someone with a labeled dataset to run. Six decision models, the new class of cheap classifiers that pick from answers you supply and return a probability, pointed at 100 production traces from an apartment-leasing agent, each asked the same question: does this trace contain a substantive failure.

The leaderboard is the least useful part of the post, and they say so. The useful part is a number almost nobody sets on purpose.

The short version

Hamel Husain and Antaripa Saha's "How To Use Decision Models for Evals" (October 20, 2026) ran Jev, Perplexity Decider, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions over 100 labeled traces and found that the default decision threshold of 0.50 can leave most of a model's performance on the table. Their method: set a precision and recall floor (80% and 80% for alerting), tune each model's threshold on your own labeled traces with cross-validation, then compare cost, which put Perplexity at 0.82 cents per 100 eval runs for F1 0.79 and OpenAI Decisions at 3.72 cents for F1 0.87. Then check calibration, because one model said 8% where 60% of traces failed. The number I add is what happens next: a classifier I shipped scored 89% on held-out data and 61% in production within six weeks. The drop is a threshold card, one per judge, six lines: failure mode, threshold, beta from the undo cost, precision and recall on the last hundred, the calibration check, and a name and a date for the next hundred.

The default nobody set

A decision model returns a score between 0 and 1. The API turns that score into a verdict at 0.50. Their example is the whole argument in two sentences: a model could assign every failure a score of 0.40 and every passing trace 0.10. It classifies perfectly at a threshold of 0.30. At the default it catches nothing.

Real scores overlap, so the threshold is a trade. Lower it and you catch more failures and raise more false alarms. Precision is the share of flags that are real; recall is the share of real failures you flagged. Their recommendation is to set the floor first. For an alerting system they want at least 80% on both, so that the pager is not lying and the misses are bounded. On their data, three of the six models could meet that floor at some threshold. The other three could not at any threshold, which is the point of setting the floor before you look at cost: it rules models out.

Then they tune, using stratified five-fold cross-validation, choosing thresholds on the training folds and measuring on the held-out ones, restricted to thresholds that meet both minimums on the training data. The post includes the prompt to hand a coding agent, and it is a good prompt, down to "if none qualifies, report that rather than relaxing either requirement" and "reuse the saved scores without making new model calls."

With thresholds tuned, cost. Perplexity was the cheapest qualifying model at 0.82 cents per 100 eval runs for an F1 of 0.79. OpenAI Decisions, which runs on GPT-6 Luna, reached 0.87 at 3.72 cents, about 4.5 times the cost. That is a real trade, and the right one depends on how many traces you score a day and what a missed failure costs you. Kill price per call as the comparison, as I argued in Jev Is 400x Cheaper Per Call. Price the Wrong Pick, and this is what the comparison looks like once the price is attached to a tuned verdict instead of a raw call.

Can you believe the 0.80

The second half of the post is the half I would hand to every PM who has ever put a confidence score on a dashboard. If a judge says 0.80, do 80% of the traces with scores like that actually fail? They plot it. One group of Clef Flash traces had an average predicted failure probability of about 8%, and 60% of them contained failures. Platt scaling, a simple curve fit on the log-odds, pulled Perplexity, Liquid d1, and Clef Flash closer to the line. Jev and d1-ultra needed little adjustment. OpenAI's got slightly worse with the version they used, which is their point: calibration is a thing you measure, not a thing you assume a fix will deliver.

That is the right instinct and it matches what happened to me when the number I trusted was my own. The pilot numbers that Vinay Goel and Ram Soma reported at Amplitude, 87.4% mean confidence against 75.3% accuracy, were the same lesson from a different vendor: confident and wrong is a real category, and escalating only the uncertain answers does not catch it.

The number they could not have

Everything above is measured on the day the hundred traces were labeled. Here is the part of the story that only shows up after that day.

A sentiment classifier I shipped, text in, one of a few labels out, a decision model before the name existed, scored 89% on held-out data. Within six weeks in production it was at 61%. The held-out set was English-heavy, and customers writing in other languages were scored as angry no matter what they wrote. The threshold was right. The held-out set was right. The traffic moved, and nothing in the system was dated.

Husain and Saha's tuning is correct and it is a measurement of October. The traces the judge sees in December are not the traces it was tuned on: a new intake form, a new region, a model upgrade upstream, a product change that adds a tool call the judge has never seen in a trace. The threshold does not know. The calibration curve, which is the thing they just taught you to draw, is the instrument that would notice first, because the predicted probabilities stop matching the observed failures before anyone looks at precision.

The Eval That Caught What the Demo Did Not is the same lesson on the input side: 200 real cases split into named slices, the average fine, the long-input slice failing outright. The slice is where drift shows up, and a judge that was tuned on the whole hundred cannot see a slice move.

The drop: a threshold card

So the deliverable is small. One card per judge, six lines, kept next to the judge's prompt in the same file or the same ticket.

1. The failure mode this judge scores. One. Husain and Saha recommend a separate judge per failure mode and kept their task broad only to make the differences between models visible. "Any substantive issue" is a research setting, not a production judge.
2. The threshold, and the model it was tuned on. The number, the model and version, and the date of the hundred traces it came from.
3. The beta. F1 weights precision and recall equally, and the post says plainly that in practice they are not equal and leaves the weighting to you. The rule I use is the undo rule from Price the Wrong Pick: what does a wrong pick do downstream, and can you reverse it on Tuesday? Tagging reviews by topic, cheap to undo, so weight recall and let the judge over-flag. Gating whether a command is safe to run, or anything that touches a customer, money, or a data model, weight precision and make every flag earn itself.
4. Precision and recall on the last hundred labeled traces. Not the first hundred. The last.
5. The calibration check. Did 0.80 mean 80% on those hundred? If the curve moved, the traffic moved, and the threshold is next.
6. A name and a date for the next hundred. Who labels them, when, and the threshold and calibration get rerun with the two prompts from their post. My 89% was true on the day it was measured, and nobody owned the next measurement. That is the whole six weeks.

Their post gives you lines one through four this afternoon, with the prompts. Lines five and six are the ones I learned the expensive way.

The thesis of Enterprise AI Agents is that the number that matters is how many agents still complete production work after ninety days and what each outcome costs. A judge with a stale threshold is how an agent fails that test while the dashboard stays green. The card is cheap. The six weeks were not.

This week: find the judge your team trusts most. Write the six lines. If line two says 0.50 and nobody remembers choosing it, run their tuning prompt on the traces you already have labeled. If line six is blank, it is not a judge yet. It is a document.

Related answer: How do you tune a decision model's threshold for evals?

Sources: Hamel Husain and Antaripa Saha, "How To Use Decision Models for Evals," hamel.dev, October 20, 2026, read in full. Randy Olson's broader decision-model versus LLM comparison is cited only as they report it. Vinay Goel and Ram Soma at Amplitude, as cited in Price the Wrong Pick. Falk Gottlob, "Jev Is 400x Cheaper Per Call. Price the Wrong Pick," "The Eval That Caught What the Demo Did Not," and "10 AI Agents I Built That Failed. The Honest Retrospective."