Why not use F1 and the default threshold?

The default was set by the API without seeing your data, and F1 weights precision and recall equally when in practice they are not equal, which Husain and Saha say in the post. Choose the weighting by what a wrong pick costs to undo: cheap to reverse on Tuesday, weight recall; touches a customer, money, or a data model, weight precision.

What is the calibration check?

Whether a score of 0.80 means about 80% of traces with that score fail. Husain and Saha plot predicted probability against observed failure rate in groups of about 20 traces; Platt scaling improved some models and made one slightly worse. If the curve moves between the hundred you tuned on and the next hundred, the traffic moved and the threshold is next.

How often do you re-tune?

On a date someone owns, with the next hundred labeled traces. My 89% held-out classifier was at 61% in production within six weeks because the held-out set was English-heavy. A threshold tuned on October's traces is a measurement of October. The card's sixth line is the name and the date of the next measurement.

How to Tune a Decision Model Threshold for Evals

How do you tune a decision model's threshold for evals?

THE SHORT ANSWER

Set the floor first, then tune, then date it. Hamel Husain and Antaripa Saha (October 20, 2026) ran six decision models over 100 labeled traces and showed the default 0.50 threshold can catch nothing: a model scoring every failure 0.40 and every pass 0.10 is perfect at 0.30. Their method: require minimum precision and recall for your use (80% and 80% for alerting), tune each model's threshold on your traces with stratified five-fold cross-validation, compare cost only after tuning (0.82 cents per 100 runs at F1 0.79 for Perplexity against 3.72 cents at 0.87 for OpenAI Decisions), and check calibration, since one model's 8% group failed 60% of the time. Then add the two lines they could not: a sentiment classifier I shipped went from 89% held out to 61% in production in six weeks, so the card carries a name and a date for the next hundred traces.

The threshold is where a score becomes a verdict, and the API chose it without seeing your traces. Tune it on your labeled hundred with the floor set first, check that the probabilities mean what they say, and write down who reruns both and when, because the tuned number is true on the day it was measured and the traffic does not stay put.

The six-line card, the beta rule, and the six weeks behind line six are in The Threshold Card: Tune the Judge Before You Trust the Score, and Date the Re-Tune. The 89% to 61% classifier and the undo-on-Tuesday rule come from Jev Is 400x Cheaper Per Call. Price the Wrong Pick, and the sliced eval that sees drift a whole-set threshold cannot is The Eval That Caught What the Demo Did Not.

Last reviewed 2026-10-20 · 1 min read

How do you tune a decision model's threshold for evals?

SOURCES

THE LONG VERSION

RELATED ANSWERS