Torres Wrote the Evals Guide PMs Needed. Here Is Step Four.

Teresa Torres published a hands-on guide to AI evals for product teams on Product Talk this month, and it is the primer I have been wishing existed so I could stop writing it badly in the margins of other posts. Read hers first. This post assumes you have.

Then I want to add three things, from the vantage point of building a multi-tenant agent product where the evals are not a measurement. They are what decides what the agents are allowed to do.

The short version

Torres's framework is three steps: error analysis by hand to find the recurring mistakes, choose an eval type that matches the error (code assertions, golden datasets, LLM-as-a-judge, or customer feedback signals), and run an experimentation loop of baseline, change, compare. Her rule that correctness is the product team's job and never the vendor's is the line to frame. I'd add a fourth step, where a passing eval on a reversible action buys autonomy and a passing eval on an irreversible one buys a draft and a human, so the eval becomes the gate rather than the scoreboard. I'd add a cost column, because every judge call is margin. And I'd add a warning: a golden dataset labeled by a cheap model measures agreement with a cheaper version of the thing you are evaluating, not correctness. Her fourth eval type, customer feedback, is the one that compounds, because it is the observation record and it ports nowhere.

What Torres got right, in her order

She starts where most guides skip: error analysis. Before you build anything, read the outputs yourself. Not a sample of five. Enough to see the two or three mistakes that keep recurring. For a personal workflow that is a deep read. For a production product it is a logged, categorized list prioritized by how much each error costs a customer. Everything downstream depends on this step being done by a person who knows what good looks like, and she is right that it cannot be delegated to the model.

Then the eval type follows from the error, not from what is fashionable. Her table is the useful artifact in the piece and I am going to restate it because it should be on a wall somewhere. Deterministic errors get a code assertion, fast and free, the kind that checks whether a quote actually appears in the transcript or whether a red-flag word slipped through. Small input, small output, one right answer gets a golden dataset, which is how you eval classification, routing, and factual questions. Semantic judgment gets an LLM-as-a-judge with a rubric. And real usage gets customer feedback signals, the implicit ones: did they regenerate, did they edit, did they ask a follow-up.

Her examples are worth the click on their own. A hallucination guard on interview summaries that searches the transcript for every quoted string. An interview coach with separate evals for leading questions, general questions, and suggestions that had already been answered, some caught by code and some by a judge. An opportunity tree eval where code counts the node distribution and the judge only gets called when a parent has too many children. That last one is the pattern to steal: filter with the cheap thing, judge only what survives.

Then the loop. Define inputs that look like production. Take a baseline. Change one thing, a prompt, a model, the orchestration. Compare across every eval, not just the one you were trying to move. Iterate.

And the line I would put on the wall next to the table: don't let a vendor define correctness for your product. That is the product team's job. A verbose explanation is right for a student and wrong for a professor asking the identical question. I made a version of this argument in The Eval Is The Spec, that the eval is the contract and the definition of done. Torres's version is more practical than mine, and she frames evals as a discovery habit sitting next to interviews and assumption tests, which closes a loop I had left open: interviews ask whether you are solving the right problem, evals ask whether your change actually worked.

Step four: the eval becomes the gate

Here is where I'd extend it.

Torres's loop answers one question: did our change work? For a workflow you run yourself or a feature a person reviews before it ships, that is the whole job. For an agent that acts, there is a next question, and it is the one that decides whether the thing is safe to run. What is the system allowed to do on its own?

The evals are how you answer it, and they answer it differently depending on what the action costs if it is wrong.

A passing eval on a reversible action buys autonomy. Draft an email, update a record with a receipt, file a summary. If the eval says the output is good and a person can undo it in a minute, let it run. A passing eval on an irreversible action buys a draft and a human. Send money, touch the ledger, message a customer under the company's name. The eval passing is necessary and it is not sufficient, because the cost of the eval being wrong is the action, and no eval score is worth that.

The mistake I see in every agent demo this month is using confidence as the gate. The model is 92 percent sure, so it acts. Confidence is a number a model produces about itself. It says nothing about what happens if the number is wrong. I wrote this up in the harness post: we gate on three things at once, how confident the system is, how reversible the action is, and how strong the evidence behind the rule that fired. The eval is the evidence axis. It is never the whole gate.

So the fourth step is: for each action your system can take, write down whether it is reversible, and let the eval's pass rate on that action decide autonomy only within that limit. Torres's loop tells you the eval is improving. The fourth step tells you what the improvement is allowed to unlock.

The cost column

Torres's ordering, code assertions first, judges last, is right, and she gives the reason as speed and cost. I'd say it louder: it is a margin decision.

Every LLM-as-a-judge call is a model call. Run a judge across a thousand-case golden dataset on every prompt change and you are spending real money to find out whether you broke something, and for anyone running hosted multi-tenant AI that spend is gross margin, not an expense line. The fix is her opportunity-tree pattern generalized. Code handles everything deterministic. A cheap tier handles the volume judgments where a miss is survivable. The frontier model handles the judgments that gate something expensive. And every eval result records which tier produced it.

That last clause is where the warning lives, and it is the thing I would most want added to her guide.

The cheap tier warning

A cheap model passes things a frontier model catches. Spotify's shunt team found this the hard way when their cheap tier missed a thread-safety bug the expensive model caught in seconds. I wrote about the enforceable half of that story earlier this week, and the eval version of it is worse than the coding version.

If your golden dataset was labeled by a cheap tier, or by the same model you are evaluating, and no person ever confirmed the labels, then your eval measures agreement with a cheaper version of the thing you are evaluating. It will report a high pass rate right up until a customer finds the class of error that both models share. That is not an eval. It is a mirror.

The fix is cheap and boring. Record the tier that produced each label: human, frontier, cheap. Get a person on every label that gates an irreversible action. And when a judge and a human disagree, that disagreement is the most valuable row in the dataset, because it is where your rubric is wrong or your judge is. The rubric answer on this site says to grade thirty real outputs on gut feel and extract the rubric from the disagreements you had with yourself. The same rule applies between you and the judge.

The eval that compounds

Torres lists customer feedback as the fourth eval type and describes it correctly: regeneration, editing, follow-up questions. I'd give it a different name, because it is not a fourth type. It is a different kind of thing.

The other three evals are things you build and can rebuild anywhere. The golden dataset is a file. The judge is a prompt. The assertions are code. All three port to the next vendor in an afternoon. Customer feedback signals are the observation record: what the system did, what a person changed, how fast, and whether the outcome closed. That record only exists because the system was acting in this product long enough to accumulate it, and it is the only eval that gets better with every interaction without anyone touching it.

I made the general argument in Context Is King. Written Context Is Rented. The eval-specific version is: the three built evals tell you whether a change worked on the cases you thought of. The fourth tells you whether it worked on the cases you didn't. Write it down per action, keyed to the outcome, and let it feed how much the system trusts its own output next time. That is the loop Torres describes, run by the product instead of by the team.

What I'm running in Heidi

Every agent action in Heidi carries a receipt and every outcome gets labeled against the decision that produced it. That started as a guard-layer requirement and it turns out to be the fourth eval type by another name. Where I fell short of Torres's guide is step one: I built the judges before I had done the error analysis by hand at the scale she describes, and I paid for it with a rubric that graded things nobody cared about. The Building Evals chapter I published alongside this post is her three steps plus the fourth, written as a weekly routine, and the AI Eval Starter Kit in the toolkit is the error analysis log, the type chooser, the assertion and judge templates, and the experiment log, so the first hour is copying instead of designing.

Try this week

Take one AI feature you own. Read fifty of its outputs yourself, and write down the three mistakes that recur. That is Torres's step one and it is the step everyone skips.

Then, for each mistake, answer one question she does not ask: if the system acts on this output and the output is wrong, can you take it back? Where the answer is no, that mistake gets a human in the loop until its eval has a hundred passes behind it and a person has confirmed the labels. Where the answer is yes, ship the eval and let it run.

That is the whole difference between an eval as a scoreboard and an eval as a gate.

Sources: Teresa Torres, "AI Evals: A Hands-On Guide for Product Teams," Product Talk (Sept 2, 2026) · Dimitri Mazmanov, "Portal by Spotify cut my Claude Code token usage by 90%," Spotify Engineering

This is one more entry in the running argument on AI Product Management: the parts of the job we trained people for got cheap, and the part nobody trained for, deciding what correct means and what a correct answer is allowed to do, is what is left.

Related answer: What is Teresa Torres's three-step approach to AI evals?