Building Evals: Error Analysis, Eval Types, the Loop, the Gate

Most teams build the judge first. They write a rubric from imagination, point a model at it, and get a score that measures nothing anyone cares about. Then they wonder why the number goes up while the customers keep complaining.

Teresa Torres's hands-on guide to AI evals fixes the order, and this chapter is her three steps written as a routine you can run every week, with one step added at the end. I wrote why the eval is the spec two waves ago. This is how you build one.

The short version

Four steps. One, error analysis by hand: read fifty outputs, write down the two or three mistakes that recur, rank them by what they cost a customer. Two, match each error to an eval type: code assertion for anything deterministic, golden dataset for one-right-answer questions, LLM-as-a-judge for semantic judgment, customer feedback signals for real usage, always cheapest first as a filter for the expensive one. Three, the loop: fixed inputs, baseline, one change, compare across every eval, log it. Four, the gate: for each action, decide whether it is reversible, and let the eval's pass history buy autonomy only on the reversible side. Correctness is the product team's definition, never the vendor's. Credit for steps one through three goes to Torres. The fourth is mine, and it is the step that makes the first three matter for an agent.

Step one: error analysis, by hand

Set aside two hours. Pull fifty real outputs from the feature or workflow. Not curated, not the best ones. The last fifty.

Read each one and write a note. Not a score, a note: what is wrong with this one, in a sentence. After fifty you will have a list of maybe fifteen distinct complaints, and three of them will have shown up ten times each. Those three are your errors. Everything else is noise for now.

For a production product, the same thing at scale: log every failure a customer reports or a reviewer catches, categorize, and rank by cost. The ranking matters. An error that embarrasses you once a month is not the same as an error that costs a customer an hour once a day, and the eval budget goes to the second one.

Torres is exact about why this cannot be skipped or delegated. Correctness is context-dependent. The same question gets a different right answer for a student and for a professor. The person doing the error analysis is the person deciding what correct means for this product, and no vendor's benchmark makes that decision for you.

Step two: match the eval to the error

Now, for each of the three errors, pick the cheapest measurement that catches it.

If the check is deterministic, write a code assertion. Does every quoted string appear verbatim in the source? Did a banned phrase appear? Is the output within a length or count range? These run in milliseconds and cost nothing, and they catch a surprising share of what matters.

If the input is small and there is one right answer, build a golden dataset. Thirty to a hundred input and expected-output pairs. Classification, routing, extraction, factual lookups. Score is exact match or close to it.

If the judgment is semantic, and only then, build an LLM-as-a-judge. Write the rubric from the disagreements you had with yourself in step one, not from a blank page. The rubric answer covers the extraction. Keep a rubric dimension only if failing it maps to a real cost.

And for real usage, instrument the implicit signals: regeneration, editing, follow-up questions, abandonment. These are not a fourth eval you build. They are the observation record the product generates on its own, and they are the only eval that improves without you.

The pattern to steal from Torres's opportunity-tree example: chain them. Code counts the children; the judge is only called when the count exceeds a threshold. Filter with the free thing, spend on what survives. That is also the margin discipline this handbook argues for in gross margin is your job now, because every judge call is a model call.

Step three: the loop

Fix the inputs. Fifty to a hundred cases that look like production, including the ugly ones. Run every eval and record the baseline.

Change one thing. A prompt edit, a model swap, a retrieval change, a reordering of the orchestration. One.

Run every eval again and compare all of them, not the one you meant to move. This is the discipline most teams lose first. A prompt change that fixes hallucinated quotes routinely makes the tone worse, and if the tone eval was not run you find out from a customer.

Log the experiment: what changed, every score before and after, and whether you kept it. The tenth experiment gets compared to the baseline and to the best-so-far, not to last week, and the log is what makes that possible. The starter kit has the template.

Iterate until the error you ranked first in step one is at a level you can defend. Then go back to step one, because the next fifty outputs have a new top three.

Step four: the gate

Torres's loop answers whether a change worked. For an agent that acts, there is a second question, and it decides whether the thing is safe to run: what is the system allowed to do on its own?

For every action the system can take, write down one thing. Can a person undo it cheaply? Draft an email, yes. Update a record with a receipt, yes. Send money, message a customer under the company's name, delete anything, no.

Then the rule. A passing eval on a reversible action buys autonomy: let it run, keep the receipt, watch the observation record. A passing eval on an irreversible action buys a draft and a human, until the eval has a long pass history and a person has confirmed the labels in the golden dataset. And model confidence is never the gate. Confidence is a number the model produces about itself. It says nothing about what happens when the number is wrong. The eval is the evidence input; the reversibility column makes the call. The harness argument has the full version of this gate.

One warning that belongs here because it breaks the gate silently. A golden dataset labeled by a cheap model, or by the model you are evaluating, measures agreement with itself. A cheap tier passes things a frontier model catches. Record which tier produced every label, human, frontier, or cheap, and put a person on every label that gates an irreversible action. Where the judge and the human disagree, keep the row. It is the most valuable case in the set.

What this looks like as a week

Monday, an hour: read the last fifty outputs, update the error list. Tuesday, two hours: build or extend the eval for the top error, cheapest type first. Wednesday, run the loop on one change. Thursday, read the observation record: what did users regenerate, edit, abandon. Friday, review the gate: did any eval earn enough history to move an action from draft-and-human to autonomous, and did anything get reversed that should move the other way.

Five hours. It replaces the review meeting where six people argue about whether the output looks right.

This chapter sits in the running argument on AI Product Management: deciding what correct means, and what a correct answer is allowed to do, is the part of the job that did not get cheap.

Start this week

Do step one and nothing else. Fifty outputs, a notes column, three recurring errors ranked by cost. Do not build a judge. Do not write a rubric.

If you cannot name the three errors by Friday, you do not know what correct means for your product yet, and no eval will tell you.

Sources: Teresa Torres, "AI Evals: A Hands-On Guide for Product Teams," Product Talk (Sept 2, 2026) · AI Eval Starter Kit, falkster toolkit