Teresa Torres wrote the hands-on guide to AI evals that product teams needed. Three steps: error analysis by hand, match the eval type to the error, run the experimentation loop. This kit is her method as files, so the first hour is copying instead of designing, plus the fourth step I argue for in the companion post: the gate that decides what a passing eval is allowed to unlock.
The short version
Six templates in one markdown file. An error analysis log that makes you read fifty outputs before you build anything. An eval type chooser that walks from the cheapest measurement to the most expensive and stops at the first that fits. Four code assertions in Python, including Torres's quote-verification hallucination guard. A golden dataset format with the two columns most teams forget, who labeled the row and who confirmed it. An LLM-as-a-judge prompt with a rubric skeleton and a rule that the judge always sees the source. And an experiment log paired with a gate table, one row per action, where reversibility decides autonomy and model confidence never does. Method credit to Torres. The gate, the tier columns, and the log are additions.
What is inside
The error analysis log is fifty rows, one note each, then a tally ranked by count times customer cost. The top three errors are the only ones that get an eval this cycle. If you take one thing from this kit, take the discipline of not opening template five until template one is full.
The eval type chooser is four questions in order. Can code decide it? Is there one right answer? Does it need semantic judgment? Can only real usage tell you? Stop at the first yes. When code can pre-screen, chain it in front of the judge, which is Torres's opportunity-tree pattern and the single biggest lever on eval cost.
The code assertions are copy-ready: quote grounding against a source, red-flag phrases, count bounds, schema checks. Each one that fails on a real output is a row for the log, not a reason to loosen the check.
The golden dataset CSV carries label_source and label_confirmed_by. A dataset labeled by the tier you are evaluating is a mirror. The Building Evals chapter explains why those two columns are the difference between a score and a measurement.
The judge prompt grades against a rubric you extract from your own disagreements in the log, returns JSON with evidence per dimension, and records which model tier did the grading.
The experiment log compares every change to the baseline and to best-so-far across every eval, never only the one you meant to move. The gate table beneath it is the fourth step: reversible and passing means autonomous with a receipt, irreversible and passing means draft and human, and the observation record, regenerations, edits, reversals, is the eval that keeps improving on its own.
How to use it this week
Monday, template one on the last fifty outputs. Tuesday, build the eval for the top error using template two to pick the type. Wednesday, one experiment, logged. Thursday, read the feedback signals. Friday, review the gate table and move any action whose eval has earned it.
Five hours, and it replaces the meeting where six people argue about whether the output looks right. The method is Torres's. Read her guide first; it is better than my summary of it.
Sources: Teresa Torres, "AI Evals: A Hands-On Guide for Product Teams," Product Talk (Sept 2, 2026) · The Eval Is The Spec, falkster handbook
Part of the running argument on AI Product Management: deciding what correct means, and what a correct answer is allowed to do, is the part of the job that did not get cheap.
Related answer: How do you write an eval rubric for an AI feature?
Downloadable bundle · 1 files
Pick your level. Grab the JD.
Copy into your ATS, fork for your org, or send to a recruiter as-is.
Frequently asked
What is in the AI Eval Starter Kit?+
Six templates in one markdown file: an error analysis log for fifty outputs with a cost-ranked tally, an eval type chooser that walks from cheapest to most expensive, four code assertion functions in Python, a golden dataset CSV format with label-source and confirmation columns, an LLM-as-a-judge prompt with a rubric skeleton, and an experiment log paired with a reversibility gate table. It follows Teresa Torres's three-step method with a fourth step added.
Which template do I start with?+
The error analysis log, and nothing else until it has fifty rows. Read the last fifty real outputs, write one sentence about what is wrong with each, tag them, and rank the tags by count times customer cost. The top three errors are the only ones that get an eval this cycle. Building the judge first is the most common mistake and the reason most eval scores measure nothing.
Why does the golden dataset have a label_source column?+
Because a dataset labeled by a cheap model, or by the model you are evaluating, measures agreement with itself rather than correctness. The column records whether each label came from a human, a frontier model, or a cheap tier, and label_confirmed_by records who checked it. Any row that gates an irreversible action needs a person in that column before the eval is allowed to unlock autonomy.
What is the gate table?+
The fourth step. One row per action the system can take, with a column for whether a person can undo it in under a minute. Reversible plus a passing eval means autonomous with a receipt. Irreversible plus a passing eval means draft and human until the pass history is long and the labels are human-confirmed. Model confidence never decides; the reversible column does.
How much does running these evals cost?+
Code assertions and golden dataset scoring cost nothing. Every LLM-as-a-judge call is a model call, which is why the chooser makes you try code first and why the judge prompt tells you to run a cheaper model on volume and the frontier model only on rows that gate something expensive, recording which tier graded what. For hosted AI products that spend is gross margin.
Comments (0)
Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn