# AI Eval Starter Kit Six templates that turn Teresa Torres's three-step evals method (error analysis, choose the eval type, run the loop) into files you can copy this afternoon, plus the fourth step from the falkster handbook: the reversibility gate. Credit for the method: Teresa Torres, "AI Evals: A Hands-On Guide for Product Teams," Product Talk, September 2, 2026 (producttalk.org/ai-evals). The gate, the tier column, and the experiment log are falkster additions. Use them in order. Do not build the judge (template 5) until template 1 has fifty rows. --- ## 1. Error analysis log **The job:** read fifty real outputs, one note each, then count. This is the step everyone skips and the one every eval depends on. ``` # Error analysis: [feature or workflow name] # Date: [YYYY-MM-DD] Reviewer: [name] Sample: last 50 outputs, uncurated | # | Input (short) | Output (short) | What is wrong, one sentence | Error tag | |---|---------------|----------------|-----------------------------|-----------| | 1 | | | | | | 2 | | | | | ... | 50| | | | | # After 50 rows, tally the tags: | Error tag | Count | Customer cost (1 low to 5 high) | Rank = count x cost | |-----------|-------|----------------------------------|---------------------| | | | | | | | | | | | | | | | # Top three errors, by rank. These are the only ones that get an eval this cycle. 1. 2. 3. ``` **Rule:** the reviewer is a person who knows what good looks like for this product. Correctness is the product team's definition. Never the vendor's, never the model's. --- ## 2. Eval type chooser **The job:** for each of the top three errors, pick the cheapest measurement that catches it. Work top to bottom and stop at the first yes. ``` Error: [tag from template 1] Q1. Can a piece of code decide pass or fail with no interpretation? (exact string present, banned word absent, count in range, schema valid) YES -> Code assertion (template 3). Stop. NO -> Q2 Q2. Is the input small and is there exactly one right answer? (classification, routing, extraction, factual lookup) YES -> Golden dataset (template 4). Stop. NO -> Q3 Q3. Does deciding require reading the output and judging it? (tone, faithfulness to a source, whether a question is leading, whether a grouping is missing) YES -> LLM-as-a-judge (template 5), with a code filter in front of it wherever Q1 can pre-screen. Stop. NO -> Q4 Q4. Can only real usage tell you? -> Customer feedback signals (template 6). Instrument, do not build. Chain rule: when Q1 can pre-screen, run the code check first and send only the survivors to the judge. Every judge call is a model call you pay for. ``` --- ## 3. Code assertion templates **The job:** deterministic checks that run in milliseconds and cost nothing. ```python # Quote verification: every quoted string in the output must appear verbatim # in the source. Torres's hallucination guard for interview summaries. import re def quotes_grounded(output: str, source: str) -> tuple[bool, list[str]]: quotes = re.findall(r'"([^"]{12,})"', output) missing = [q for q in quotes if q not in source] return (len(missing) == 0, missing) # Red-flag words: phrases the output must never contain. RED_FLAGS = ["as an ai", "i cannot", "guaranteed", "[insert"] def no_red_flags(output: str) -> tuple[bool, list[str]]: hits = [w for w in RED_FLAGS if w in output.lower()] return (len(hits) == 0, hits) # Structure bounds: counts within a range. Torres's opportunity tree check: # code counts the children, the judge is only called above a threshold. def children_in_range(node: dict, lo: int = 2, hi: int = 7) -> bool: return lo <= len(node.get("children", [])) <= hi # Schema: the output parses and has the fields downstream code needs. REQUIRED = {"summary", "decisions", "open_questions"} def has_required_fields(obj: dict) -> tuple[bool, set[str]]: missing = REQUIRED - set(obj) return (len(missing) == 0, missing) ``` **Rule:** an assertion that fails on a real output is a row for the error analysis log, not a reason to loosen the assertion. --- ## 4. Golden dataset format **The job:** small inputs, one right answer each, thirty to a hundred rows. Score is exact match or a tolerant match you define in code. ```csv id,input,expected,label_source,label_confirmed_by,notes 001,"Customer asks for a refund on a cancelled order",refund,human,falk, 002,"Customer asks how to change their password",account,human,falk, 003,"Customer says the invoice total is wrong",billing,cheap-model,,needs human check 004,"Customer asks whether the API supports webhooks",product,frontier,, ``` **The two columns that are not optional:** `label_source` (human, frontier, cheap-model) and `label_confirmed_by`. A dataset labeled by the tier you are evaluating measures agreement with itself. Every row that gates an irreversible action needs a person in `label_confirmed_by` before the eval is allowed to unlock anything. **Scoring script sketch:** ```python def score_golden(rows, predict) -> dict: hits = sum(1 for r in rows if predict(r["input"]) == r["expected"]) return {"n": len(rows), "pass_rate": hits / len(rows)} ``` --- ## 5. LLM-as-a-judge prompt **The job:** semantic judgment against a rubric you extracted from template 1, not from a blank page. Keep a dimension only if failing it maps to a real cost. ``` You are grading the output of a system, not writing a better one. SOURCE MATERIAL (ground truth): {source} OUTPUT TO GRADE: {output} Grade each dimension 1 to 5 and give one sentence of evidence per score, quoting the output where possible. 1. Grounded: every claim in the output is supported by the source material. 5 = every claim traceable, 1 = claims with no basis in the source. 2. Complete: the output covers what the source material requires it to cover. 5 = nothing material missing, 1 = a decision or fact a reader needs is absent. 3. [Product-specific dimension from your error analysis, e.g. "Not leading": the suggested question does not presuppose an answer.] 4. [Second product-specific dimension, only if failing it costs something.] Return JSON: {"grounded": n, "complete": n, "": n, "": n, "evidence": {"grounded": "...", "complete": "...", ...}, "fail": true|false} fail is true if any dimension scores 2 or below. ``` **Rules:** the judge sees the source material, always. The judge never sees the previous score. Run the judge with a cheaper model on volume and the frontier model on anything that gates an irreversible action, and record which one graded each row (`judge_tier`). When the judge and a human disagree, keep the row; it is where the rubric or the judge is wrong. --- ## 6. Experiment log and the gate **The job:** one change at a time, compared across every eval, and the reversibility decision for each action the system can take. ``` # Experiment log: [feature] | Exp | Date | What changed (one thing) | Code pass | Golden pass | Judge mean | Judge fail % | Feedback signal | Kept? | Why | |-----|------|--------------------------|-----------|-------------|------------|--------------|-----------------|-------|-----| | 000 | | BASELINE | | | | | | n/a | | | 001 | | | | | | | | | | | 002 | | | | | | | | | | Compare every experiment to 000 and to best-so-far, never to the previous row. A change that improves the target eval and degrades any other is not kept until the degradation is understood. ``` ``` # Gate: what the system may do on its own | Action the system can take | Reversible by a person in under a minute? | Eval pass history (n runs, rate) | Labels human-confirmed? | Current mode | |----------------------------|-------------------------------------------|----------------------------------|-------------------------|--------------| | Draft reply email | yes | | | autonomous | | Update CRM field w/ receipt| yes | | | autonomous | | Send email as the company | no | | | draft + human| | Issue refund | no | | | draft + human| Mode rules: - Reversible + eval passing -> autonomous, keep the receipt. - Irreversible + eval passing -> draft + human, until pass history is long and every label is human-confirmed. - Any action, eval failing -> draft + human, and a row in template 1. - Model confidence is never the gate. It is one input; the reversible column makes the decision. ``` **Customer feedback signals to instrument (the eval that compounds):** regeneration rate, edit distance between output and what the user kept, follow-up question rate, abandonment, and for agents, reversal rate and time-to-reversal per action. Keyed per action, per outcome. This is the observation record. It is the only eval on this page that cannot be exported to another vendor and the only one that improves while you sleep. --- Weekly cadence that uses all six: Monday, template 1 on the last fifty outputs. Tuesday, build or extend the eval for the top error via template 2. Wednesday, one experiment, logged in template 6. Thursday, read the feedback signals. Friday, review the gate table.