How I Build Eval Rubrics

A product leader I respect DMed me last week asking for eval rubric resources. She runs product in a hard domain, connected labs and drug discovery, where a wrong answer has a real cost. I sent her Hamel Husain and Shreya Shankar, the two best public writers on this, and Anthropic's docs. Good links. Then I realized I disagree with how most teams, including some following those exact links, actually build a rubric. So here is the method I use instead.

The one-line version: the rubric comes last, not first.

The short version

Building an eval rubric is not a writing task, it is an extraction task. Do not start by writing criteria. Start by grading thirty real outputs with nothing but gut feel and a notes column, then extract the rubric from your own disagreements with yourself. Keep a dimension only if failing it maps to a real cost (user harm, trust erosion, or rework), which is downside exposure applied to the rubric. Score each dimension binary, pass or fail, not on a 1-5 scale, because scales hide disagreement and binaries force it into the open. Version the rubric like code, with every model or prompt change as a review trigger. Then ship it as a template the team runs on every diff. The template is not the hard part. The judgment you extract before you write it is.

1. The rubric comes last, not first

Every team that starts by writing criteria produces the same thing: an academic rubric, six tidy dimensions, a 1-5 scale, and a document nobody opens again after launch. It looks rigorous. It is theater. You wrote down what you imagined good would look like before you had looked at a single real output.

Invert it. Pull thirty real outputs from the feature, or from a prototype of it. Open a two-column sheet: the output, and a notes column. Grade each one on gut feel alone. Good, bad, something in between, and a sentence on why. No rubric yet. You are allowed to be inconsistent, because the inconsistency is the data.

By output thirty, two things have happened. Your notes have started repeating the same words, "invented a number," "buried the answer," "hedged when it should have committed." And you have caught yourself grading two similar outputs differently. Those repeated words are your dimensions. Those disagreements with yourself are where the rubric has to get precise. The rubric is what you extract from the notes column, not what you paste in above it.

This is the same move as grading before you write the answer key. You cannot write a good key for questions you have not tried to answer yourself.

2. Tie every dimension to downside exposure

Once you have candidate dimensions, most of them do not survive contact with a simple question: what does failing this cost me?

A dimension earns its place in the rubric only if failing it maps to a real cost. User harm, trust erosion, rework, a support ticket, a lost renewal. "Tone feels off" is not a dimension until you can name what the wrong tone costs. Sometimes you can (a compliance tone in a consumer product reads as cold and churns users), and then it stays. Often you cannot, and it goes.

This is the same lens I use to decide which features even deserve evals, in Rank AI Features by Downside Exposure. Here I am pointing it one level down, at the rows of a single rubric. A rubric built this way has three or four dimensions that each hurt when they fail, instead of eight that mostly measure vibes. Evals are how you actually measure downside exposure in practice, so the rubric and the risk framework should share a spine.

3. Binary beats Likert

Score each dimension pass or fail. No 1-5 scale.

I used to reach for 1-5 rubrics, and you will see them in my own five-row eval template as one scorer option. I have moved off them for the human judgment dimensions, and here is why. A five-point scale hides disagreement inside the middle. When two reviewers land on a 3, they feel aligned and have agreed on nothing. The 3 absorbs every hard case. The scale gives you the comfort of a number and none of the sharpness of a decision.

Binary removes the hiding place. This output either fabricated a metric or it did not. It either buried the answer below the fold or it did not. When two people disagree on a binary, they have to argue, and the argument is where the rubric gets sharp, because it forces you to write the tie-breaking rule down. If you need a number for a threshold, aggregate: the pass rate across your cases is your score, and "passes the no-fabrication dimension on 100% of cases" is a far stronger bar than "averages 4.2 on faithfulness."

Keep exactly one escape hatch. A small set of critical dimensions (safety, fabrication, legal) are pass-fail on every single case, no aggregation, no averaging away one miss.

4. The rubric is a living artifact

A rubric is not a launch document, it is code. Version it, and treat specific events as review triggers.

Every model update is a trigger, because the failure modes shift under you. Every prompt change is a trigger. Every new class of bad output from production is a trigger, and it usually adds a dimension you did not know to look for. Put the rubric in the repo next to the eval, so the rationale for each dimension lives where the check fails and is read every time it fails.

This is the fast-cadence half of the dual-cadence measurement model I have written about. Outcomes move on a slow clock. The rubric moves on the fast one, every diff, because the agent iterates ten times a week and a rubric that only changes quarterly is measuring last quarter's model.

5. Ship the template

The last step is the one that actually spreads the practice. Do not describe the rubric, hand people the sheet.

I have attached the eval rubric template I use: the thirty-output grading sheet, the extraction prompt, and the binary dimension format with a cost column. Same play as shipping the PRD replacement in the five-row post. A template turns a method into something a reader can run on their own feature this afternoon, which is the only version of advice that changes what teams do. When the rubric is solid, encode it in the five-row eval template so it runs in CI, and connect it to the eval-first org that reads the scorecard every morning.

Why I am writing this at all

Because someone asked. The DM was a signal, not just a favor to return. If a question keeps landing in your inbox, that is a post you have not written yet, and the next time someone asks you send them your own link instead of someone else's. So if you are stuck on how to build a rubric for your feature, DM me. The good ones become the next post.

Sources: Hamel Husain on evals, Shreya Shankar on LLM evaluation, Anthropic: building evals.