The one-line version: the rubric comes last, not first. Building an eval rubric is not a writing task, it is an extraction task.
1. Grade thirty outputs before you write a single criterion
Every team that starts by writing criteria produces the same thing: an academic rubric, six tidy dimensions, a 1-5 scale, and a document nobody opens again after launch. It looks rigorous. It is theater. You wrote down what you imagined good would look like before you had looked at a single real output.
Invert it. Pull about thirty real outputs from the feature, or from a prototype of it. Open a two-column sheet: the output, and a notes column. Grade each one on gut feel alone, with a sentence on why. No rubric yet. You are allowed to be inconsistent, because the inconsistency is the data.
By output thirty, two things have happened. Your notes have started repeating the same words ("invented a number," "buried the answer," "hedged when it should have committed"). And you have caught yourself grading two similar outputs differently. Those repeated words are your dimensions. Those disagreements with yourself are where the rubric has to get precise. The rubric is what you extract from the notes column, not what you paste in above it.
2. Tie every dimension to a cost
Most candidate dimensions do not survive one question: what does failing this cost me? A dimension earns its place only if failing it maps to a real cost: user harm, trust erosion, rework, a support ticket, a lost renewal. "Tone feels off" is not a dimension until you can name what the wrong tone costs. This is downside exposure pointed one level down, at the rows of a single rubric. A rubric built this way has three or four dimensions that each hurt when they fail, instead of eight that measure vibes.
3. Score binary, not 1-5
Score each dimension pass or fail. A five-point scale hides disagreement inside the middle: when two reviewers land on a 3, they feel aligned and have agreed on nothing. Binary removes the hiding place. This output either fabricated a metric or it did not. When two people disagree on a binary, they have to argue, and the argument is where the rubric gets sharp, because it forces you to write the tie-breaking rule down. If you need a number for a threshold, aggregate: the pass rate across your cases is your score. "Passes the no-fabrication dimension on 100% of cases" is a far stronger bar than "averages 4.2 on faithfulness."
Keep exactly one escape hatch. A small set of critical dimensions (safety, fabrication, legal) are pass-fail on every single case, no averaging away a miss.
4. Version it like code
A rubric is not a launch document, it is code. Every model update is a review trigger, because the failure modes shift under you. Every prompt change is a trigger. Every new class of bad output from production is a trigger, and it usually adds a dimension you did not know to look for. Put the rubric in the repo next to the eval, so the rationale for each dimension lives where the check fails.
5. Ship the template
Do not describe the rubric, hand people the sheet: the thirty-output grading sheet, the extraction prompt, and the binary dimension format with a cost column. A template turns a method into something a reader can run on their own feature this afternoon, which is the only version of advice that changes what teams do. When the rubric is solid, encode it in a runnable five-row eval template so it runs in CI on every diff.