The Eval Rubric Template

A rubric is not something you write, it is something you extract. This template walks the extraction: grade thirty real outputs on gut feel, pull the dimensions out of your own notes, keep only the ones that cost you something when they fail, and score them binary. Copy the three blocks below into a sheet and work top to bottom. The first pass takes an afternoon.

---

Block 1: The grading sheet (fill this first, before you have any rubric)

Pull 30 real outputs from the feature or a prototype of it. Grade each on gut feel alone. No criteria yet. Being inconsistent is allowed, the inconsistency is the data.

#	Output (or link)	Gut call (good / mixed / bad)	Why, in one sentence
1
2
...
30

Two things to watch for as you go, because they are the whole point:

• Words that repeat in your "why" column (for example "invented a number", "buried the answer", "hedged"). Those are your candidate dimensions.
• Cases where you graded two similar outputs differently. Those disagreements with yourself are exactly where the rubric has to get precise.

---

Block 2: Extract the dimensions (do this after all 30 are graded)

For each repeated phrase in your notes, write one candidate dimension, then apply the downside test. A dimension stays only if failing it maps to a real cost.

Candidate dimension	What failing it costs (harm / trust / rework / $)	Keep?
e.g. Fabricates a metric	User acts on a false number, trust gone	KEEP
e.g. Buries the answer	Extra read time, mild annoyance	maybe
e.g. Tone feels off	Can't name a cost	CUT

Rule: if you cannot name the cost in the middle column, cut the dimension. Aim for three or four dimensions that each hurt, not eight that measure vibes. If you want a prompt to help, paste your 30 graded rows into a model with:

"Here are 30 outputs I graded by gut feel with a one-line reason each. Cluster my reasons into 3 to 5 named failure dimensions. For each, quote the rows that show it, and name the real-world cost of failing it. Flag any dimension where the cost is vague so I can cut it."

---

Block 3: The rubric (binary, one row per surviving dimension)

Score pass/fail per dimension per case. No 1-5 scale. If you need a number, use the pass rate across cases as the score.

Dimension	Pass means	Fail means	Cost when it fails	Critical?
No fabrication	Every number/claim traces to the input	Any invented figure	User acts on false data	YES: 100% every case
Answers the question	The ask is addressed in the first line	Answer buried or absent	Rework, re-prompt	pass rate >= 90%
Commits when it should	States a call on high-signal input	Hedges to avoid being wrong	Useless output	pass rate >= 90%
Fits the schema	Validates against the brief schema	Any schema break	Breaks downstream step	YES: 100% every case

• Critical dimensions (safety, fabrication, legal, schema) are pass/fail on every single case. No aggregation, no averaging one miss away.
• Non-critical dimensions use an aggregate pass-rate threshold you can defend.
• Tie-breaks: the first time two graders disagree on a binary, write the deciding rule into the "Pass means" cell. That argument is the rubric getting sharp.

---

Keeping it alive

Put this rubric in the repo next to the eval. Review it on every trigger:

• a model update (failure modes shift under you),
• a prompt change,
• a new class of bad output from production (usually adds a dimension).

When the rubric is stable, encode it as runnable rows using the five-row eval template (Behavior, Input, Expected, Scorer, Threshold) so it runs in CI on every diff.

Companion to the post "How I Build Eval Rubrics" on falkster.com.