Scoring Your Own Work
A rubric review is a different meeting from a critique. Scores, disagreements classified, and one decision: ship, fix, or change the rubric.
The short version
- A design rubric review takes thirty minutes, three people, and no scoring in the room. Two scorers score ten real outputs independently beforehand, a facilitator tabulates and marks the disagreements without resolving them, and the surface owner sits in to listen rather than defend. The meeting skips everything the scorers agreed on, because agreement is not interesting and it eats the clock, and spends its time classifying disagreements into three types: the rubric is vague, the rubric is wrong, or the quality call is really contested. It ends with one decision and one change, each with a name and a date. Weekly on one surface beats monthly on everything.
- You wrote the rubric. Now somebody has to use it, on real work, on a Tuesday, without you standing next to them explaining what you meant.
- That is a different meeting from the one design teams already have. A critique produces ideas and ends when everyone feels heard. A review produces three things: scores, a classified list of disagreements, and one decision. If a review ends without a decision, it was a critique wearing a rubric's clothes, and you should notice that rather than count it.
Scoring happens before the meeting
This is the rule people break first and it wrecks everything downstream.
Two scorers, working independently, before anyone gets in a room. Ten real outputs from the last week, against the current rubric version. Fifteen minutes each if the rubric is any good, and if it takes more than twenty, the rubric is too long and you should cut a dimension before the next round.
Scoring in the room produces agreement that is really deference to whoever spoke first. It always looks like alignment. What it records is the seniority gradient in the room, which you already knew.
Sampling is the facilitator's job, and so is tabulating. The sample has to include at least two outputs the team would rather not look at, chosen by a rule stated in advance. Lowest confidence, most edited, most reversed. State the rule so nobody can be accused of stacking the set, and so nobody has to decide, output by output, whether to include the embarrassing one.
Three people minimum. Five is the maximum before it turns into a presentation.
Two scorers who did not make the work. A facilitator who runs the clock, reads scores, and classifies disagreements, and who does not score, because doing both means doing neither for thirty minutes. And the owner of the surface, present, listening, not defending.
That last role is the one that needs saying out loud at the start of the first few sessions. The owner's job in this meeting is to hear the scores. It is unnatural, the instinct to explain each output is strong, and removing that instinct is most of what the protocol is for. If the owner is also a scorer, you have lost the independence that made the scores worth collecting in the first place.
The thirty minutes
Five minutes on the tape. The facilitator reads the aggregate: pass rate per dimension, change since last week, blockers found. No discussion. Everyone starts from the same numbers instead of from their own impressions, which is a surprisingly large fraction of the value.
Fifteen minutes on disagreements only. Skip the agreements entirely. For each disagreement the facilitator asks one question, which of the three is this, and the answer determines the action.
If the rubric is vague, the test gets rewritten this week, by a named person, not in the room. Wording produced live in a meeting is wording nobody has slept on.
If the rubric is wrong, meaning both scorers read it the same way and both feel the result is wrong, the dimension gets flagged for replacement, and it does not block a release in the meantime.
If it is a real disagreement about quality, the design lead makes the call, now, in the room, and it goes into the decision record. Do not defer this one to a follow-up thread. Follow-up threads are where taste goes to die, and this is the exact moment where the practice's taste gets set.
Two minutes per disagreement, hard stop. A disagreement that needs longer is a real quality disagreement and it needs a decision, not more discussion.
Seven minutes on the decision. Ship, fix, or change the rubric, said out loud, with an owner and a date. Any blocker in the sample means do not ship, and that rule cannot be negotiated in the room or it is not a rule. Majors get counted against a threshold you set in advance rather than debated. Minors get logged and the rate watched.
Three minutes on the one change. One. The highest-value change to the rubric or the product coming out of this review, with a named owner, done before the next one.
A protocol that produces four actions produces zero. I have watched that happen enough times to make it a rule rather than a preference.
The disagreement rate is the health metric
Track it. It tells you more about the system than the scores do.
Near zero is the reading people misread as success. It almost always means the scorers talked before scoring, or the dimensions are so obvious that nobody needed a rubric to catch them. Pull a harder sample, or check that the independence held.
Low but present is healthy, and needs nothing.
Around a third means one dimension is vague. Find it, rewrite the test.
Above half means the rubric is a document rather than shared understanding. Stop reviewing and recalibrate from scratch: five outputs, scored together, out loud.
Cadence, and the two ways this dies
Weekly, one surface, rotating, same three people, same slot. Monthly does not work for generated products. Model behavior, prompts, and retrieval drift faster than a month, so a monthly review finds problems that already shipped for four weeks. Same argument as Continuous Discovery on Autopilot, applied to quality instead of to learning.
Both failure modes are quiet.
It becomes a critique. Someone starts redesigning in the room, the clock goes, and the decision never happens. The facilitator's line is that it is a good idea and it is not this meeting.
Or scores start attaching to people. The moment a score reads as a judgment of whoever made the thing, honest scoring stops. Score outputs, never people, and never carry these numbers into a performance conversation. Do that once and the system does not recover, because everyone remembers.
This week
Run it once, badly.
Ten outputs, two people scoring independently, fifteen minutes on the disagreements. The first run mostly teaches you that your rubric is vaguer than you thought, which is the finding worth having and the reason not to spend three weeks perfecting the rubric first.
The full protocol, with the sample rule, the three roles, the minute-by-minute agenda, and the disagreement rate table, is in The Review Protocol. If you need the compressed version for someone who has to run it on Thursday, that is How do you run a design rubric review in thirty minutes? The rubric itself, and how to extract one from your own grading, is The Rubric Is the Spec.
Chapter 13 of a series on the design operating model. Next: the ten worst plausible outputs of a feature, listed before anybody designs the good one.
Take the template
Review protocol, thirty minutes
Frequently asked
How do you run a design rubric review?+
Thirty minutes, three people, and the scoring happens before the meeting rather than in it. Two scorers score ten real outputs independently against the current rubric version, a facilitator tabulates and marks disagreements without resolving them, and the surface owner attends to listen. The meeting itself skips everything the scorers agreed on and spends its time only on the disagreements. It ends with one decision and one change, both with named owners.
Who should score design work in a rubric review?+
Two people who are not the owner of the surface. The owner attending as a scorer destroys the independence that makes the scores worth collecting, because nobody scores their own work honestly under observation. The facilitator does not score either, since running the clock and classifying disagreements is a real job for thirty minutes and doing both means doing neither. Three people minimum, five is the sensible maximum.
What does disagreement in a design review actually mean?+
One of three things, and naming which is the whole method. The rubric is vague, meaning both readings are defensible and the test needs rewriting. The rubric is wrong, meaning both scorers agree on the reading and both feel the result is wrong, so the dimension needs replacing. Or there is real disagreement about quality, which is rare, and the design lead makes the call in the room and it gets written down.
What is a healthy disagreement rate between design reviewers?+
Low but present. Near zero usually means the scorers talked before scoring or the dimensions are so obvious they were never worth measuring, which is the failure most teams miss because it looks like success. Around a third means the rubric is vague on at least one dimension, so find it and rewrite the test. Above half means the rubric is a document rather than shared understanding, and you recalibrate from scratch on five outputs together.
How often should you run a design rubric review?+
Weekly, on one rotating surface, with the same three people. Monthly reviews do not work for generated products because model behavior, prompts, and retrieval drift faster than a month, so by the time a monthly review finds something, four weeks of output already shipped on the old assumption. Thirty minutes weekly beats two hours monthly for the same reason continuous discovery beats quarterly research.
How is a rubric review different from a design critique?+
A critique produces ideas and a rubric review produces a decision: ship, fix, or change the rubric. A critique can end with everyone feeling good and nothing recorded, and that is an acceptable outcome for a critique. A review that ends without a decision was a critique wearing a rubric's clothes. Both meetings are worth having, and they should not be the same thirty minutes.
Related reading
Chapters and essays on the same thread, across both handbooks.
Copilot Adopters Added 1.4 Hours of Collaboration a Month. That's Critique. Plan It.
Copilot adopters added 1.4 hours of collaboration a month, much of it vetting AI output. That's critique with no format. Plan it, split it, and log it.
The Rubric Is the Spec
Your taste lives in a recurring meeting, which caps quality at what you can personally review. A rubric is that taste in a form that runs without you.
Figma Says Design Is the Differentiator. Design Jobs Hit a Record Low. The Bar Is the CPO's Now.
Figma calls design the differentiator while open design roles hit a record low. The design work left the design org, so the CPO decides where the bar lives.
Reddit's Handoff Ships a Draft PR. Design Should Hand Engineering the Tests.
Reddit's feature capsule hands engineering code that may or may not merge. When prototypes are production-shaped, design hands engineering test rows, not code.
Figma's Trial Closed 70% of the PM Gap. The Gate Is the Rubric, Not the Designer.
Figma's randomized trial put PMs 35% faster with Figma Make. Nielsen says make designers the quality gate. A person caps throughput. The rubric is the gate.