How do you run a design rubric review in thirty minutes?

Answered by Falk Gottlob · Reviewed · 4 min read

THE SHORT ANSWER

Score before the meeting, never in it. Two people score the same ten real outputs independently against the current rubric version, a facilitator tabulates and marks disagreements without resolving them, and the surface owner attends to listen rather than defend. The thirty minutes go five on the aggregate, fifteen on disagreements only, seven on the decision, and three on the single change coming out of it. Every disagreement is classified as the rubric being vague, the rubric being wrong, or a real quality dispute, and each classification has a different action.

A rubric review is a different meeting from a design critique, and the difference is the output. A critique produces ideas and ends when everyone feels heard, which is a legitimate thing for a meeting to do. A review produces three things: scores, a classified list of disagreements, and one decision. Ship, fix, or change the rubric.

If a review ends without a decision, it was a critique wearing a rubric's clothes.

Before the meeting, about an hour total

The facilitator pulls ten real outputs from the last week. The sample includes at least two the team would rather not look at, selected by a rule announced in advance, such as lowest confidence, most edited, or most reversed. State the rule so nobody can be accused of stacking the set.

Two scorers score all ten independently against the current rubric version. Fifteen minutes each if the rubric is any good. If it takes more than twenty, the rubric is too long and you cut a dimension before the next round.

The facilitator tabulates and marks disagreements, and does not resolve them.

The roles

Two scorers who did not make the work. One facilitator who runs the clock, reads scores, and classifies disagreements, and who does not score. And the owner of the surface, present, listening, not defending. That last part is unnatural and worth saying out loud for the first few sessions.

Three people minimum, five maximum before it becomes a presentation.

The thirty minutes

Minutes 0 to 5, the tape. The facilitator reads the aggregate: pass rate per dimension, change since last week, blockers found. No discussion. Everyone starts from the same numbers instead of their own impressions.

Minutes 5 to 20, disagreements only. Skip everything both scorers agreed on, because agreement is not interesting and it eats the clock. For each disagreement the facilitator asks one question, which of the three is this.

The rubric is vague. Both readings are defensible. Rewrite the test this week, by a named person, not in the room, because wording produced live is wording nobody has slept on.

The rubric is wrong. Both scorers read it the same way and both feel the result is wrong. Flag the dimension for replacement, and do not let it block a release in the meantime.

Real disagreement about quality. Rare. The design lead makes the call in the room and it goes into the decision record. Do not defer this to a follow-up thread. This is the moment the practice's taste gets set.

Two minutes per disagreement, hard. A disagreement that needs longer is the third category and it needs a decision, not more discussion.

Minutes 20 to 27, the decision. Ship, fix, or change the rubric, said out loud, with an owner and a date. Any blocker in the sample means do not ship, and that cannot be negotiated in the room or it was never a rule. Majors are counted against a threshold set in advance rather than debated. Minors are logged and the rate watched.

Minutes 27 to 30, the one change. One. A protocol that produces four actions produces zero.

Cadence, and the two ways it dies

Weekly, one surface, rotating, same three people. Monthly does not work for generated products, because model behavior, prompts, and retrieval drift faster than a month, so a monthly review finds problems that already shipped for four weeks.

It dies when it becomes a critique, which starts with someone redesigning in the room. The facilitator's line is that it is a good idea and it is not this meeting.

It also dies when scores start attaching to people. The moment a score reads as a judgment of whoever made the thing, honest scoring stops. Score outputs, never people, and never carry these numbers into a performance conversation. Do that once and the system does not recover.

Run it once badly this week: ten outputs, two independent scorers, fifteen minutes on the disagreements. The first run mostly teaches you your rubric is vaguer than you thought. Full agenda and the disagreement rate table are in The Review Protocol, and the argument is in Scoring Your Own Work.

SOURCES

THE LONG VERSION

RELATED ANSWERS