Kill the Thumb. The Preset Reason Is the Rubric Dimension.

Jakob Nielsen's roundup this morning has a section on the thumbs under AI answers, and he opens it with the sentence most teams shipping those buttons would rather not read: the votes become training data.

He is right about that, and about most of what follows. The fix he lands on is the one I want to argue with, because it fixes the button and the button was never the problem.

The short version

In his October 23, 2026 roundup, Jakob Nielsen reports that thumbs data helped make OpenAI's 2025 GPT-4o update sycophantic, by OpenAI's own account, and summarises a study by Alisa Frik and co-authors at ICSI and eBay: 40 AI products reviewed, 8 with no feedback mechanism, 24 eBay experts on a heuristic evaluation of 12, 13 eBay users on a prototype, users preferring thumbs to five stars, unsure what a thumb rated, and liking about eight preset reasons. His recommendation is to usability-test the feedback feature. The widget was never the failure. The aggregate was. A thumb is one bit over a whole output, and the design handbook's rule for rubrics is pass or fail per dimension with a severity, because a five-point scale lets two reviewers write 3 for different reasons and believe they agreed. Nielsen's own observation that "accurate" and "liked the tone" are different kinds of praise is the dimension argument, and the eight presets are a rubric wearing a widget. Kill the thumb that lands in a table with no dimension, no output id, and no slice. The replacement is thumb plus preset as a pass or fail on a named dimension of a named output, joined to its slice, with the weekly number being where thumbs and rubric disagree per slice. The 200-row sliced eval that found a hole in a fifth of real traffic is why: an aggregate thumb would have averaged over that fifth exactly the way the demo did.

What Nielsen reports

The facts first, as he gives them.

Benchmarks measure what the labs prioritise and user feedback measures what users value, so the thumbs matter. They also mislead: OpenAI's account of the 2025 GPT-4o update is that thumbs data helped make it sycophantic, because a thumbs-up cannot distinguish accuracy from flattery. His caption makes the sharper point, that products usually ask why after a thumbs-down and rarely after a thumbs-up, so the model learns what to stop doing and not what to keep doing.

Then the study. Alisa Frik and her co-authors at the International Computer Science Institute and eBay reviewed the feedback features in 40 AI products and found 8 with no feedback mechanism at all. Twenty-four eBay experts ran a heuristic evaluation, using Nielsen's ten heuristics, on 12 of the products, and 13 eBay users tested a prototype. The findings are concrete. Buttons hidden behind hover or a menu are effectively invisible, so put the thumbs under the output. Users preferred thumbs to a five-star scale, one of them calling the scale "too many shades of gray." Users disagreed about what a thumb was rating, the text, the recommended products, or the whole feature, which Nielsen calls thumbiguity, so label the scope. Users liked a compact menu of about eight preset reasons in plain language, and the presets tell you why, since "accurate" and "liked the tone" are different kinds of praise. Keep the thank-you short.

His recommendation is the heading: the thumbs that train AI deserve usability testing. With the honest caveat that 13 users are enough to find problems and not enough to pick a winner between variants.

I have no quarrel with any finding in that list. The eBay team did what design teams should do and mostly do not, and Nielsen says so.

The widget was never the failure

Go back to the GPT-4o case. The button worked. People found it, understood it, pressed it. The model got worse anyway, because the thing that was optimised was the count.

A thumb is one bit over the whole output. That is the problem, and it does not go away when the bit is easier to press.

The Rubric Is the Spec has the rule I use for scoring generated work, and it runs the other way: score each dimension pass or fail with a severity attached, because a five-point scale lets two reviewers write 3 for different reasons and believe they agreed. The eBay users were right to reject the stars, and for that reason. But the thumb is the stars with two shades instead of five. It is the right shape and the wrong unit. Pass or fail is correct. Pass or fail on what is the question the thumb cannot answer.

Nielsen's own sentence answers it. "Accurate" and "liked the tone" are different kinds of praise. Those are dimensions. The eight preset reasons his users liked are a rubric that got shipped as a feedback widget and never joined to anything.

So this is the kill. Not the thumb as a gesture. The thumb that lands in a table with no dimension, no output id, and no slice. That is the only form in which a thumb can make a model sycophantic, because it is the only form in which flattery and accuracy share a column.

The instrument already exists

The rubric is not a new thing to build. How I Build Eval Rubrics is the method: grade 30 real outputs on gut feel with a notes column, extract the dimensions from the phrases that repeat in your own reasoning, keep only the ones whose failure costs something you can name, score each binary, calibrate with two people on five outputs before you call it real. Figma's Trial Closed 70% of the PM Gap. The Gate Is the Rubric, Not the Designer gives the sizes I use in place of a PRD: three to five criteria, at least 20 test cases, graded binary per dimension.

Now put the widget on top of it. Each preset reason maps to a dimension. A thumb plus "accurate" is a pass on the accuracy dimension of that one output. A thumb-down plus the matching negative reason is a fail on the same dimension, with the severity the rubric already assigns. The output id joins the vote to the row, and the row carries the slice the output came from.

That last join is the one that matters, and I have the number for why.

The Eval That Caught What the Demo Did Not is the launch where three hand-picked demos were flawless and the room wanted to ship. I ran the eval written before the build, the same feature against 200 real cases in named slices. On average it scored well. One slice, roughly a fifth of real traffic, failed badly. We fixed the slice and shipped a week later.

A thumbs-up rate across all traffic would have reported that feature as fine. Four slices passing and one failing averages to a number that looks like a pass, which is exactly what the demo did with three lucky draws. The aggregate thumb is a demo that runs continuously.

What you read on Monday

Once the join exists, the weekly number changes. Not thumbs-up rate. Disagreement.

Per slice, where do the thumbs and the rubric part company? A slice where users thumb up outputs the rubric fails on accuracy is the sycophancy slice, and you have found it before a training run does. A slice where the rubric passes and users thumb down is a dimension you have not extracted yet, and the preset reasons in that slice are where to look for it.

Nielsen's other section this morning, on putting the evidence beside the answer, is the same move for the reader. A claim with the supporting passage next to it is a scored dimension the user can check in place. His line for the alternative, a link to the first page of a long document, is "a treasure map without the X." The thumb with no dimension is the same map.

Under AI Product Management the argument is that AI collapsed the cost of the parts PMs were trained for and left the parts nobody trained for. Reading feedback was always one of the trained parts. Deciding what the feedback is allowed to mean, before anyone trains on it, is the other kind.

This week: pull the schema your thumbs land in. If the row has no dimension column, no output id, and no slice, you have the GPT-4o table. Add the three columns, map your preset reasons to the rubric's dimensions, and read the disagreement per slice on Monday instead of the rate.

Related answer: Should thumbs-up and thumbs-down feedback train your AI model?

Sources: Jakob Nielsen, "UX Roundup: Boomerang Browsing | Forecasting | Notifications | AI Attitude Gap | Signal Detection Theory | Supporting Evidence | User Feedback | Disabled Buttons | Embodied Interaction | Support Escalation | Blank Prompt," UX Tigers, October 23, 2026; the sections "The Thumbs That Train AI Deserve Usability Testing" and "Put the Evidence Beside the Answer" read in full. The study by Alisa Frik and co-authors at the International Computer Science Institute and eBay (arXiv 2610.02631) and OpenAI's account of the GPT-4o update are cited only as Nielsen reports them; neither was opened. Falk Gottlob, "The Rubric Is the Spec" (design handbook), "How I Build Eval Rubrics," "The Eval That Caught What the Demo Did Not" (August 12, 2026), and "Figma's Trial Closed 70% of the PM Gap. The Gate Is the Rubric, Not the Designer" (October 19, 2026).