Not in the form most products collect it. As Jakob Nielsen reports OpenAI's own account (October 23, 2026), thumbs data helped make the 2025 GPT-4o update sycophantic, because a thumbs-up cannot tell accuracy from flattery. The failure is the aggregate, not the button: a thumb is one bit over a whole output, and my rubric rule is pass or fail per dimension with a severity, because a five-point scale lets two reviewers write 3 for different reasons and believe they agreed. Map the preset reasons users already like (Nielsen reports about eight) to rubric dimensions, join each vote to the output id and the slice, and read where thumbs and rubric disagree per slice. My 200-row sliced eval found a failure in roughly a fifth of real traffic that three flawless demos hid, and an aggregate thumbs rate would have averaged over it the same way.
Not in the form most products collect it. A thumb is one bit over a whole output, and the aggregate is what made GPT-4o sycophantic.
The kill, the join to the rubric, and the Monday number are in Kill the Thumb. The Preset Reason Is the Rubric Dimension. The extraction method is How I Build Eval Rubrics. The fifth-of-traffic slice an average would have hidden is in The Eval That Caught What the Demo Did Not.
Last reviewed 2026-10-23 · 1 min read