The Three-Call Theme Was the Renewal. Count What Landed, Not Fixes

Jakob Nielsen's roundup last Thursday carried a section that research leads will be forwarding for months. MeasuringU reran its 2023 coding test with a current model, and the model now agrees with UX researchers about as well as they agree with each other. Nielsen's conclusion is the right one: the humans were never the answer key, so stop grading the AI against them and grade it on whether the product got fixed.

Then he gives guidelines, and two of them are the ones I'd rewrite. I have the week that rewrote them for me.

The short version

Jakob Nielsen reports that MeasuringU scored GPT-5.6 against three UX researchers' codings of 154 open-ended website comments, with prompt and codings unchanged from 2023. The AI agreed with the humans as well as they agreed with each other (kappa 0.75), agreed with itself at 0.95, and split finer. He moves the test to what he calls fixfluence, whether findings become shipped design changes, and advises reading every comment in the top three themes before presenting. In my agent discovery week, the opportunity ranker scored on frequency, so a theme in 31 calls beat a theme in 3 calls, and the 3 were our largest enterprise account describing an integration regression that put the renewal at risk. Two edits follow. Read the bottom of the ranked list before you trust the top, because a consistent coder buries the rare theme the same way every run. And count landed findings, not shipped fixes: re-code next quarter's comments against the same codebook and report which themes shrank.

What the rerun found

Nielsen's account is worth reading whole. The parts that matter here are short.

In May 2023 MeasuringU pitted three ChatGPT-4 runs against three UX researchers on 154 comments about frustrations with the Office Depot, AT&T, and Food Lion websites. Will Schiavone and colleagues have now rerun it with GPT-5.6, keeping the comments, the prompt, and the human codings unchanged. The model agreed with the humans as well as the humans agreed with each other, possibly slightly better, with the difference not significant. Each human coder agreed more with every GPT-5.6 run than with any human colleague. The kappa of 0.75 counts as substantial, and the AI's own runs sit at 0.95. It also slices finer: 12 substantive themes per dataset on average against 7 for the humans.

The sharper point is the one about the yardstick. For the same 50 grocery comments, one human made 25 themes and another made 7, and nobody can say which is right, because no correct grouping exists. Grading AI against human coders, in his phrase, is grading a student against a classmate's homework.

So he moves the exam. Level 1 is producing the codes. Level 3, for research, is a product team that understands the finding, believes it, and fixes the design. He calls it fixfluence, and his evidence that AI may already win there is a coder who labeled a theme "Design," which tells a PM nothing, against the model's "Unappealing, bland, or outdated visual design." Finer splits route problems to their owners. An orphaned theme gets no fix.

Then the guidelines. One run is enough at 0.95 self-agreement. Ask for a two-level codebook whose subthemes name the owner. Read every comment in the top three themes before you present them. Track fixfluence: count how many findings become shipped design changes.

Two of those four I'd change.

The three-call theme

In May I ran a discovery week with three agents and no live calls, and wrote it up in Ship Story: The Discovery Week I Ran With Three Agents and No Calls. Six weeks of Gong calls, three months of support tickets, a sales-notes export. Three agents: a transcript triager, a theme synthesizer, an opportunity ranker. About 20 hours of human time across four days, against the 25 to 35 hours of call execution a live-call week costs before anyone synthesizes anything.

The synthesizer did what Nielsen's rerun says it should. Its second run produced eleven themes, and a senior PM reading them blind agreed that nine were real patterns rather than hallucinated categories. On the first run, four of the themes were ones I could not defend. So, roughly, one iteration to get to a coder I'd trust.

The ranker is where the week turned.

Its confidence score ran on frequency. A theme that showed up in 31 calls beat a theme that showed up in three. The three calls were our largest enterprise account talking about an integration regression that was about to put their renewal at risk.

The agent was technically correct and strategically wrong. Low frequency, low confidence, bottom of the list. A human had to pull that theme up the stack, and the only reason a human saw it is that I read the bottom of the ranker's output before I trusted the top. That has been the rule ever since. The weighting layer I promised in that post, a frequency multiplier for tier-one accounts, I still haven't built. The reading rule has done the job.

Here's why Nielsen's rerun makes this worse rather than better. He notes himself that a wrong AI analysis is convincingly wrong and, at 0.95 consistency, wrong the same way every time. A frequency-weighted coder that buries a three-comment theme buries it in every run. A second run doesn't surface it. A finer codebook doesn't surface it either, it just gives the three comments a more precisely named subtheme at the bottom of the list. The top-three rule reads the themes that were never at risk of being missed.

So the first edit: read every comment in the top three themes, and then read every theme with fewer than five comments. The rare theme is where frequency-coded synthesis fails, and it fails quietly.

Shipped is not landed

The second edit is on the exam itself.

"Count how many findings become shipped design changes" is a real improvement over counting agreement. It's also shipping, and shipping is the thing I've spent a year arguing is not the end. The Launch That Worked and Did Not Land is the case: coverage, signups, and week-one activation all green, a launch team moved on to the next thing, and a usage curve that was flat two months later. The launch measured what I could make happen and stayed quiet on the one thing I couldn't.

A research finding has the same two halves. The fix shipping is the half the team controls. Whether the comment stops arriving is the half it doesn't.

And here the thematic analysis has an advantage a launch dashboard never had. The instrument that found the theme can check whether the fix landed, at no extra cost. Keep the codebook stable, re-code the next quarter's comments against it, and report which themes shrank. If "unappealing, bland, or outdated visual design" holds the same share of comments after the redesign as before it, the fix shipped and did not land, and that's a finding too.

That's the rewrite for fixfluence. Not how many findings became shipped changes. How many themes stopped arriving.

Where this sits

The design handbook's Research at Generation Speed argues that research didn't get less valuable when generation got cheap, it got harder to deliver on time, and that the fix is a weekly cadence with one question and one thing written down. Nielsen's rerun is the evidence that the coding step of that cadence can run on a model now. This post is about the two places the human still has to read: the bottom of the ranked list, and the next batch.

It belongs to the AI Product Management argument that discovery is becoming a synthesis problem rather than a signal problem, and that the PM's job moves from collecting the comments to deciding which rare one is the renewal.

This week: take the last ranked synthesis your team produced, by a person or a model, and read the themes at the bottom with fewer than five comments. Then pick one theme from the top that you fixed last quarter and count it in this quarter's comments. If it didn't shrink, you have your next finding.

Related answer: How do you measure whether a research finding landed, not just shipped?

Sources: Jakob Nielsen, "UX Roundup: Erik the Red | AI Thematic Analysis | Nano Banana 2.1 | Scrolling | Fast Asymptote in AI Skills | Average Users | Wishlists | Usability Testing Without Findings | Naming Consistency," UX Tigers, October 9, 2026, the section "AI Now Agrees With UX Researchers as Much as They Agree With Each Other," read in full; the MeasuringU rerun by Will Schiavone and colleagues is cited only as Nielsen reports it. Ship Story: The Discovery Week I Ran With Three Agents and No Calls, falkster.com, for the 31-call and 3-call themes, the 11 and 9 theme counts, and the 20-hour figure. The Launch That Worked and Did Not Land, falkster.com, for the launch that was flat at month two.