Why is a shipped fix not enough?

Because shipping is what the team controls and landing happens on the customer's side. I shipped a launch that hit coverage, signups, and week-one activation and was flat two months later. A research finding has the same two halves: the fix leaving the building, and the comment stopping. Only the second one says the finding was right.

What does the check cost?

Nothing beyond keeping the codebook stable. Nielsen's rerun of the MeasuringU test found a current model codes comments with a self-agreement of 0.95, so the next quarter's comments can be coded against the same themes by the same run and the shares compared. If the fixed theme holds its share, it shipped and did not land.

Why read the bottom of the ranked list?

Because frequency is not severity. In my discovery week the ranker scored a theme that appeared in 31 calls above one that appeared in 3, and the 3 calls were our largest enterprise account describing an integration regression that put the renewal at risk. A consistent coder buries that theme the same way every run, so a human reads every theme with fewer than five comments before trusting the top.

Measuring Whether a Research Finding Landed

How do you measure whether a research finding landed, not just shipped?

THE SHORT ANSWER

Re-code the next batch of user comments against the same codebook and report which themes shrank. Jakob Nielsen's fixfluence counts how many findings became shipped design changes, and shipping is the half the team controls. Whether the comment stops arriving is the half it doesn't, and the same thematic analysis that found the theme can check it at no extra cost if the codebook stays stable between rounds. Two rules from my own agent discovery week: read the bottom of the ranked list before the top, because a frequency-weighted ranker put a 3-call theme under a 31-call one and the 3 calls were the renewal, and count a finding as landed only when its theme shrinks in the next batch.

Nielsen's fixfluence is the right move: grade research on whether the product got fixed rather than on whether the AI matched a human coder, since the humans never agreed with each other either. The verb is where I'd push. Shipped is the half the team controls.

The landed check is cheap because the instrument that found the theme can run it. Keep the codebook, re-code the next batch, report which themes shrank. The theme that didn't shrink is the next finding.

The 31-call and 3-call themes, the 11 and 9 theme counts, and the two rules are in The Three-Call Theme Was the Renewal. Count What Landed, Not Fixes. The week itself is Ship Story: The Discovery Week I Ran With Three Agents and No Calls.

Last reviewed 2026-10-15 · 1 min read

How do you measure whether a research finding landed, not just shipped?

SOURCES

THE LONG VERSION

RELATED ANSWERS