Five Real Users Beat Fifty Synthetic Ones, If You Ask Them How It Felt

Jakob Nielsen's roundup this week has a section on who to recruit, and one line from it belongs on every research brief: a product tested only on ideal users has been tested on nobody. He then puts synthetic users in the ideal-user category and calls their output fake data. I'd argue with the word fake, because a twin of my users would have had the facts right. What it could not have had is the column that got the feature pulled.

The short version

Nielsen's October 30 roundup says the ideal test participant is young, fluent, unimpaired, and upbeat, that the newest ideal participant is a synthetic persona that never tires or gets angry, and, reporting a 2026 paper by Yuanzi Li and co-authors, that aligned language models are too agreeable to stand in for users. Recruit the people who break designs instead: older adults, low tech confidence, screen readers and zoom, small phones, slow networks, a bad day. Five of them find more problems than 50 imagined users; synthetic users pilot the test plan and real users run the test. My onboarding agent found every real stuck point, and a twin built from that behavior would have had them all right and waved it through. Users read it as surveillance, activation sat below the control group, and it cost a quarter. The twin's data was true about where and empty about how it felt. The real users' behavior data was empty there too. What held the reaction was asking. The rewrite is two columns in the evidence log: where, which the twin may fill, and felt, which only a real person, asked, may fill.

What Nielsen said

The section is short and I'll keep the paraphrase close.

Ideal participants are convenient. Young, tech-fluent, unimpaired, upbeat. The newest ideal participant is a synthetic AI persona, and it never gets tired or angry. Nielsen reports a 2026 study by Yuanzi Li and co-authors that found aligned language models too agreeable to work as synthetic users, and his reaction is that this is no surprise, since real people aren't aligned with anything.

So recruit for the traits and circumstances that break designs. Older age. Low tech confidence. Screen readers and zoom. Small phones. Slow networks. A bad day. Five real users like that will uncover more problems than 50 imagined users, because imagined users share the designer's blind spots. Synthetic users are for piloting your test plan. Real users are for the actual test. And when real customers are reachable, ask them.

I run research the way the design handbook's Research at Generation Speed(coming Dec 16) describes it, a weekly cadence with synthetic users in a narrow role, testing the instruments. So the sequence is the same as his. The disagreement is one word and one missing step.

The twin would have had the facts

The Twin Would Have Shipped It is the long version of this story. The short one.

I shipped an onboarding walkthrough agent. It watched new users in their first session, noticed where they got stuck, and offered contextual help at exactly those moments. The stuck points were real. Technically, the agent was correct.

Now build a twin of those users from their behavior data, the way the digital-twin products promise. It would have had the same stuck points. Every one. It is hard to see how a simulation grounded in that behavior objects to help arriving at the moment the behavior says help is needed. The twin would have waved it through.

Users hated it. They read it as surveillance. Activation among the users who got the interventions was lower than the control group, and the feature cost a quarter of decreased activation plus the engineering to build it and then take it out.

That data was not fake. The twin's where column would have been accurate to the click. What the twin could not produce was a reaction to a feature that did not exist yet, from people who were not grounded in a world where it did. And Li's finding, as Nielsen reports it, is the mechanism. An aligned model asked how it feels about help arriving says help is nice. A too-agreeable user is exactly the one who never says "this feels like being watched."

The meeting summary agent, number nine in the same retrospective, failed from the other side: accurate summaries, and people stopped raising hard topics once it was in the room. Same shape. Correct about the what. Blind to the felt.

The step the list leaves out

Here is the part I'd add to Nielsen's recruiting list, and it's the reason I'm writing this instead of just forwarding him.

The behavior data of the real users did not hold the reaction either.

Those were real people, with bad days and small phones, in their actual first sessions. Their stuck points matched what the twin would have predicted. Their clicks after the help arrived did not say "surveillance." The activation gap showed up in the aggregate weeks later, and the word surveillance showed up only when we asked. Prototype the intervention with a small group, ask how it felt, then scale. That's the rule I took out of it.

So five real users beat fifty synthetic ones only if the session has a question in it. Five real users on a camera find the stuck points, which the twin already found. Five real users asked how the help felt find the thing that pulled the feature. The recruiting list gets you the right people in the room. It doesn't say what to do with them, and watching is not enough.

The rewrite

"Synthetic users pilot the test plan, real users run the test" becomes two columns in the evidence log.

Where. Stuck points, drop-offs, the edge case, the flow that breaks for a segment you can't recruit this week. The twin may fill this column. So may analytics. So may a usability session you only watched.

Felt. How the help landed. Whether the summary changed what people said in the room. Whether the shortcut read as a shortcut or as a nudge. Only a real person, asked, may fill this column. Not inferred from clicks. Asked.

A decision needs both columns. A decision with where full and felt blank is how a technically correct feature ships and gets pulled a quarter later, and I have two of those in one retrospective.

The handbook chapter Evidence as an Asset(coming Dec 9) already argues for one register, one row per claim, with the strength and the source written down. This is one more field on that row: which column the row fills, and whether a person was asked.

It belongs to the argument on AI Product Management: AI made the where column nearly free, and left the felt column exactly as expensive as it was.

This week: open whatever holds the evidence behind your next launch. For each row, mark it where or felt. Then look at the rows marked felt and check whether a person was asked, or whether somebody inferred it from a chart. If the felt column is empty and the launch is a feature that watches users in order to help them, that's the five sessions to book.

Related answer: Can synthetic users replace real user testing?

Sources: Jakob Nielsen, "UX Roundup: AI Persuasion | Fake Users vs. Real Users | Summaries as Dark Design | Agentic Video Production | Feedback | High AI Subscription Churn | Sparring with Voice AI | Silence | Split View," Jakob Nielsen on UX, October 30, 2026, the section "Recruit the Users You Will Actually Get," read in full; the Yuanzi Li paper (arXiv 2608.24912) was not opened and is cited only as he reports it. Falk Gottlob, "The Twin Would Have Shipped It" (September 30, 2026) and "10 AI Agents I Built That Failed. The Honest Retrospective." (May 4, 2026).