How do you make the AI-only version fairly?

One standard prompt, no designer, no iteration. In the Conlon and Schwardmann study the AI got a single brief asking it to make each game more creative, engaging, and fun, and no playtesting, while humans got three rounds with playtesting. The baseline is meant to be what you'd get for nothing.

How many users does the test need?

Start with five real users seeing both versions, which is Jakob Nielsen's standing suggestion for a quick test. For high-traffic surfaces, a standing panel that answers within days is the better habit. What matters is that they're real people who differ from each other.

Can synthetic users run the comparison instead?

No. In the Rehearsim study by Hangyue Zhang and colleagues at the University of Illinois, researchers' perceived improvements matched their documented revisions only 52% of the time. PickFu's Jasper Wolf says a real 60/40 split can come out 96/4 with synthetic respondents, and Rachel Banawa at Nielsen Norman Group found they collapse onto a dozen labels where humans used more than 50. Use them to rehearse the guide.

What if the AI-only version wins?

Ship it, and spend the design hours on the next question. The study's humans were confident they'd win and were wrong, so a loss is information, not a verdict on the designer.

Who should own the verdict?

A named designer writes the question before the test and a one-line verdict after: ship ours, ship the AI's, or neither, and why. The verdict goes straight into the queue the PM ships from, which is why design, research, and product sit in one loop under the CPO.

Should Your Design Beat the AI-Only Version?

Should a design have to beat the AI-only version before it ships?

Answered by Falk Gottlob · Reviewed Nov 10, 2026 · 1 min read

THE SHORT ANSWER

Yes, for any decision that matters. Generate the version an AI makes alone from one standard prompt, put it next to the designed version in front of real users, and ship the designed one only if it wins. That's Jakob Nielsen's advice from a study by John Conlon and Peter Schwardmann, where games an AI designed alone fetched $0.47 against $0.37 for amateurs' games, a 27% edge, beat professionals too, and earned no penalty for being AI-made. Use simulated users to rehearse the test, never to decide it: they pick the same winner louder and lose the people the AI version fails.

Yes. Every surface now has a free competitor, the version an AI makes from one prompt with no designer, and the designed version should beat it with real people before it ships.

The evidence. John Conlon and Peter Schwardmann had people improve games with an AI coding agent over three rounds while the AI worked alone from one prompt. Buyers paid more for the AI's games, $0.47 against $0.37 for amateurs in the study Jakob Nielsen describes, professionals lost too, and a human-made label added nothing.

Who votes. Real people. Simulated users pick the same winner louder and lose the tail, and the tail is who the AI version fails. Real sessions are getting fast: Salesforce closed Listen Labs, with more than 50 million participants, on November 2, 2026.

Who decides. A named designer, with the verdict landing in the same queue the PM ships from.

The full argument is part 6 of The Design Transition, the series runs in order on the hub, and The Product OS shows how design, product, and research fit one loop under the CPO.

Should a design have to beat the AI-only version before it ships?

SOURCES

THE LONG VERSION

RELATED ANSWERS