People Paid More for AI-Designed Games. Make Your Design Beat the AI Version.

On Monday, Jakob Nielsen wrote up a study that I'd put on the agenda of every design critique this month.

John Conlon at Ohio State and Peter Schwardmann at Carnegie Mellon, in Creative destruction, had people improve simple black-and-white games by instructing an AI coding agent over three rounds, with playtesting between rounds. The AI got one standard prompt and no playtesting at all. Then consumers bid real money on the games, and the producer kept the bid.

The AI's games won. In the study Nielsen describes, they fetched $0.47 against $0.37 for the amateurs', a 27% edge. The 180 professionals beat the amateurs and still lost to the AI. Telling buyers a game was human-made bought no premium. And the humans didn't see it coming: amateurs expected to beat the AI by 9 cents, professionals by 17.

Nielsen's advice from it is the sentence this part is built on. Generate an AI-only version first, and ship your design only if it beats that version with real users.

That's a new job for research. Not the study that tells you what to build six weeks from now. The tiebreaker between generated versions, run at the speed they're generated.

This is part 6 of The Design Transition. Part 1 moved design's job from the screen to the spec. Part 5 gave design a number for after launch, the repair rate. This one is about before the ship: the evidence that decides which version goes out at all.

The short version

• Every surface now has a free competitor, the version an AI makes alone from one prompt, and design work has to beat it with real people before it ships.
• Conlon and Schwardmann found AI-only games outbid games from amateurs and professionals, with no premium for human-made and humans who were sure they'd win, so taste claimed in a critique doesn't settle anything.
• Simulated users can't settle it either: in the Rehearsim study, what researchers thought improved matched what they changed only 52% of the time, PickFu's Jasper Wolf says synthetic respondents can turn a real 60/40 split into 96/4, and Rachel Banawa at Nielsen Norman Group found they collapse onto a dozen answers where humans give more than 50.
• Real people are getting fast, though, with Salesforce closing Listen Labs and its 50 million participants on November 2.
• So the scarce work moves to the question and the verdict, design should own both, and the verdict has to land in the same queue the PM ships from, under the CPO.

The AI version is the baseline now

The caveats first, because they're real. The games were one-minute designs, and the professionals came from an online panel. Nielsen expects humans to keep the lead on larger projects for another year or two.

But look at what the study sets up. There's now a free default for every surface you design. One prompt, no designer, no playtesting, and in this market it beat people who iterated three times. The question in critique stops being "is this good?" It becomes "does this beat what we'd get for nothing, with the people who'll use it?"

That question can't be answered by opinion. The humans in the study were confidently wrong about their own work, and they only changed their minds once they read the AI's idea. Nielsen's line from the same roundup fits: "Data is the only monarch that abdicates when better data arrives."

I think this is good news for designers. It hands design a clean, falsifiable bar and a reason to be in the room early. Nielsen's own conclusion is to put the effort into deciding what to build and why. I'd add one thing: and into proving it, every time.

Simulated users rehearse. Real people vote.

The tempting shortcut at generation speed is to let simulated users judge the bake-off. This fortnight's evidence says don't, and it's specific about why.

Hangyue Zhang and colleagues at the University of Illinois built Rehearsim, which generates participants from published research and runs your interview guide against them. Nielsen reported on it on November 2. Twenty-one qualitative researchers tried it. Seventeen revised their protocols and they added 12 questions, which is a useful result. But their perceived improvements matched their documented revisions only 52% of the time, and one dismissed a surprising theme with "Oh, it's AI." Nielsen's summary: "AI is too polite in either chair."

Jasper Wolf at PickFu, launching a synthetic audience product on November 1, is candid about the same flaw. Synthetic respondents usually pick the same option real people do, but if real people split 60/40, they "might split 96/4." And Rachel Banawa at Nielsen Norman Group, comparing the two in October, found synthetic respondents collapsed onto about a dozen labels where 604 humans used more than 50.

Put those together and you get the problem. Simulation picks the same winner, louder, and loses the tail. For a baseline test that's fatal, because the whole point of putting the AI version in front of people is to find who it fails. The skeptic. The person with the odd workflow. The one who doesn't know what they want before they open the app.

So rehearse the instrument with simulated users. Let real ones vote.

Real people, at the speed of generation

The old excuse for skipping the vote was that sessions were slow. That excuse is going away.

Salesforce closed its acquisition of Listen Labs on November 2. Its agents design studies, recruit from a network of more than 50 million participants, and interview in more than 120 languages around the clock. Alfred Wahlforss, its CEO, put it simply: "With AI, a company can now have thousands of real conversations at once." Worth noticing that the same platform also sells digital twins, which I tried against my own users in The Twin Would Have Shipped It. Both are on the menu. For a ship decision, order the real ones.

When running sessions is cheap, two things stay scarce. The question, and the verdict. Both belong to design.

Rudrajit Choudhuri and colleagues at Oregon State and Microsoft, surveying 331 Microsoft developers, found that 12% of what AI unlocked was brand-new work, like writing standards for agents, and they name the risk that comes with it: the "hollow orchestrator," someone who approves output they can no longer evaluate. A research function that runs a thousand AI-moderated interviews and forwards the summary is exactly that.

What I'd build instead is a standing baseline test. For each generated decision that matters, the AI-only version and the designed one go in front of real people who differ from each other, within days. A named designer writes the question before and the verdict after. The verdict is one line: ship ours, ship the AI's, or neither, and why.

The verdict has to land where things ship

Here's where research stops being a design problem and becomes an org problem.

Part 2 of The PM Transition argued yesterday that the PM's day is a queue of decisions, proposals arriving already built. A baseline verdict is an input to exactly that queue. Product marketing needs the same evidence for its claim, which is why PM and PMM write it together now. The forward deployed engineer brings back the field's rows, and the FDE org belongs under the CPO for the same reason.

If research sits in its own org with its own roadmap, the verdict arrives the week after the ship. The generated version goes out by default, and nobody chose it.

That's design's piece of the one-function case. The test only works if the verdict and the ship decision are read against one outcome, by one loop, under one owner. The CPO Transition keeps arriving at the same seat for this reason, most recently when the field reported to product.

What I expect

These are my predictions, not facts.

By the end of 2027, at AI-native software companies, "beats the AI-only version with real users" will be a written ship criterion for design work on high-traffic surfaces, sitting in the design rubric next to the quality bar.

27%

By 2028, research capacity stops being planned in studies per quarter. Teams keep standing panels of real people who answer within a day, and the researcher's job is the question and the verdict, not the logistics.

Synthetic users will be everywhere for rehearsing guides and generating eval inputs. The teams that let them vote on ship decisions will be the ones surprised by their own repair rates.

One thing to try this week

Thursday is World Usability Day, and Nielsen's suggested way to mark it is to test with five users. Do it with one change.

Pick one design decision shipping this month. Generate the AI-only version with a single standard prompt, no designer, no iteration. Put both in front of five real users. Write down, in one line, what yours had to beat and whether it did.

If it didn't, ship the AI's and spend the hours on the next question. That's not a loss. That's the job.

Sources: Jakob Nielsen, "UX Roundup: Human 'Taste' Less Unique, AI Teammates, Evidence, and more," UX Tigers, November 9, 2026, for the study details, bids, and expectations as he reports them, his advice, and his write-up of Choudhuri and colleagues. John Conlon and Peter Schwardmann, "Creative destruction: Human-AI competition in an experimental market," arXiv, October 12, 2026. Jakob Nielsen, "UX Roundup: Practice Interviewing With AI, AI Answer Complexity, and more," UX Tigers, November 2, 2026, on Hangyue Zhang and colleagues' Rehearsim study (arXiv 2610.27333). Jasper Wolf, "Why we built Synthetic Audiences," PickFu, November 1, 2026. Rachel Banawa, "Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation," Nielsen Norman Group, October 16, 2026. Salesforce, "Salesforce Signs Definitive Agreement to Acquire Listen Labs," September 29, 2026, with an editor's note that the deal closed November 2, 2026, for the participant and language figures and Alfred Wahlforss's quote. Rudrajit Choudhuri and colleagues, "To Boldly Go Where No One Has Gone Before: What Has AI Let Developers Unlock?," arXiv, October 16, 2026, with the figures as Nielsen reports them.

Part of the running argument on AI Product Management: AI collapsed the cost of the parts product teams were trained to be good at, and for design that means making a good version got cheap while proving yours beats the free one, with real people, before it ships, became the job.

Related answer: Should a design have to beat the AI-only version before it ships?