84,000 Chats Show When Users Re-Ask the AI. Design Should Own the Repair Rate.

Your users already tell you every time an AI answer misses. They ask again.

Yingcan Carol Wang, Iman Munire Bilal, and Qamar Zaman at Stripe Partners, a London research consultancy that has nothing to do with the payments company, went and counted. Their paper, When AI Becomes Hard to Understand, looks at more than 84,000 real ChatGPT and Gemini conversations about money and health. They tracked two behaviors: people repeating or rephrasing a prompt, and people saying they couldn't follow the reply. Jakob Nielsen wrote it up on Monday and calls them "huh?" signals.

The headline finding is the one that matters for design. No single property of the answer explained the confusion. Length didn't, readability didn't, vocabulary didn't. It depended on how they combined. The authors call it a conversational complexity budget. Nielsen calls it answerload, and his summary line is good: "Simplicity is a budget, not a dial."

This is part 5 of The Design Transition. Part 1 moved design's job from the screen to the spec. Part 4 was about what the agent reads before it makes anything. This one is about what happens after, when the output is something nobody on the team has seen.

The short version

When the output can't be predicted, the design is the states around the answer, plus a live number that says which state failed. The states are five: confident, unsure, refusing, waiting, and wrong. The number is the repair rate, how often users re-ask, rephrase, or say they're lost. Wang, Bilal, and Zaman used exactly those behaviors across more than 84,000 ChatGPT and Gemini conversations and found no single property of an answer explains confusion, which is why you need the behavior and not a readability score. Tanner Kohler at Nielsen Norman Group found AI-native build processes have no usability signals in their monitoring at all. And order is a state too: Claire Wang and colleagues at the University of Illinois found an AI summary shown first moved ratings about twice as much as the same summary shown second. My rule is that the repair rate belongs to design, by surface, read weekly with a sample of the real conversations.

You can't draw the answer. Draw the states.

I argued in You Cannot Mock a Distribution that a mockup of a generated surface is one sample presented as the decision. That's still true, and it leaves the obvious question. If not the answer, what does design actually draw?

The states. There are five, and each already has a chapter in the handbook, so I won't rewrite them here.

Confident is the default and needs nothing. Unsure is The Shape of Uncertainty: show confidence, hedge, stay silent, or ask, chosen by what a wrong answer costs. Refusing is Designing Refusal: the reason in a category the user can classify, then the nearest thing you can do. Waiting is Latency Is Interface, picked by the slow case, not the median. Wrong is Recovery and Trust Repair, the thirty seconds after the product was confidently wrong.

What's missing in most teams isn't any one of those. It's the inventory. One row per surface, one column per state, and a cell that says what this surface does in that state. Most AI features I look at have the confident column filled in and a generic error string in the other four.

The inventory is the spec. And it raises the next problem straight away. How do you know which cell failed last week?

The re-ask is a usability test you already run

Every product with a text box runs this test thousands of times a day. Someone reads the answer, doesn't get it, and tries again. That second prompt is the most honest feedback the product gets, and almost nobody is counting it.

Wang, Bilal, and Zaman did, and the details are worth knowing, as Nielsen reports them. Repeats were flagged at a 0.72 similarity between prompts. Readability scores had nothing to do with re-asking, and easier text had slightly higher odds of a confusion complaint. Each extra formatting element per 30 words, the bullets and bold and headers models love, raised the odds of re-asking by 9 to 12%. Simple lookups broke down less. Finance users trying to learn something had 2.6 times the odds of saying they were lost.

The caveats are real. It's observational, the chats are from 2025, and re-asking is a noisy proxy. Gemini showed about four times ChatGPT's odds of re-asking, but it was 3.5% of the data and nobody was randomly assigned, so I wouldn't build anything on that one.

But look at what the finding means for the dashboard. If no single property of the answer predicts confusion, you can't lint your way to clear answers. You have to watch the behavior.

Kohler, in The Role of UX in Agentic Software Factories at NN/g on October 30, describes the gap from the build side. In the AI-native lifecycles he looked at, monitoring watches technical health. A feature can be abandoned because people are confused, and nothing fires, because, in his words, "There are no usability-focused signals built into the monitoring system." His fix is to add UX distress signals, like abandonment and support spikes, next to the uptime alerts.

The repair rate is the cheapest one of those. It's already in your logs.

Order is a state too

There's a second study from this fortnight that changes how I'd fill in the inventory.

Claire Wang and colleagues at the University of Illinois, with Agam Goyal, Frederick Choi, Koustuv Saha, and Eshwar Chandrasekharan, ran a preregistered experiment with 278 people rating Amazon products. Each saw an AI summary of reviews and four real reviews, in one order or the other. By Nielsen's calculation from their coefficients, in his October 30 roundup, a summary shown first moved attitudes about 0.9 points on a five-point scale. Shown after the reviews, about 0.4. And people said they trusted the reviews more while being moved by the summary.

So the first thing on the screen wins. That has a direct consequence for the unsure state. If the product isn't sure, the doubt has to come first. A confident paragraph with a caveat in the last line has already done its work by the time anyone reaches the caveat.

Nielsen's line on it is the right posture: "Anchoring is unavoidable; deception is optional."

In the same roundup he restates his 1993 response-time limits, and it fits here too. Acknowledge every action within 0.1 seconds. Past ten seconds, honest progress and a cancel. His warning: "An endlessly spinning circle leaves users guessing." A user who's guessing re-asks. That shows up in the same number.

The repair rate is design's number

Here's where I'd draw the line between the functions.

Engineering logs it: the re-ask, the rephrase, the "I don't understand," per surface, per week. That's a pipeline job, and it should be boring.

Design owns what it means. Every week, someone on design reads a sample of the conversations behind the number and tags each miss with the state that should have caught it. Was the product unsure and sounded confident? Did it refuse in a way nobody could classify? Did the wait run long with no word? Was it wrong, with no repair? Or was it just too much at once, the answerload problem, where every fact was right and the person still got lost?

Then design changes the inventory cell, the copy, or the answer budget for that kind of task. That's what owning the quality bar means when you can't predict the output. Not reviewing every answer. Owning the number that says which answers to go look at.

It's The Rubric Is the Spec running in production. The rubric tells you what good looks like before launch. The repair rate tells you, after launch, where reality disagreed. I'll come back to where it sits on a design scorecard later in the series. For now, it needs an owner, and it's design.

One thing to try this week

Pull 50 conversations from last week where the user re-asked within two turns. Most teams can get that list with one query.

Read them. Tag each one with the state that should have caught it: unsure, refusal, wait, wrong, or too much at once. Don't fix anything yet.

Count the piles. The biggest one is your next spec. And if you couldn't pull the list at all, that's the first ticket.

Sources: Yingcan Carol Wang, Iman Munire Bilal, and Qamar Zaman, "When AI Becomes Hard to Understand: Cognitive Demands in Real-World Human-AI Conversations," arXiv, September 15, 2026. Jakob Nielsen, "UX Roundup: Practice Interviewing With AI, AI Answer Complexity, and more," UX Tigers, November 2, 2026, for the panel details, effect sizes, and the 0.72 threshold as he reports them. Tanner Kohler, "The Role of UX in Agentic Software Factories," Nielsen Norman Group, October 30, 2026. Claire Wang, Agam Goyal, Frederick Choi, Koustuv Saha, and Eshwar Chandrasekharan, "First Impressions: How Placement Shapes the Influence of AI Summaries," arXiv, September 14, 2026. Jakob Nielsen, "UX Roundup: AI Persuasion, Summaries as Dark Design, Feedback, and more," UX Tigers, October 30, 2026, for the 0.9 and 0.4 point estimates, his calculation, and the response-time limits.

Part of the running argument on AI Product Management: AI collapsed the cost of the parts product teams were trained to be good at, and for design that means drawing the answer got cheap while specifying every state around it, and owning the number that says which state failed, became the job.

Related answer: What is a repair rate, and why should design own it?