What counts as a repair?

Two behaviors: the user repeats or rephrases their prompt, or says they couldn't follow the reply. Wang, Bilal, and Zaman flagged repeats at a 0.72 similarity between prompts, as Jakob Nielsen reports in his November 2, 2026 roundup.

Why not just use a readability score?

Because it doesn't predict confusion. In Nielsen's write-up of the study, readability scores were unrelated to re-asking, and easier text had slightly higher odds of a confusion complaint. Each extra formatting element per 30 words raised the odds of re-asking by 9 to 12%. The demands combine, which the authors call a conversational complexity budget.

Who should own the repair rate?

Engineering logs it, per surface, per week. Design owns what it means: someone reads a sample of the conversations behind the number, tags the state that should have caught each miss, and changes the spec. Tanner Kohler at Nielsen Norman Group found AI-native build processes monitor technical health and have no usability signals at all.

Which states does the tagging use?

Five: unsure, refusal, wait, wrong, and too much at once. Each maps to a part of the spec: how the product shows uncertainty, how it refuses, how it handles latency, how it repairs trust after a confident error, and how much it puts in one answer.

How do I start this week?

Pull 50 conversations from last week where the user re-asked within two turns. Read them, tag each with the state that should have caught it, and count the piles. The biggest pile is your next spec. If you can't pull the list, that's the first ticket.

The Repair Rate, and Why Design Owns It

What is a repair rate, and why should design own it?

THE SHORT ANSWER

The repair rate is the share of conversations where the user has to fix the exchange: repeating or rephrasing a prompt, or saying they didn't understand the reply. A study by Yingcan Carol Wang, Iman Munire Bilal, and Qamar Zaman of more than 84,000 ChatGPT and Gemini conversations used exactly those behaviors and found no single property of an answer, like length or readability, explains confusion. So you can't lint for clear answers, you have to watch the behavior. Engineering logs it. Design reads a weekly sample, tags which state should have caught each miss, and changes the spec.

The repair rate is how often users have to fix the exchange with an AI product: they re-ask, rephrase, or say they don't follow. It's the cheapest usability signal you have, and it's already in your logs.

Why it's the right signal. Wang, Bilal, and Zaman studied more than 84,000 ChatGPT and Gemini conversations and found no single property of an answer explains confusion. Length, readability, and vocabulary only matter in combination. So you watch the behavior instead of scoring the text.

Why design owns it. The number only matters through what it changes. Engineering logs it. Design reads a weekly sample, decides which state failed, and changes the inventory: unsure, refusal, wait, wrong, or too much at once.

Why order matters too. Claire Wang and colleagues at the University of Illinois found an AI summary shown first moved product ratings about twice as much as the same summary shown second. If the product is unsure, the doubt has to come first.

The full argument is part 5 of The Design Transition, and the series runs in order on the hub. For how to write the unsure state itself, see How should a product show it is uncertain?

Last reviewed 2026-11-03 · 1 min read

What is a repair rate, and why should design own it?

SOURCES

THE LONG VERSION

RELATED ANSWERS