Fifteen Models Met 90% of the Design Brief. Kill 'Meets the Brief' as the Pass.

Kombai's benchmark, per Nielsen: 15 models met over 90% of 160 design briefs, humans 98%. A check everything passes measures nothing. Keep what fails.

Falk Gottlob · · 7 min read

Execution Falkster cover on teal: a grid of eight identical cream dashboard cards, each with a navy checkmark, and one tilted card in the middle with a different layout and an orange cross.

Jakob Nielsen's November 16 roundup carries a benchmark that most people will quote for its first number. Frontier models now meet 99% of a design brief. This is a Kill List entry for that number as a pass mark, because the same study shows what it cannot see.

The short version

  • As Nielsen reports it, Aman Rusia and colleagues at Kombai gave 15 frontier models and several human designers the same 160 briefs for screens such as dashboards and forms. Every model met more than 90% of the criteria, two met 99%, and the humans met 98%.
  • The designs were also interchangeable: two models' designs for one brief scored 0.59 on a 0 to 1 distinctiveness scale, and moving a brief to another product domain changed a model's design by 0.58 against 0.90 for humans.
  • Nielsen's fix is to ask for 10 designs and not 1, at about $53. Do that. Then retire "meets the brief" as the pass, since a check that 15 models and the humans all clear separates nothing, and the 0.58 says the brief never carried the product.
  • The Design Operating Model's rubric chapter describes the same failure inside a team: a rubric that scored everything 4 out of 5 and stopped being opened.
  • Before converging, delete every criterion nothing failed, add the lines only this product has from thirty real outputs, and have two people sort the ten separately.

What the benchmark measured

The setup is clean. 160 briefs, each with criteria a screen has to meet. 15 current models and a group of human designers. Nielsen credits the authors for testing models that are still on the market, and flags two things a reader should know: Kombai sells AI design tools, and the humans could browse inspiration libraries while each model worked alone.

On criteria, everyone did well. More than 90% for every model. 99% for Claude Opus 5.5 and GPT-6 Astra. 98% for the humans.

On distinctiveness, the models clustered. Two models answering the same brief produced designs that scored 0.59 apart, where a human design and a model design scored 0.76. And when the same brief was moved to a different product domain, a model's design changed by 0.58. A human's changed by 0.90. Nielsen's image for it is an architect who designs a hospital, a school, and a prison in the same house style.

0.58

is how much a model's design changed when the same brief moved to a different product domain, on a 0 to 1 scale, against 0.90 for human designers.

His advice follows from the second number. The study took each model's first idea from a fresh context. No designer stops at one, so ask for ten built on different organizing principles, then merge the best. Ten from the priciest model tested, at $5.33 a design, is about $53.

I would do that, and I would not stop there.

The first number is the one to retire

Look at the criteria score again. The spread from the worst model to the best is under ten points, and the humans sit inside it.

A check with that spread is not ranking anything. If I handed a design lead 16 portfolios and a rubric on which every one scored between 90 and 99, the lead would not conclude that all 16 were excellent. The lead would conclude the rubric was too easy.

The 0.58 says what kind of easy. The model's design barely moved when the product changed, and it still met the criteria. So the criteria were ones a hospital dashboard and a school dashboard both satisfy: the fields are present, the hierarchy is legible, the primary action is visible. Those are worth checking. They are a floor.

The brief did not carry the product, so nothing in the score could tell whether the design did.

I have seen this rubric before

In The Rubric Is the Spec, the chapter of the Design Operating Model on writing a design rubric, I describe the version that fails. Someone smart writes it from first principles on a good afternoon. It comes out with clarity, consistency, and delight on it. Then it scores every output 4 out of 5 forever and quietly stops being opened.

That is the Kombai criteria score inside a team. The dimensions are the ones you would expect a design team to care about, which is why they carry no information. They cannot separate this week's output from last week's.

The one that worked was extracted. Pull thirty real outputs from the surface. Grade each good, mixed, or bad with one sentence of why, and no criteria yet. Then read your own why column for the phrases that repeat. When we did this for an AI feature, the dimension that mattered most was not on the list we had written beforehand. It had been sitting in my review notes the whole time.

Those lines are specific to one product. "It invented a number." "It buried the answer." A hospital dashboard and a school dashboard would not share them, which is the point.

Three steps before you converge

Nielsen's ten alternatives are the diverge half, and they are now nearly free. The converge half is where the 99% does damage, because ten designs that all meet the brief leave the pick to whoever is most senior and how they feel that day. I made that argument about a canvas of three options in Figma's Agent Made Three Prototypes From "Surprise Me." Kill the Brief. Write the Rubric First. This is the same problem with a rubric that exists and has gone flat.

So, with the ten in front of you:

Run the criteria across all ten. Any criterion that nothing failed gets deleted from the scoring sheet, or moved to a floor checklist a script can run. It was not helping you choose.

Add the lines only this product has. Take them from thirty real outputs of this surface, in the words you used when you graded them. A good line fails most of the ten.

Have two people sort the ten separately, then compare. Scoring Your Own Work(coming Nov 18) has the protocol. Where the two sorts split, the brief is thin, and that is the sentence to write next.

This week

Take the last brief your team gave a model. Run its criteria across every alternative that came back and count how many criteria failed at least once. If the answer is zero or one, you have a floor and no rubric, and the next hour is thirty outputs and a why column. This is the craft end of AI Product Management: the spec that matters is the one that can fail the work.

Related answer: What does it mean when AI designs meet 99% of the brief?

Sources: Jakob Nielsen, "UX Roundup: Death of AI-Human Synergy, AI Answers Design Briefs, and more," Jakob Nielsen on UX, November 16, 2026, the item "AI Designs Meet 99% of the Brief but Look Alike" read in full. The 15 models, 160 briefs, 90%, 99%, 98%, 0.59, 0.76, 0.58, 0.90, $5.33, and $53 are his report of the benchmark by Aman Rusia and colleagues at Kombai; I did not open the paper. The 4 out of 5, the thirty outputs, and the dimension missing from the list are my own, from the Design Operating Model chapters linked above.

Share this post

Frequently asked

What did the Kombai design benchmark find?+

As Jakob Nielsen reports it in his November 16, 2026 roundup, Aman Rusia and colleagues at Kombai gave 15 frontier models and several human designers the same 160 briefs for UI screens. Every model met more than 90% of the briefs' criteria, Claude Opus 5.5 and GPT-6 Astra met 99%, and the human designers met 98%. On a distinctiveness scale from 0 to 1, two models' designs for the same brief scored 0.59 against 0.76 for a human design paired with a model design, and moving a brief to a different product domain changed a model's design by 0.58 against 0.90 for humans. I have not opened the paper itself.

What does Nielsen recommend?+

Ask for 10 designs and not 1. He calls it diamond prompting: diverge with many deliberately different designs, then converge by merging the best ideas. He prices ten alternatives from the most expensive model tested at about $53 and says the real cost is the time to review them. He also notes the study measured only each model's first idea and that Kombai sells AI design tools.

Why kill 'meets the brief' as the pass mark?+

Because everything passed. Fifteen models and the human designers all landed between 90% and 99%, so the criteria cannot rank them. The cross-domain number explains it: a model's design barely changed when the product changed, which means the criteria were ones any product satisfies. A criterion that nothing fails carries no information about the design in front of you.

What replaces it?+

Three steps before converging on one of the ten alternatives. Run the brief's criteria across all ten and delete every criterion nothing failed. Add the lines only this product has, extracted from thirty real outputs graded by gut with one sentence of why. Then have two people sort the ten separately, because where they split is where the brief is thin.

Where does the thirty-outputs method come from?+

From the Design Operating Model chapter The Rubric Is the Spec. A rubric written from first principles came out with clarity, consistency, and delight on it, scored every output 4 out of 5, and stopped being opened. Grading thirty real outputs first, and reading the reasons that repeat, produced a dimension that was not on the list written beforehand and mattered most.

Jakob NielsenAman RusiaKombaiDesigndesign rubricdesign briefAI designbenchmarks
Helpful?

Keep reading

The short answer

The Impact LoopThe daily loop the AI-era product org runs: customer, spec, build, story, adoption. One measure per stage, one owner for the whole loop.Read the chapter

Part of

AI Product Management

About the author

Falk Gottlob

Falk Gottlob

Product Executive · Founder, Falkster.AI

Thirty years shipping product, from Microsoft Research and Adobe to Salesforce, where he grew Quip into what became Slack Canvas. Four startup exits, including a $6.5B healthcare merger and a company Microsoft bought. Four-time Chief Product Officer. Now founder of Falkster.AI, building Heidi, the company brain for AI-native businesses. This notebook is written from inside the build, not above it.

Sign in with LinkedIn to leave a comment.

Sign in with LinkedIn