Jakob Nielsen's roundup this morning has a section titled "AI Output Outruns Human Review," and the sentence in it that a CPO should read twice is this one: the productivity gain "arrives with an unpaid invoice for human attention." His example is a marketing manager who asks for campaign concepts and gets 40. Generating them took seconds. Comparing them could eat the afternoon. Ask for another batch and the review debt compounds "like credit-card interest."
I audited that invoice on my own fleet this spring. The expensive line was not the hours.
The short version
Jakob Nielsen's October 16 roundup argues that AI output outruns human review, so the real productivity number has to count generation, review, and repair together, and the output should be designed so people inspect the weightiest decisions first. The measurement is where I would push. Review time only counts the review that happened. When I audited the 39 agents I had appointed by April, 13 were orphans whose output nothing outside the fleet ever read, and 13 of the 39 builds I said yes to in 80 days ended with zero references anywhere else in the system. That invoice was never presented. The number to report next to review time is the count of outputs nobody opened. The kill is the batch, 40 concepts or 10 variants as the default shape of what an agent hands a person. The replacement is three directions with assumptions stated, a reader named before the run, a 20-minute kill decision logged, and the unopened count printed next to the generated count. The design handbook chapter Disclosure Without Overwhelm has the ladder and the triggers that make a low default safe.
What Nielsen is asking for
His section is short and it makes two moves.
The first is about measurement. "Measure the total time spent on generation, review, and repair. Compare it with the time needed to achieve the same quality without AI." Counting only generation speed, he says, leaves the human work off the invoice. He repeats a line from his midyear assessment: limited human review capacity could become the bottleneck for adopting agents.
The second is about design. Present a few distinct directions, explain their assumptions, reveal supporting detail on demand, and let the user pick a direction before the system elaborates it into a mountain of material. Then watch people review the result and log where they lose their place, repeat comparisons, or open another tool to verify a claim. Each stumble is a design opportunity. His image for the whole section is a small robot presenting an enormous pearl that the recipient needs a ladder and a magnifying glass to appraise.
The design half I would sign. The measurement half has a hole, and the hole is where the money went in my fleet.
The invoice nobody presented
Review time can only be measured for review that happened. The failure mode I found was the other one.
By April I had appointed 39 agents. The audit I wrote up in 44 Appointed Agents, One Attention Budget: Three Lines on the Charter found that 13 of them fired between 7 and 9 a.m. without anyone having planned that, and 13 were orphans: their output was never read by anything outside the fleet. The fleet is 44 today. Those 13 did not cost me an afternoon of comparison. They cost me nothing, which is the problem, because they also produced nothing. Nielsen's marketing manager at least looks at the 40 concepts.
The build side had the same shape. In Twenty Minutes Is Not a Reason to Build I counted 39 yeses in 80 days, every build minutes long, and 13 of them ended with zero references anywhere else in the system. Nobody reviewed those slowly. Nobody reviewed them.
So the invoice Nielsen wants to put the human work on was, in my case, never presented. The generation ran. The review did not. A time measurement reports that as zero cost, which is the opposite of the truth.
The number that catches it is simpler than a time study: how many outputs did nobody open. Print it next to the count generated. On my fleet dashboard that pair was 39 and 13 for the agents and 39 and 13 for the builds, and I would have rather seen it in week two than in month four.
The half that worked
The agents that survived tell you what the fix is, and it is not more review time.
In Count the Kills, Not the Prototypes, my Instant Prototype Agent's output split into thirds: about a third became production after engineering cleanup, a third triggered a better idea, and a third were killed in about 20 minutes instead of costing a three-week spike. That third that died fast is the cheapest review there is, and it is the one Nielsen's afternoon-of-comparison example does not have, because 40 concepts of equal weight give the reviewer nothing to kill first.
Across the whole fleet, a named owner was the best predictor of survival. Not the quality of the render, not the model, not the prompt. Whether a person's name was on the reader line before the run.
The kill, and the ladder
The practice to retire is the batch. Forty concepts. Ten variants. "Here are some options." That shape is the default output of most agents I review, and it does exactly what my design handbook chapter Disclosure Without Overwhelm warns about: showing everything is not transparency, it is a way of making the user responsible for checking work they never asked to supervise. A batch makes the reader the quality control for a system whose length costs it nothing.
Nielsen's design advice is the right replacement and the chapter gives it a structure. Five disclosure levels: the answer alone, provenance in one line, the shape of the reasoning in two or three steps, the full chain on request, the raw trace. The decision is which level is the default for which reader at which stakes, and what one action moves them up a rung. A first-time user starts at provenance. A reviewer starts at the reasoning shape. Nobody starts at the raw trace.
The rule I would add to his section is the one that makes a low default defensible: the system raises the level on its own. Confidence below the stated floor, an action that cannot be undone, an output that contradicts something said earlier, money or permissions or deletion involved. Those triggers go in the eval set so they survive model changes. Without them, every argument about how much to show collapses into a fight between the person who wants a clean interface and the person with the scarier hypothetical, and the hypothetical wins. With them, three directions is a safe default, because the fourth appears when it should.
The Receipt and the Work(coming Oct 28) is the chapter underneath this one: the artifact a system hands you is the receipt, and a receipt anyone can produce in ninety seconds stops working as proof that the work was done. A batch of forty is a very long receipt.
Four lines instead of forty
What replaces the batch, on every agent surface I would ship:
Three directions, each with its assumptions stated, before anything is elaborated. Nielsen's number, and the right one.
A reader named before the run. Blank reader, no run. This is the charter line from the attention budget post, and it is the one that would have caught 13 orphans in week one.
A kill decision logged within 20 minutes of first render. Not a review, a kill. The third of prototypes that died there paid for the tool.
The unopened count next to the generated count on the dashboard. If the first number is zero, the second one means something. If it is not, no amount of review time measurement will tell you what the generation was worth.
This belongs to the Enterprise AI Agents argument that agent count is a vanity metric and the number that matters is how much of the fleet's work is still used after ninety days. Nielsen is counting the review. I am counting the reader.
This week: pull up the agent or feature on your roadmap that hands a person "options." Count the options. Then find the dashboard that counts generations and ask it how many were opened. If it cannot answer, that is the ticket.
Related answer: How do you measure the cost of reviewing AI agent output?
Sources: Jakob Nielsen, "UX Roundup: Skewed AI Use | Reviewing AI Results | Writing & Design | Losing Users' Work | Autofill | Hover Effects | Time Gains | Asking What You Know | Multilingual Video | CAPTCHA | Social Presence," UX Tigers, October 16, 2026, the section "AI Output Outruns Human Review" read in full. Falk Gottlob, "44 Appointed Agents, One Attention Budget," "Twenty Minutes Is Not a Reason to Build," and "Count the Kills, Not the Prototypes."
Also on Medium
Full archive →AI Agents and the Future of Work: A Pixar-Inspired Journey
What product managers can learn about AI agents from how Pixar runs a film team.
Many AI Agents Are Actually Workflows or Automations in Disguise
How to tell agents from workflows from cron jobs, and why it matters for what you ship.
Frequently asked
What does Jakob Nielsen mean by AI output outrunning human review?+
In his October 16, 2026 roundup he argues that AI produces work faster than people can judge it, so the productivity gain arrives with an unpaid invoice for human attention. His example is a marketing manager handed 40 campaign concepts: seconds to generate, an afternoon to compare, and each new batch compounds the review debt. His remedy is to measure total time across generation, review, and repair, and to design output so people inspect the weightiest decisions first.
Why is review time the wrong number for agent output?+
Because it only counts the review that happened. When I audited the 39 agents I had appointed by April, 13 were orphans whose output nothing outside the fleet ever read, and 13 of the 39 builds I said yes to in 80 days ended with zero references anywhere else in the system. That invoice was not paid late. It was never presented. The number that catches it is the count of outputs nobody opened, reported next to the count generated.
What should replace a batch of options as the default agent output?+
Four lines. Three directions with their assumptions stated, instead of forty. A reader named before the run, because across my fleet a named owner was the best predictor of survival. A kill decision logged within 20 minutes, which is where a third of my prototypes ended. And the unopened count printed next to the generated count on the dashboard.
What is a disclosure ladder?+
Five levels of how much an AI product shows: the answer alone, provenance in one line, the shape of the reasoning in two or three steps, the full chain on request, and the raw trace. The design decision is which level is the default for which reader at which stakes, and what single action moves them up one rung. Showing everything is not transparency, it makes the user responsible for checking work they never asked to supervise. It is a chapter of my design handbook, Disclosure Without Overwhelm.
How does a low disclosure default stay safe?+
The system raises its own level when a trigger fires: confidence below a stated floor, an action in an irreversible class, an output that contradicts something said earlier, a recent reversal of a similar output, or money, permissions, external communication, or deletion involved. Without triggers every argument about defaults turns into a fight between a clean interface and coverage. With them, a low default on routine output is defensible.
How did the prototypes in your fleet split?+
In my Instant Prototype Agent, about a third of prototypes became production after engineering cleanup, a third triggered a better idea, and a third were killed in about 20 minutes instead of costing a three-week spike. The third that died fast is the half of the review invoice Nielsen's section does not count, and it is the cheapest review there is.
Comments (0)
Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn