Why not just measure review time, as Nielsen suggests?

Because a time study reports an output nobody reviewed as zero cost, and in my fleet that was the expensive case. Thirteen of 39 agents produced work that was never read by anything outside the fleet. The generation ran, the review never happened, and the time measurement would have called it free.

What replaces a batch of forty options?

Three directions with assumptions stated, a reader named before the run, a kill decision logged within 20 minutes of first render, and the unopened count printed next to the generated count. Across my fleet a named owner was the best predictor of whether an agent's output survived, and about a third of prototypes were killed in 20 minutes, which is the cheapest review there is.

How much should an agent show the reviewer?

Pick a default per reader and per stakes from a five-level disclosure ladder (answer alone, provenance, reasoning shape, full chain on request, raw trace), and let the system raise the level on its own when confidence drops, the action cannot be undone, or money, permissions, or deletion are involved. Showing everything makes the user responsible for checking work they never asked to supervise.

How Do You Measure the Cost of Reviewing AI Output?

How do you measure the cost of reviewing AI agent output?

THE SHORT ANSWER

Count the unopened, not only the hours. Jakob Nielsen's October 16, 2026 roundup says AI output outruns human review and the fix is to measure generation, review, and repair time together. Review time only counts the review that happened. When I audited the 39 agents I had appointed by April, 13 were orphans whose output nothing outside the fleet ever read, and 13 of the 39 builds I approved in 80 days ended with zero references anywhere in the system. Report the count of outputs nobody opened next to the count generated, name a reader before every run, cap the output at three directions instead of a batch of forty, and log a kill decision within 20 minutes.

Nielsen's section is right about the shape of the problem and wrong about the number. The afternoon spent comparing 40 concepts is a cost you can see. The cost I found on my own fleet was invisible to a time study: 13 of 39 agents whose output nothing ever read, and 13 of 39 builds that nothing else in the system ever referenced. The invoice was never presented, so review time said zero.

The argument, the four lines that replace the batch, and the disclosure ladder from my design handbook are in Kill the Batch of Forty: Count the Unopened, Not Review Time. The fleet audit is 44 Appointed Agents, One Attention Budget, and the thirds split is Count the Kills, Not the Prototypes.

Last reviewed 2026-10-16 · 1 min read

How do you measure the cost of reviewing AI agent output?

SOURCES

THE LONG VERSION

RELATED ANSWERS