What is Nielsen's gate?

The first of his seven guidelines for joint cognitive systems: measure joint performance, evaluating the human and AI ensemble on real tasks against human-alone and AI-alone baselines, and ship only when the team wins on accuracy, speed, and error recovery. He credits the frame to Erik Hollnagel and David Woods, 1983, and cites it only as he reports it.

Why is refusal a separate column?

Because a team can match the human on accuracy against the written policy, beat the human on speed, and have nothing to recover from, while refusing less than the human did. The expense agent approved every submission that matched the policy. The human approver had slowed down on the fourth 'dinner with customer' from one submitter in a month, for a reason the policy never carried. That refusal was the judgment the seat held, and the three-column gate never measured it.

What fills the refusal column?

The human's reject pile, recorded before the agent sits down: every case the person in the seat said no to in the last quarter, with the reason in their words, plus the ten worst plausible outputs the design handbook's failure inventory asks for. The gate ledger then records each refusal the agent makes or fails to make, so the column has a number by week two.

Which of Nielsen's guidelines explains the failure?

The fifth, keep the human active: a person whose only job is to watch for rare failures will miss them, so give that person a live decision role in the loop. Finance's job had become watching. A standing watch is not a control; a specific gate with a ledger entry is.

How to Measure a Human and AI Team Before Shipping

How do you measure a human and AI team before shipping it?

THE SHORT ANSWER

Score the human and AI team as one unit against human-alone and AI-alone baselines, as Jakob Nielsen's October 26, 2026 reading of Hollnagel and Woods's joint cognitive system says, and add a fourth column to his three: refusal rate, held against what the human alone refused, with the reasons. My expense agent won Nielsen's three columns, accuracy, speed, and error recovery, for two weeks. It auto-approved expenses under $500 with a matching receipt, then 'dinner with customer' with a vague receipt passed every time, approvals rose 40%, about $2,400 was wrongly approved, and finance turned it off within the month. The human approver had refused the fourth dinner from the same person, and nobody had written that down. Fill the fourth column from the human's reject pile before the agent takes the seat: the last ten refusals in that seat with the reasons, next to the ten worst plausible outputs from the failure inventory. If the pile is empty, the baseline is a guess.

Score the team as one unit against both baselines, as Nielsen says, and add the column his gate is missing: refusal rate against what the human alone refused, with the reasons.

The expense agent that won the three columns and lost the fourth, and the reject pile that fills it, are in Nielsen's Joint Scorecard Has Three Columns. My Expense Agent Failed the Fourth. The agent itself is in Agents Fit the Human Pattern Nine Times in Ten. Design the Tenth., and the control-side argument is in Kill 'Human in the Loop' as a Control. Keep a Gate Ledger.

Last reviewed 2026-10-27 · 1 min read

How do you measure a human and AI team before shipping it?

SOURCES

THE LONG VERSION

RELATED ANSWERS