
Two people I read every week published on the human reviewer within two days of each other and came out on opposite sides. This is a Rewrite, because I think each is right about a different moment, and the test one of them recommends is missing the arm that would show it.
The short version
- Jakob Nielsen's November 16, 2026 roundup reports AutoAnalysis, a meta-analysis by Michelle Vaccaro and co-authors that pools 1,955 effect sizes from 420 papers.
- Where AI was also measured alone, it beat humans in 117 of 171 comparisons, and human and AI teams lost on average to the stronger solo performer. His advice is a three-arm test, human, AI, and team, then let the winner work alone.
- Harrison Chase, with Mihir Garimella and Anshul Gupta of Actively AI, argued the same week that human feedback is the bottleneck for agents and that an agent's scope should be designed around how easy it is to review. Both hold.
- The three arms score one task once, and Chase is describing what happens between tasks. My creator scan runs unattended, so a three-arm test would score the human at zero, and yet 13 of its first 42 reports hit the same step-in until one line in the prompt ended it.
- The rewrite is a fourth arm: the AI alone on the next run, with what the person wrote down after this one.
What Nielsen reports
The study is unusual in form. AutoAnalysis updates itself monthly, with agents doing the searching and screening. So far, as Nielsen reports it, they have sifted 15,495 records to find 420 qualifying papers.
The finding he leads with is that only 171 of the 1,955 effect sizes, 8.7%, came from experiments that measured AI working alone. Most studies compare a team against a human, see a gain, and stop. Where the AI did compete solo, it won 117 of 171 matchups. Humans won 48, and 6 were ties.
Teams did not rescue the humans. On average they lost to AI alone, and they beat both solo performers in only 32% of comparisons. The split that matters for anyone designing a workflow: when human and AI handled different subtasks, the team matched AI alone. When both worked the same task, the team lost. The classic setup where a person reviews the AI's output falls in the losing group.
His guidelines follow. Assume AI wins until a test says otherwise. Run three arms on at least 30 representative tasks. Ties go to the AI. Split the work instead of stacking a person on top. And apply Paul Meehl's broken-leg test: override the formula only when you know something decisive that it cannot see.
What Chase reports
"Goal Engineering" starts from the other end. When an agent gets something wrong, the usual cause is missing context, and the hardest context to supply is the kind that is not written down anywhere. It has to come from a person, as feedback.
So Chase, Garimella, and Gupta argue for scoping the agent around the feedback. Actively's first design was one agent per sales account. Reviewers had to hop between agents, each with a few items waiting, and a lesson about outbound taught to one agent had to be taught again to the next. The second design is agents per motion: one job, like outbounding, across many accounts. Similar work arrives together, and feedback on one task carries to the rest.
Their line on review is that it takes a lot of time and is worth it, because it is how the agent learns the domain.
They are scoring different moments
Put the two side by side and they look like a disagreement. One says the human in the loop is often in the way. The other says the human's feedback is the scarce input.
The three-arm test scores a task. One human, one AI, one team, one output each. In that frame the reviewer can only touch the output, and the data says touching it mostly makes it worse.
Chase is describing the gap between one task and the next. The reviewer's contribution there is not an edit to the output. It is a sentence that changes what the agent does next time, and every time after.
A person can add nothing inside a run and still be the reason run ten is better than run one. A single-task test cannot see that.
My scan would score me at zero
My creator scan is a scheduled agent. It runs while I am doing something else, so on any single run my contribution to the output is zero. Put it through three arms and the verdict is to let it work alone. Which it does.
But in Blend's Agent Saves 4.5 Hours a Loan. The Number Alex Wang Asks For Is How Often Someone Stepped In. I went back through the scan's reports and counted. 13 of the first 42 recorded the same step-in: the browser connector asked which Chrome to use, and no person was there to answer. The run fell back to a second browser every time and wrote that down every time. One line in the prompt, the browser's id, ended it.
13 of 42
And in Nielsen Says Review the Decisions You Never Made. The Card Is a Missing Line in the Brief. I counted the lines themselves. The two skill files that run my scheduled agents carry 30 lines stamped with my name and a date.
Those 30 lines are the whole human contribution, and all of it happened between runs.
The fourth arm
So I would keep Nielsen's three arms and add one.
Arm four is the AI alone on the next run, with whatever the person wrote down after the previous one. No review of the output. Just the note, placed where the agent reads it at the start.
Compare arm four with plain AI alone. If it wins, the person has a job on this work, and the job is writing the line. If it does not, the task has nothing unwritten left in it, and Nielsen's conclusion stands.
This is where his own broken-leg guideline leads. He says the human edge that remains is informational, something the AI cannot access, and he expects it to erode as AI reads more. Writing it into the brief is how it erodes on purpose. Chase's point about scope is the practical half: arrange the work so the same note helps as many tasks as possible.
It also changes what you measure. I argued in Kill "Human in the Loop" as a Control. Keep a Gate Ledger. that an approval step with no numbers is a hope. The fourth arm gives the reviewer a number to own: how much better the next run got because of what they wrote.
This week
Pick one agent that repeats a job. Take the last note a reviewer gave it, in any form, and check whether it reached the file the agent reads at start. Then run the job twice, with and without that note, and compare. That is arm four on a sample of one, and it tells you whether your review step is producing instructions or just edits. More on which agents hold up under this kind of accounting is in Enterprise AI Agents.
Related answer: Should a human review an AI agent's work or let it run alone?
Sources: Jakob Nielsen, "UX Roundup: Death of AI-Human Synergy, AI Answers Design Briefs, and more," Jakob Nielsen on UX, November 16, 2026, the item "Human + AI < Best Solo Performer: No Synergy in 420 Studies" read in full. The 420, 1,955, 15,495, 171, 8.7%, 117, 48, 6, 32%, and the 30 tasks are his report of AutoAnalysis by Michelle Vaccaro and co-authors; I did not open the paper or its site. Harrison Chase, Mihir Garimella, and Anshul Gupta, "Goal Engineering: How to Design Goals," LangChain, November 14, 2026, read in full. The 13 of 42 and the 30 lines are my own, from the posts linked above.
Frequently asked
What does the meta-analysis Nielsen reports say about human and AI teams?+
As Jakob Nielsen reports it in his November 16, 2026 roundup, AutoAnalysis by Michelle Vaccaro of Harvard Business School and co-authors from MIT pools 1,955 effect sizes from 420 papers on people working with generative AI. Only 171 of those came from experiments that also measured AI alone. There, AI beat humans in 117 of 171 comparisons. Human and AI teams lost on average to AI alone, beat both solo performers in 32% of comparisons, and did worst when human and AI worked the same task. I have not opened the paper itself.
What is the three-arm test?+
Nielsen's recommendation: score a human alone, the AI alone, and the human and AI team on at least 30 representative tasks, then give the work to the winner. Without the AI-alone arm, a team result can hide a human contribution that made the output worse. He also says to rerun it with every model generation.
What does Harrison Chase argue in Goal Engineering?+
With Mihir Garimella and Anshul Gupta of Actively AI, he argues that the hardest context to give an agent is the context not written down anywhere, that it arrives as feedback from a human expert, and that an agent's scope should be designed to make that feedback easy to give. Actively moved from one agent per sales account to agents per motion, one job across many accounts, so a reviewer could look at similar work together and a lesson learned once would carry over.
How can both be right?+
They describe different moments. The three arms score a single task once, where a reviewer added to a capable model tends to subtract. Chase describes what happens between tasks, where a reviewer's note becomes an instruction the agent reads on the next run. A person can add nothing inside a run and still be the reason run ten is better than run one.
What is the fourth arm?+
The AI alone on the next run, with whatever the person wrote down after the previous one. Compare it with plain AI alone. If it wins, the human has a job, and the job is writing the line, not reviewing each output. If it does not, Nielsen's conclusion stands for that task.
What is your own evidence?+
My creator scan runs on a schedule with nobody watching, so the human contribution inside any run is zero. Yet 13 of its first 42 reports recorded the same step-in that nobody was there to make, and one line added to the prompt ended it. The two skill files that run my scheduled agents carry 30 lines stamped with my name and a date. Those lines are the human contribution, and all of it sits between runs.


Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn