How do you measure how often people have to intervene in a long-running AI agent?

Answered by Falk Gottlob · Reviewed · 1 min read

THE SHORT ANSWER

Keep a step-in ledger: one row every time a person touches a run, with the stage, the kind (planned approval, correction, unblock, or takeover), the minutes, whether it has been seen before, and what it became (a rule, a planned approval, or nothing). Report unplanned step-ins per 100 runs and the share that are repeats. Alex Wang's November 12, 2026 edition asks for exactly this count, and the Blend figures he cites (4.5 hours automated and about $600 saved per loan) do not include it. In my own creator scan, 13 of 42 reports recorded the same step-in, a browser connector asking which Chrome to use with nobody there, until one line in the prompt on November 12 ended it.

Count the touches, and count the repeats separately. Hours saved and cost per unit are output numbers. The supervision cost of a long-running agent lives in how often a person has to step in, and whether it is the same step-in as last week.

The ledger, its six columns, and the 13-report example are in Blend's Agent Saves 4.5 Hours a Loan. The Number Alex Wang Asks For Is How Often Someone Stepped In.. The report the ledger sits inside is in Nielsen Says a Map Beats a Babysitter. Write the Map for the Next Run..

SOURCES

THE LONG VERSION

RELATED ANSWERS