If an agent does the work of a team member, measure it like one. The industry talks about agents as magic that either works or does not, and when it does not you tweak the prompt until the demo looks good again. That is not how anything that does real work gets managed. Last quarter I ran a performance review on every agent in my fleet. Three failed and I shut them down.
The four-metric scorecard
I grade every agent on four things, and they travel across any agent from a research summarizer to a bug-fixer.
Precision. The share of outputs correct and usable without rework. The headline number, and a liar on its own.
Escalation rate. How often the agent flagged its own uncertainty versus how often it should have. This is the one nobody tracks and the one that matters most. An agent that never escalates is making calls it should be handing up.
Time-to-output. Wall-clock from trigger to usable result. An agent that is correct but slower than I am is a net loss no matter how impressive the output.
Trust trend. Over the quarter, are reviewers approving more of its work untouched, or less? The trend is more informative than any single score.
Why the four resist gaming
A single metric invites gaming. Optimize for precision alone and an agent learns to only answer easy cases while staying silent on hard ones, which tanks coverage. Optimize for speed and quality drops. The four metrics pull against each other, so you cannot juice one without exposing another. Trust trend is the anti-gaming backstop: it measures whether humans actually rely on the output more over time, which no single-run trick can fake.
The three agents I fired did not fail because they never worked. They failed on the metric that gets people hurt. They were confidently wrong in ways I could not detect at review time. One was a competitive-intel summarizer that produced beautiful briefs with occasional invented details I only caught because I happened to know the source. The clean output was exactly what made it dangerous. A worse-looking agent would have made me check.
Push from pull to push
On the measurement side, reliability also depends on not relying on your own attention. Dashboards are pull-based, and the human attention span is the bottleneck. A measurement agent stack runs variance detection on a schedule and flags drops the morning they happen instead of on day three. Start with a three percent week-over-week variance flag, then tune up until you settle on the threshold that produces one or two flags per week. That way your reliability signal reaches you before the data team or an exec asks.
Run the review on your fleet this quarter. Pick the four metrics, grade every agent, and look hardest at the escalation rate, because that is where the dangerous ones hide.