What is the tool good for today?

The discovery pass and the top-down floor. Hamel reports it found issues with human handoff, formatting, and voice out of the box. Checks that can be written from the task description alone, such as format and required fields, are also safe to let it draft.

Why count times customer cost?

Because frequency alone ranks the annoying above the damaging. My error analysis log is fifty rows with one note each, then a tally where each error's count is multiplied by what it costs the customer. Only the top three get an eval in a cycle.

Why one failure per grader?

A flag that passes only when four checks pass cannot tell you which one failed, and it runs an LLM judge where three of the four could be decided by code. Separate graders are cheaper and each one moves on its own.

Did this change your view on models and bottom-up evals?

Partly. In August I wrote that a model is bad at the half of evals that comes from reading outputs. Hamel's review says candidate discovery has improved, and I take that. Choosing which failure matters still needs someone who knows what it costs.

Should an Auto Eval Tool Pick Your First Eval

Should an auto eval tool pick your first eval?

THE SHORT ANSWER

No. Use the tool to find candidate failures and do the picking yourself. Hamel Husain's review of Claude's auto eval tool found that it recommended an eval from a menu before he had read any data, and also that its one-shot issue discovery was the strongest he has seen. So take the list and close the menu. Read fifty real outputs by hand, one note each, rank the errors by count times customer cost, and build an eval for the top three only. Then write one grader per failure. I had a feature score well on average across 200 real cases while a slice worth about a fifth of our traffic failed badly, and a single flag that bundles four checks hides a failure the same way an average does.

Anthropic's claude-api plugin can now propose and build evals inside Claude Code, and The Menu Is Not the Error Analysis is my read of Hamel Husain's review of it: keep the discovery, skip the menu. The fifty-row log I use for the ranking step is in The AI Eval Starter Kit: Six Templates, Error Analysis to Gate, and the case for breaking results into slices is The Eval That Caught What the Demo Did Not.

It belongs to the AI Product Management argument: deciding what correct means, and which failure matters, is the part of the job that did not get cheap.

Last reviewed 2026-10-01 · 1 min read

Should an auto eval tool pick your first eval?

SOURCES

THE LONG VERSION

RELATED ANSWERS