Bottom-up evals are the checks that only exist after you have read the outputs. The distinction is Shreya Shankar's, and the uncomfortable corollary is mine: the half of your eval suite a model can write for you is the half that catches nothing new.
Why the model is fluent at one half and useless at the other
A top-down eval is a restatement. You hand over the task description, the model hands back the same information in a checkable shape. That is a translation task, and translation is what these systems are best at. It also means the output contains exactly zero information that was not already in the brief.
Bottom-up evals are the opposite. They are new information, generated by a person with a stake in the outputs, doing something no document can substitute for: looking.
The example that makes it concrete
Brief: "summarize the customer call into a note the account team can act on."
Top-down set, written in a vacuum, and fine as far as it goes:
- Is there a decisions section
- Are action items attributed to a named person
- Is it under 400 words
- Does it avoid inventing attendees who were not on the call
Now read forty real outputs. Around the fifteenth, the pattern shows: every note is technically correct and reads like a press release. Confident, tidy, hollow. The customer said three contradictory things and the summary smoothed all three into one agreeable paragraph.
Nothing in the task description told you to check for that. It could not have. You did not know it was a failure mode until you had seen it thirty-eight times in a row.
The conversion rule
Every gut reaction has to become a yes/no that a grader can answer without judgment, or it gets dropped.
The dropping matters as much as the converting. A check that still requires taste is not an eval, it is an opinion with a checkbox next to it.
- Fails: "Is the tone appropriate."
- Passes: "Does the note preserve at least one point where the customer disagreed with themselves or with us, when the transcript contains one."
The second one is uglier. It is also answerable, which is the only property that matters. Most of the work lives in that rewrite, and about half the time the interrogation reveals the wince was actually about something else. That is the loop working, not failing.
What this caught on Heidi
Heidi runs agents that take real actions inside a company, so the eval surface is not "is the text good," it is "was that act okay."
The top-down set took an afternoon. Inside the tenant's permission scope. Reversible. Guard layer logged a receipt. Run stayed inside budget.
Then I watched real runs, and the failure that kept bothering me was none of those. The agent would do something fully permitted, fully reversible, correctly logged, in budget, and still wrong. It was acting on a signal that was real but thin: one data point, one stale source. Enough to clear the permission bar, nowhere near enough to justify moving.
No permission model produces that check, because permission is a top-down artifact that comes straight from the spec. Evidence quality only appears when you sit with the outputs.
That one wince became four things on the roadmap: evidence strength graded on signals, an autonomy gate with three axes instead of one (confidence, reversibility, and evidence grade together), a verifier portfolio where each verifier holds narrow authority instead of one omniscient judge, and gates that either enforce themselves or fail loudly.
The RAG harness had taught me the same lesson a year earlier. Retrieval precision looked great. The answers were correct and useless. Apparently I needed it twice.
The one-hour version
Block an hour. Read 50 outputs from whatever you are shipping. Write down every single wince without filtering, including the ones you cannot articulate. Then convert three of them into checks a grader can answer without judgment.
Three, not thirty. The other forty-seven are next week's.
Top-down evals are downstream of the spec. Bottom-up evals are upstream of it, because they are how the spec gets discovered in the first place. Teams that build only the first half watch every number go green and ship something that passes every check and still makes users flinch.