The Wince Is the Spec: Bottom-Up Evals a Model Can't Write

The half of your evals a model can write for you is the half that catches nothing.

That's the uncomfortable version of a distinction Peter Yang shared this week, from Shreya Shankar. Top-down evals are what you'd write in a vacuum, handed nothing but the task description. Claude is good at those. Bottom-up evals are the ones that come out of reading a pile of sample outputs and noticing what bothers you. Claude, per Shreya, is very bad at those. That part is all you.

I'd push it one step further. Top-down evals aren't just easier for a model to produce. They're a restatement of the brief in a different format. Same information, new shape. Of course the model is fluent at it, that's a translation task, and translation is what these things do best.

Which is exactly why they miss so much.

The short version

Top-down evals are the checks you can write from the task description alone: required fields, format, length, no invented facts. A model drafts those well, because they are the brief translated into a new shape, and you should let it. Bottom-up evals come from reading fifty real outputs and noticing what bothers you, and no model can produce them, because they require having read the outputs and caring whether they're good. The conversion rule is one line: every gut reaction becomes a yes/no a grader can answer without judgment, or it gets dropped. On Heidi, that loop caught actions that were permitted, reversible, logged, in budget, and still wrong, which turned into evidence-strength grading and a three-axis autonomy gate. Top-down evals are downstream of the spec. Bottom-up evals are upstream of it, because they're how the spec gets discovered.

What a task description can't tell you

Say the brief is "summarize the customer call into a note the account team can act on." A top-down eval set writes itself. Is there a decisions section. Are action items attributed to a person. Is it under 400 words. Does it avoid inventing attendees who weren't on the call.

All fine. All necessary. That's your floor, and a floor is worth having, because format violations and missing fields are cheap to catch and annoying to catch by hand.

Now read forty real outputs.

Somewhere around the fifteenth one you'll notice that every note is technically correct and reads like a press release. Confident, tidy, and hollow. The customer said three things that contradicted each other and the summary smoothed all three into a single agreeable paragraph. Nothing in the task description told you to check for that. It couldn't have. You didn't know it was a failure mode until you saw it thirty-eight times in a row.

That reaction is the spec. It lives in your gut, not in the brief, and no model can extract it from a document nobody has written yet.

The rule I use

Every gut reaction has to become a yes/no that a grader can answer without judgment, or it gets dropped.

That's it. That's the whole conversion. And the dropping matters as much as the converting, because a check that still requires taste to evaluate isn't an eval, it's an opinion with a checkbox next to it. "Is the tone appropriate" fails. "Does the note preserve at least one point where the customer disagreed with themselves or with us, when the transcript contains one" passes. It's uglier. It's also answerable.

Most of the work is in that rewrite. You feel something vague, you interrogate it until you find the observable thing underneath, and half the time the interrogation tells you the wince was really about something else. Good. That's the loop doing its job. It's the same extraction move as building an eval rubric from thirty graded outputs, pointed at the checks instead of the dimensions.

Where this bit me on Heidi

Heidi runs agents that take real actions inside a company. So the eval surface isn't "is the text good," it's "was that act okay."

Our top-down set was straightforward and I wrote most of it in an afternoon. Was the action inside the tenant's permission scope. Was it reversible. Did the guard layer log a receipt. Did the run stay inside its budget.

Then I watched actual runs, and the failure that kept bothering me was none of those. The agent would do something fully permitted, fully reversible, correctly logged, and still wrong in a way I couldn't name at first. It was acting on a signal that was real but thin. One data point, one stale source, enough to clear the permission bar and nowhere near enough to justify moving.

You cannot derive that from a permission model. Permission is a top-down artifact, it comes straight from the spec. Evidence quality only shows up when you sit with the outputs.

That wince turned into a chunk of roadmap: evidence strength graded on signals, an autonomy gate with three axes instead of one (confidence, reversibility, and evidence grade together), a verifier portfolio where each verifier has narrow authority instead of one omniscient judge, and gates that either enforce themselves or fail loudly. Every one of those started as me squinting at a run trace and going, hm, that's technically fine and I hate it.

The RAG harness taught me the same lesson a year earlier and I apparently needed to learn it twice. Retrieval precision looked great. The answers were correct and useless. Same shape as the eval that caught what the demo missed: everything you thought to measure was green.

So what is the model actually good for here

Plenty, just not the part everyone wants to hand off.

Give it the floor: let it draft the top-down set from your task description, then keep the ones that are cheap and unambiguous. That's real time saved and you should take it. If you want the smallest honest container for the result, the five-row eval template is where mine go.

Then use it as a converter, not a source. I'll paste in a messy note ("this summary flattens disagreement") and ask for three candidate yes/no phrasings that a grader could answer with no judgment call. It's good at that. It's also good at clustering, once you have sixty scrappy observations and want to see which ones are the same complaint wearing different clothes.

What it can't do is have the reaction. The reaction requires having read the outputs, and having a stake in whether they're good.

That's the whole asymmetry. Top-down evals are downstream of the spec. Bottom-up evals are upstream of it, because they're how the spec gets discovered in the first place. The eval is the spec is only true if you build the half that isn't already in the brief. Teams do the first half, watch the numbers go green, and ship something that passes every check and still makes users flinch.

One thing to try this week: block an hour, read 50 outputs from whatever you're shipping, and write down every single wince without filtering. Then convert three of them into checks a grader can answer without judgment. Not thirty. Three.

The other forty-seven are still useful. They're next week's.

Sources: Shreya Shankar, whose top-down / bottom-up distinction this is built on. Peter Yang, who surfaced it. Hamel Husain on evals, still the best practical writing on looking at your data.