The Eval That Caught What the Demo Did Not

This is the story I tell right after the prototype-beat-the-spec one, because they are two halves of the same lesson. The first is about building before you write. This one is about measuring before you ship, and it is the time a demo almost took me for a fool.

The short version

We built an AI feature, demoed it on three hand-picked cases, and it was flawless. The room wanted to ship. I ran the eval I had written before building: the same feature against 200 real cases, split into named slices. On average it scored well. On one slice, roughly a fifth of our actual traffic, it failed badly, in a way that would have reached customers and embarrassed us. The demo was not dishonest. It was three lucky draws from a distribution with a hole in it, and a demo only ever shows you the draws someone chose. The eval showed the distribution, including the hole. We did not ship. We fixed the slice, re-ran, and shipped a week later on a number instead of a feeling. The demo is a performance. The eval is the only reviewer in the room that is not trying to impress you.

The demo everyone loved

The feature was an agent that read a messy input and produced a clean structured result. The specifics do not matter; what matters is that it was the kind of thing that demos beautifully, because when it works, it looks like magic. We picked three examples, ran them live, and the room lit up. Someone said "ship it." Someone else asked when, not whether. The energy in the room was the energy of a thing that is obviously done.

I have been in enough of those rooms to know the feeling is not evidence. The three cases were chosen, by us, from the inputs we already knew the thing handled well. That is not cheating. It is what a demo is. You show the best version of the idea so people can see what you are aiming at. The problem is only that the room then wants to make a shipping decision on the basis of a performance, and a performance is a sample of size three, curated by the people with the most to gain from it looking good.

The eval nobody was excited about

Before I built the feature, I had written the eval. Not elaborate: a definition of what a good output looked like, and a set of real inputs to grade against. That is the eval-is-the-spec discipline, and the reason it has to come first is exactly this situation. If I had written the eval after the demo, I would have unconsciously built it around the cases the feature already handled, and it would have agreed with the demo, because I would have taught it to.

So after the room said ship, I said give me a day, and I ran it. Two hundred real inputs, pulled from actual traffic, split into a few slices I suspected might behave differently: short inputs, long inputs, inputs in a second language, and ambiguous ones where a human would also have to guess. The average score came back fine. Good enough to ship, if the average were the thing that mattered.

The average is never the thing that matters. The long-input slice, which was about a fifth of our real traffic, failed. Not marginally. The feature produced confidently wrong structured output on long inputs, the kind of failure that is worse than a blank, because it looks right and is not. A customer hitting that slice would not have seen an error. They would have seen a clean, plausible, incorrect result, and trusted it.

That failure was completely invisible in the demo, because none of the three demo cases were long. We had, without meaning to, demoed the exact slice that worked and skipped the exact slice that didn't. The average hid it because four-fifths of the traffic was fine and dragged the mean up over the failing fifth.

Why the eval saw what the demo could not

The difference is not effort or sophistication. It is what each one is built to show you.

A demo shows a chosen best case. Its whole job is to communicate the idea at its strongest, which means it systematically excludes the cases that would undercut the idea, not out of dishonesty but out of function. Nobody demos the input that breaks the thing. So a demo is structurally blind to failure. It cannot tell you the product is not ready, because it was never sampling for readiness.

An eval shows the distribution. It grades the product against inputs that represent what will actually arrive, and the named slices are the part that carries the weight, because an average is a blindfold. Our feature scored well on average and was not shippable, and the only artifact in the building that could hold both of those facts at once was the eval with the slice broken out. The slice is what turns "it works" into "it works here and fails there," which is the only sentence a shipping decision should ever be made on.

What it would have cost to ship the demo

Run the counterfactual, because it is the reason I bother. If we had shipped on the demo, one in five real interactions would have returned confidently wrong results. That is not a bug ticket. That is a trust event. The failure signature of landing is month-two silence, and nothing produces month-two silence faster than a tool that gives a user a clean answer that turns out to be wrong, because after that happens once, they stop believing the clean answers that are right. We would have shipped a feature that demoed like magic and quietly taught a fifth of our users not to trust it.

The eval cost a day. The demo-based ship would have cost a slice of our credibility that we would then have spent two quarters trying to earn back. That asymmetry is the entire case for eval-driven development, and it is the same asymmetry as the prototype story: a cheap measurement up front against an expensive failure downstream.

What I do now

The rule is simple and I hold it hard: a demo can start a conversation, but it cannot end one. Nothing ships on the strength of a demo. The demo aligns the room on what we are trying to do. The eval decides whether we did it, and the eval has to exist before the thing does, graded against real inputs, broken into slices chosen to find the failure rather than hide it.

The teams I have seen ship reliable AI products all have this reflex. When someone says "the demo looks great," the next sentence in the room is "what does the eval say," and if there is no eval, the demo is treated as what it is: encouraging, and not evidence. The eval-first product org is not a process document. It is this one habit, repeated until it is boring.

Try this week

Take the AI feature closest to shipping in your world, the one with the good demo and the momentum behind it. Before it ships, write down how you would grade it against 100 real inputs. Build that set from actual traffic, and deliberately include one or two slices you suspect might fail, the long ones, the weird ones, the ones nobody demos.

Run it and look at the slices, not the average. The gap between what the demo showed you and what the eval tells you is the gap between how good the feature feels and how good it is. Only one of those two reaches your customer, and it is not the one that got applause in the room.