# What can an eval catch that a demo cannot? > A demo is a performance on inputs someone chose; an eval is a measurement across inputs that represent reality. We demoed an AI feature on three hand-picked cases and it was flawless, then ran the eval against 200 real cases split into named slices. On average it scored fine. On one slice, about a fifth of real traffic, it produced confidently wrong output, the kind of failure worse than an error because it looks right. The demo was blind to it because a demo never samples the case that breaks the thing. The eval caught it because named slices expose what an average hides. A demo can start a shipping conversation; only an eval can end one. Source: https://falkster.com/answers/what-can-an-eval-catch-that-a-demo-cannot Author: Falk Gottlob Pillar: Ship Stories Published: 2026-08-12 Last reviewed: 2026-08-12 Reading time: 2 min read --- A demo shows a chosen best case. An eval shows the distribution. That difference is the whole story, because a shipping decision should only ever be made on the distribution. ## The failure that almost shipped We built an AI feature, demoed it on three hand-picked cases, and it was flawless. The room wanted to ship. I ran the eval I had written before building: 200 real inputs, split into named slices. On average it scored well. The long-input slice, about a fifth of real traffic, produced confidently wrong output, the failure that is worse than an error because it looks right and the user trusts it. None of the three demo cases were long, so the demo was structurally blind to it, and the average hid it because four-fifths of traffic dragged the mean up over the failing fifth. ## Why the eval saw it A demo's job is to communicate the idea at its strongest, which means it excludes the cases that undercut the idea. It cannot tell you the product is not ready because it never sampled for readiness. An eval grades against inputs that represent what will actually arrive, and the [named slice](/blog/the-eval-that-caught-what-the-demo-missed) is what turns "it works" into "it works here and fails there." Shipping the demo would have returned wrong results to one in five users, a trust event that produces exactly the month-two silence a landing failure looks like. The eval cost a day. The rule that came out of it: a demo can start a conversation but cannot end one, and the eval has to exist before the thing does. ## Sources - [The Eval That Caught What the Demo Did Not, falkster.com](https://falkster.com/blog/the-eval-that-caught-what-the-demo-missed) ## Related essays - [The Eval That Caught What the Demo Did Not](https://falkster.com/blog/the-eval-that-caught-what-the-demo-missed) - [The Eval Is the Spec](https://falkster.com/handbook/the-eval-is-the-spec) --- Canonical HTML: https://falkster.com/answers/what-can-an-eval-catch-that-a-demo-cannot Markdown: https://falkster.com/answers/what-can-an-eval-catch-that-a-demo-cannot.md (or send `Accept: text/markdown`) Site index for agents: https://falkster.com/llms.txt · https://falkster.com/agents.md · https://falkster.com/sitemap.xml Citation: attribute to "Falk Gottlob, falkster.com" with the canonical URL.