A demo shows a chosen best case. An eval shows the distribution. That difference is the whole story, because a shipping decision should only ever be made on the distribution.
The failure that almost shipped
We built an AI feature, demoed it on three hand-picked cases, and it was flawless. The room wanted to ship. I ran the eval I had written before building: 200 real inputs, split into named slices. On average it scored well. The long-input slice, about a fifth of real traffic, produced confidently wrong output, the failure that is worse than an error because it looks right and the user trusts it. None of the three demo cases were long, so the demo was structurally blind to it, and the average hid it because four-fifths of traffic dragged the mean up over the failing fifth.
Why the eval saw it
A demo's job is to communicate the idea at its strongest, which means it excludes the cases that undercut the idea. It cannot tell you the product is not ready because it never sampled for readiness. An eval grades against inputs that represent what will actually arrive, and the named slice is what turns "it works" into "it works here and fails there." Shipping the demo would have returned wrong results to one in five users, a trust event that produces exactly the month-two silence a landing failure looks like. The eval cost a day. The rule that came out of it: a demo can start a conversation but cannot end one, and the eval has to exist before the thing does.