You Cannot Mock a Distribution

The short version

When a screen is generated at runtime there is no final state to draw, so the specification is a range instead of a mockup. Three columns: the best case worth aiming at, the worst case you would still ship, and what the product refuses to do. Everything between the second and third is allowed and ships without your review, which is how a designer stops being the bottleneck on a surface that produces a thousand states nobody drew. The worst acceptable column does most of the work. The test that it is real is whether two people sort ten actual outputs the same way.

A mockup of a generated surface is one sample from a distribution, presented as if it were the decision.

Useful thing to make. It aligns a room and gives engineering something to react to. What it cannot do is specify the product, because the product that ships is a thousand states, and the mockup covered one of them, and it happened to be the one where the retrieval worked and the input was clean and the user's account had data in it.

Every designer working on an AI feature already knows this. Most are still specifying with mockups anyway, because there was no other instrument.

Three columns

Best case. Worst acceptable. Refuses to.

Best case is the target. What a great output looks like when everything is available and clean, so the team knows what to optimize toward.

Worst acceptable is the floor. Below it, do not ship. This column does more work than the other two combined, because it is the only thing that converts "good enough" from a mood into a decision. Without it, good enough is whatever the most senior person felt about the last five examples they saw.

Refuses to is the boundary. Not a quality bar, a hard line where the product declines rather than degrades.

A support-reply drafter inside an agent console, done properly. Best case: the draft answers the customer's actual question in three sentences, cites the policy article it drew from, matches the tone of the last five replies this agent sent, and needs no edit before sending. Worst acceptable: the draft is factually correct and cites its source, reads generic, needs light editing, and the agent sends it after under thirty seconds of work. Refuses to: draft anything containing a commitment about refunds, dates, legal outcomes, or account access, which return an empty draft and a pointer to the human process.

Look at what the refusal column is doing there. It is not a claim that the model is bad at refunds. It is a judgment that the cost of a confident wrong answer about a refund is high enough that shipping nothing is better. That is a design decision and it belongs to design.

The sentence that makes this worth doing

Everything between the second and third columns is allowed and does not need your review.

That is the payoff, and it is the reason to write a range spec rather than a longer principles document. A specified range is permission. It tells a team exactly how far they can go without asking, which means the work moves at generation speed instead of at the speed of one person's calendar. I described the general shape of that collapse in Design Just Got Promoted, and the range spec is where it becomes operational.

Teams that skip it end up with a designer reviewing outputs one at a time forever.

No amount of care fixes that. The problem is arithmetic.

Write the worst acceptable column first

It is the hardest one, and it disciplines the other two.

Teams that start with best case write an aspiration and then reverse-engineer a floor to match it, which produces a floor nobody would actually enforce. Starting at the floor forces the real conversation, which is what you would tolerate on a bad Tuesday with a real customer in front of you.

Write each cell as what the user sees, not what the system does. "Retrieval returns three documents" is not a cell. "Answer with two cited sources, both opened from the answer in one click" is.

Make refusal a category rather than a topic. "Refuses medical advice" is a topic and your team will argue about its edges until the product is sunset. "Refuses to state a dosage, a diagnosis, or a course of treatment" is a category, and a reviewer can catch it in an output.

And keep probabilities out. "Correct ninety percent of the time" is a model target, not a design specification, and it tells nobody what to do with the other ten percent. Frequency belongs in the eval report. Behavior belongs in the spec.

At Smartcat everyone could agree on a perfect translation in five minutes. We spent a month fighting over the worst one we'd auto-send without a human. That fight was the product.

The four states everyone forgets

Most first drafts specify the happy middle and nothing else. Four states need an explicit answer, and each one lands in one of the three columns.

Nothing to work with. Empty account, first run, no context. What does the best case even look like when there is no data, and should the feature appear at all yet.

Partial input. Half the fields, an ambiguous request, a truncated document. The system will produce something. Decide now whether that something is acceptable or refused.

Conflicting sources. Two retrieved documents disagree. Does the product pick one, show both, or decline. This is the state where products most often invent a confident synthesis nobody asked for.

The long tail language or format. Input arrives in a language, format, or domain nobody tested. Refuse, degrade, or attempt. Pick one on purpose rather than discovering the answer from a support ticket.

The sorting test

This is how you find out whether the range is specified or merely written.

Pull ten real outputs. Real ones, from a prototype or from production, not ones you composed to make a point. Have two people independently sort them into four piles: best case, acceptable, below the floor, should have been refused. Then compare.

Agreement on eight or more means the range is doing its job. Below that, the disagreement will not be spread evenly, it will cluster at exactly one boundary. That boundary is the column to rewrite. Rewrite it, sort again.

Run it before launch and once a quarter after. The boundaries move as the model and the product change, and a range spec nobody re-tests becomes fiction inside two releases.

The spec then goes in three places: the brief, the ticket, and the eval set. That last link is what keeps it honest. When the worst acceptable column changes, the eval changes with it, and the mechanics for that are the same ones in The Eval Is The Spec. A range spec that is not wired to an eval is a document describing a product nobody is building anymore.

If you want to see the range before you write it, generate against it. Twenty outputs at the floor and twenty at the boundary tell you more in an afternoon than a week of discussion, which is the same argument as Instant Prototyping pointed at specification instead of build.

This week

Take the feature you are working on right now and fill in the refusal column only. Not the other two. Just write down what the product will decline to do.

Send that list to your PM and your engineering lead. If anybody is surprised by a line on it, you just found the most important conversation of your week, and it was going to happen eventually with a customer instead.

The full three-column template, the sorting test, the four forgotten states, and the worked example are in The Range Spec. The compressed version is How do you design a screen that is generated at runtime?

---

Chapter 5 of a series on the design operating model. Next: how a product says it is not sure without looking broken.