# The Measurement Starter Set > How do you instrument a product whose surface is assembled at runtime? The > funnel assumed everyone saw the same screens in the same order. Nobody does > now. This is the replacement: eight events, four slices, and three questions > the instrumentation has to be able to answer. --- ## What breaks about the funnel A funnel measures progression through fixed steps. It works when the steps are the same for everyone. In a generated product, two users on the same feature can see different content, different amounts of it, in a different order, with different quality. A conversion rate across that population averages over the thing you actually want to know, which is *which version of the experience worked*. The replacement is not a better funnel. It is measuring **the output the user got, what they did with it, and whether they came back to that surface.** --- ## The eight events Start here. Eight is enough to answer real questions and few enough that they get implemented properly. | # | Event | Fires when | Key properties | |---|---|---|---| | 1 | `surface_shown` | A generated surface renders | surface, variant, confidence band, latency ms, input size | | 2 | `output_accepted` | User takes the offered action as-is | surface, output id, time to accept | | 3 | `output_edited` | User modifies before using | surface, output id, edit size | | 4 | `output_ignored` | Session ends or user proceeds without engaging | surface, dwell ms | | 5 | `output_reversed` | Undo, delete, or correction after acceptance | surface, time since accept, method | | 6 | `output_reported` | User flags it as wrong or bad | surface, category, free text | | 7 | `escalated_to_human` | User leaves for support, docs, or a person | surface, from where | | 8 | `refusal_shown` | Product declined | surface, refusal category | Two of these do the heavy lifting and are usually missing. **`output_edited` with edit size.** The single most informative signal in a generated product. Accepted-as-is and heavily-edited are completely different outcomes and most instrumentation records both as "used." **`output_reversed` with time since accept.** Reversal after ten seconds is a mistake caught. Reversal after two days is a mistake discovered, and those need different design responses. --- ## The four slices An aggregate number across a generated product tells you almost nothing. Every metric gets sliced these four ways, or it does not get looked at. **By confidence band.** Acceptance at high confidence versus low. If they are the same, your confidence signal is not reaching the user or is not calibrated. **By input size or difficulty.** Easy inputs mask hard-input failure. The aggregate is usually carried by the easy half. **By user tenure.** New users accept more because they do not know better yet. Experienced users' acceptance rate is the honest quality signal. **By segment or account size.** The largest, messiest accounts are where generated features fail first, and they are the ones you can least afford to lose. --- ## The three questions Whatever else you measure, the instrumentation has to answer these. If it cannot, add events until it can. ### 1. Is the output good enough to use as-is? Read: acceptance rate, edit rate, edit size. Sliced by confidence. Watch for a rising acceptance rate combined with a rising reversal rate. That combination means people are trusting output they should not, which is worse than low acceptance and looks better on a dashboard. ### 2. When it is wrong, do people catch it? Read: report rate and reversal rate against your own eval failure rate. This is the ratio that matters most and almost nobody computes it. If the eval says one in ten outputs on this surface fails, and the report-plus-reversal rate is far below that, users are not catching the failures. That is not a good number. It means the bad outputs are shipping into their work unchallenged. ### 3. Do they come back to this surface? Read: repeat use of the specific surface, not the product. Product retention hides feature abandonment. People keep using the product and quietly route around the generated feature, which is the most common way an AI feature dies, and the slowest to detect. --- ## What not to measure **Time on task, as a quality proxy.** Faster can mean better, and it can mean the user gave up reading. Without acceptance and reversal alongside it, it is noise. **Aggregate satisfaction on a generated feature.** It moves too slowly and it averages across the slices where the answer differs. **Volume of generations.** A vanity metric that goes up when the feature gets worse and people retry. --- ## The starter dashboard One screen. Six numbers, each sliced by confidence band. 1. Acceptance rate 2. Edit rate, with median edit size 3. Reversal rate, with median time to reverse 4. Report rate 5. Escalation rate 6. Repeat use of the surface, week over week Plus one number next to them all: the current eval failure rate for that surface. Putting the internal quality measure next to the behavioral one is what makes the dashboard honest, because the gap between them is the story. --- ## The one thing to do this week Check whether you can currently distinguish accepted-as-is from accepted-then- edited on your main generated surface. Most teams cannot. That one event, with edit size, will tell you more than the next three weeks of dashboard work. --- _From "Reading Behavior You Did Not Design", chapter 18 of The Design Operating Model._ _falkster.com/design/reading-behavior-you-did-not-design_