The gap between your prototype and your production is where most PMs fall off the Builder path. You ship a beautiful prototype, engineering wires it up, it goes live, and a week later you ask "how is it doing" and get a shrug. Nobody set up instrumentation. By the time dashboards are stitched together it is three weeks post-launch, half the customers who tried it have bounced, and the learning window is closed. The rule fixes this: no feature leaves staging without the traces, metrics, and evals that tell you whether it is working, before your first customer hits it.
The instrumentation contract
Before any code, PM and engineer write a one-page contract with seven items.
- Success metric. The one number that tells us this works. Not engagement. The actual outcome, tied to the user job.
- Leading indicators. Earlier signals that predict whether the success metric will land. Adoption in week one, time-to-first-success, percent reaching the aha moment.
- Cost meter. Cost per successful action, real-time. If this goes wrong, the rest of the metrics do not matter.
- Eval set. Named set, current score, and the threshold below which the feature should not ship. Runs daily against production once live.
- Trace points. Which user actions, system calls, and model calls get traced, at what sample rate and retention. Instrument the path that matters, not everything.
- Dashboard URL. Where all of this is visible in one place, refreshed automatically. Exists before launch, even if empty.
- Kill condition. If metric X stays below threshold Y for time Z, the feature is reviewed for deprecation. Written down before launch.
Nothing ships without all seven.
Definition of done upgrade
Add four lines to your Definition of Done: instrumentation contract written and approved, eval set in place with score above threshold, dashboard live with real staging data, and cost-per-action validated end to end. If any is missing, the feature is code complete, a different state from done. This hardens the practice into a checklist item in the merge process rather than a PM preference.
Enough, not everything
The trap on the other side is over-instrumentation, which gives you a dashboard nobody can read and a pipeline that costs more than the feature. The right amount is the minimum that lets you make the next decision. Imagine the dashboard a week post-launch and ask what the first decision will be. Instrument exactly enough to support it.
Observability and evals are the same loop from two angles. Evals tell you what the system is doing. Observability tells you what users are experiencing. High eval score with flat usage is a discovery problem. Low eval score with high usage is a quality problem. Watch both on the same page.
This week, open a page titled with your feature name and the words instrumentation contract, write the seven items, share it with your engineer and data person, and do not ship until all seven are in place.