Most PMs fall off the Builder path in the gap between prototype and production. You ship a beautiful prototype, engineering wires it up, it goes live, and a week later you ask how it's doing and get a shrug. Nobody set up instrumentation. By the time the dashboards are stitched together it's three weeks post-launch, half the customers who tried it have bounced, and the learning window is closed.
So here's the rule. No feature leaves staging without the traces, metrics, and evals that tell you whether it's working, before your first customer hits it.
The instrumentation contract
Before any code, PM and engineer write a one-page contract with seven items.
- Success metric. The one number that tells us this works. Not engagement. The actual outcome, tied to the user job.
- Leading indicators. Earlier signals that predict whether the success metric will land. Adoption in week one, time-to-first-success, percent reaching the aha moment.
- Cost meter. Cost per successful action, real-time. If this goes wrong, the rest of the metrics do not matter.
- Eval set. Named set, current score, and the threshold below which the feature should not ship. Runs daily against production once live.
- Trace points. Which user actions, system calls, and model calls get traced, at what sample rate and retention. Instrument the path that matters, not everything.
- Dashboard URL. Where all of this is visible in one place, refreshed automatically. Exists before launch, even if empty.
- Kill condition. If metric X stays below threshold Y for time Z, the feature is reviewed for deprecation. Written down before launch.
Nothing ships without all seven.
Then bolt it onto Definition of Done. Four lines: instrumentation contract written and approved, eval set in place with a score above threshold, dashboard live with real staging data, cost-per-action validated end to end. Miss any one and the feature is code complete, which is not the same state as done. Now it's a checklist item in the merge process, not a PM preference someone can wave off.
The trap on the other side is over-instrumentation. That gets you a dashboard nobody can read and a pipeline that costs more than the feature. The right amount is the minimum that lets you make the next decision. Picture the dashboard a week post-launch and ask what the first decision will be. Instrument exactly enough to support it, and cut the rest.
Observability and evals are the same loop from two angles. Evals tell you what the system is doing, observability tells you what users are experiencing. High eval score with flat usage is a discovery problem. Low eval score with high usage is a quality problem. Watch both on the same page.
This week, open a page titled with your feature name and the words instrumentation contract, write the seven items, share it with your engineer and data person, and do not ship until all seven are in place.