You ship a feature Tuesday. Wednesday, onboarding completion drops 8 percent. Nobody notices until the Monday metrics review, and by then the root cause is buried under five other things. You argue about it, cannot reproduce the state, add it to the list to investigate, and never do. The KPI Watchdog fixes that Wednesday: at 8:17am Slack says onboarding completion dropped 8.2 percent, the likely cause is the new custom-fields screen (91 percent confidence), and a clickable prototype of a simpler screen is already live.
The seven components
You do not need exotic tooling. First, a KPI registry, a file in your repo (I use kpis.yaml) that defines each KPI: name, source query, healthy band, investigation threshold, escalation threshold. Humans edit it, git tracks it, the agent reads it. Second, a metric collector, a 30-line Python job that pulls each KPI hourly and computes the delta versus the 7-day average. Third, a change log that pulls deploys, flag flips, and vendor changes into one feed to correlate against.
Fourth, a signal ingester that pulls the last 48 hours of support tickets, Gong or Chorus call snippets, NPS responses, and cancellations when an alert fires. Fifth, the correlation step, the one piece that requires craft. Sixth, the prototype builder, the money move. Seventh, an operator interface where the agent posts findings with one-click actions.
The correlation step is the hard part
Most components are mechanical. The correlation prompt is where judgment lives. It takes the drop, the change log, and the qualitative signal and returns a ranked list of causes, each with a confidence level, supporting evidence, counter-evidence, and the smallest intervention that would test the hypothesis. Two things separate it from amateur versions: it asks for counter-evidence so it does not just confirm the first story, and it asks for the smallest intervention, so the candidate fix is a one-line copy change on the affected step, not "redesign onboarding."
The prototype builder
For the top-ranked cause, the agent generates a clickable prototype using Claude Code with your design system, source code, and the user flow loaded. Three to eight minutes to a live URL. Across four surfaces at Smartcat over a year: about a third of prototypes are not worth A/B testing (you rejected a bad fix in 20 minutes), a third become the real fix (cycle time from drop to shipped goes from 3 weeks to 4 days), and a third trigger a better idea. All three beat the old outcome where nothing tangible shows up until next Monday.
Do not build the whole thing this week. Pick one KPI, define its healthy band and investigation threshold in a small YAML file, and write a 30-line Python script that pulls it hourly and posts to Slack when it moves. Run it a week, tune the thresholds, then add the correlation step, then the prototype builder for one recurring alert type. The full version takes a month of evenings and is the highest-ROI agent I have built.