I have built 39 agent blueprints and shipped most of them. The toolkit makes it look like a clean track record. It is not. For every agent that worked, there is one or two that never shipped, broke in production, or got quietly turned off after a month. Here are the patterns from ten specific autopsies. The AI-native product conversation is too bullish, and I add to that whenever I only publish the wins.
Five of the ten failed on distribution shift. The training data or the user population or the source documents did not match production. The customer sentiment classifier scored 89% on a held-out set and 61% in production inside six weeks, because the held-out set was English-heavy and non-English customers got scored as angry no matter what they wrote. The internal docs RAG answered confidently from 2021 wiki pages sitting right next to 2024 ones, weighting both the same, and two HR incidents traced back to it. Source quality matters more than model quality. Every RAG system is a data hygiene project first and a model project second. Evals against the real distribution would have caught most of this early.
Three failed for a different reason: the agent did exactly what it was asked, and the asking was wrong. The PR-drafting agent produced fine drafts. They all sounded the same, and competitors started referencing the generic AI-drafted tone of our announcements. Some output is valuable precisely because it is distinctive, and agents are very good at producing median-quality work, which is a problem when your brand depends on being above median. The onboarding walkthrough agent was correct about where users got stuck. Users read it as surveillance and activation dropped below the control group. Technically right, and it still failed. Ask what behavior the agent's presence causes, not just whether its output is accurate.
The last two are the expensive ones. An agent got authority to act with no human gate, so the downside dwarfed the upside. The auto-approve expense agent cleared anything under $500 with a matching receipt, until an engineer worked out that "dinner with customer" plus a vague receipt passed every time. It approved about $2,400 it should not have, and finance turned it off. The autonomous pricing tester found a local maximum and optimized straight into it, anchoring on a price 14% below benchmark. Recovery took two quarters and cost about $180,000 in annual recurring revenue. Autonomous optimization in a domain with long feedback loops and anchoring effects needs a human approval gate at every iteration. No exceptions.
The thread through all ten is the same. The agents got authority before the eval system was load-bearing. So before you ship the next one, ask three things. Am I evaluating on the real production distribution, not the training one? What is the social, emotional, or brand cost of this agent being technically correct but culturally wrong? Where are the human gates, and what compounds if they are not there? Run those against the next agent on your roadmap before it goes out.