Rank AI Features by Downside Exposure, Not Engagement

The most important AI feature in your product is not the one people use most, and it is not the one closest to the invoice. It is the one where a silent quality drop costs you customers fastest.

The short version

Stop ranking AI features by engagement or revenue attribution. Rank them by downside exposure: if this feature's quality quietly dropped 20% tomorrow, how fast would it cost you customers? Downside exposure is a function of three things, revenue flowing through the feature's workflow, the cost of a wrong answer, and how reversible the damage is. Score every AI feature on those axes and the ranking rarely matches your dashboard. The flashy high-engagement feature lands mid-table, and the boring feature buried in the customer's core workflow tops the list. That top feature is, almost by definition, the one nobody is watching and the one with the stalest evals. Point your eval budget at downside exposure, not usage.

This came out of a comment thread last week. Someone asked which metric should rank AI features: engagement, revenue attribution, or usage depth. My answer was none of the three, at least not alone. Engagement rewards the features people fiddle with. Revenue attribution flatters whatever sits closest to the money. Both miss the feature that can actually hurt you.

The test

If this feature's quality quietly dropped 20% tomorrow, how fast would it cost us customers?

That is downside exposure. It is a function of three things:

Revenue through the workflow. Not revenue attributed to the feature, revenue flowing through the workflow the feature sits in. A summarization step inside your billing pipeline touches every dollar, even if nobody would list it as a revenue driver.

Cost of a wrong answer. A bad movie recommendation costs a shrug. A bad dosage extraction, a bad compliance flag, or a bad number in a customer-facing report costs trust, and sometimes more than trust. This is the cost of being wrong applied one feature at a time.

Reversibility. Some damage is a bug fix. Some damage is a churned account that tells three peers why. Rank how hard the trust is to win back once it breaks, not how hard the bug is to fix.

Score every AI feature on those three axes and the ranking rarely matches your dashboard. The flashy feature with the great engagement numbers usually lands mid-table. The top of the list is the boring feature buried in the customer's core workflow that everyone assumes just works.

Which is exactly the problem

The feature with the highest downside exposure is, almost by definition, the one nobody is watching. It is not in the demo. It is not in the board deck. Its evals were written at launch and have not been read since, because it just works.

Until it works 20% less, silently, and the first signal you get is a churn call.

The fix is not more dashboards. It is pointing your eval budget at the ranking this test produces instead of the ranking your engagement metrics produce. Eval coverage should follow downside exposure, not usage. This is the operating principle behind the eval-first product org: quality coverage goes where a silent failure hurts most. If you need a place to start, the five-row eval template is enough to cover the top of the list this week.

The features people fiddle with can afford a bad week. The features people rely on cannot.

Run the test on your own product. If the feature at the top of your downside list is also the one with the stalest evals, you already know what this quarter's eval work is.