Why does it matter which line inference cost sits on?

Because gross margin is what boards and buyers read, and it only reflects the cost of delivering an AI product if serving cost is in cost of revenue. If the bill lands in R&D or a shared cloud line, the product's margin looks like a 2022 SaaS margin while the real cost grows unseen.

What about training and evaluation costs?

Keep them out of cost of revenue. They are investment, not the cost of serving a customer today. The S-1's compute line mixes training, research, and serving, which is why subtracting all of it from revenue understates a lab's gross margin.

How granular should the split be?

By feature, at minimum for the top AI features. The unit that matters to product is cost per successful outcome, and that needs the serving cost of each feature divided by the outcomes it produced.

Who should ask for the split, product or finance?

Product. Finance can measure margin, but only product can change it without cutting the feature, so the CPO has the reason to ask and the use for the answer.

What should a company do with a vendor price cut once costs are visible?

Decide on purpose and write it down: pass it through as price, spend it on quality, or keep it as margin. Anthropic says its newest model cut customer cost per task by about 40 percent. Without a policy, usage growth usually absorbs the saving.

Where Should AI Inference Cost Sit on the P&L?

Where should AI inference cost sit on the P&L?

THE SHORT ANSWER

Put the cost of serving customers in cost of revenue, split by feature, and leave training, evaluation, and internal experiments in operating expense. Otherwise gross margin flatters the product and nobody who designs the feature sees what a successful action costs. Anthropic's S-1, filed October 26, 2026, books $8,945 million of compute and infrastructure as operating expense against $16,286 million of first-half revenue and reports no gross margin; Tomasz Tunguz derived 85 percent from a footnote, while subtracting the whole compute line gives about 45. Where one cost sits moved the answer 40 points.

In cost of revenue, split by feature. Training, evaluation, and experiments stay in operating expense.

The reason is visibility. A product team that never sees the serving bill can't make the trade-offs that move margin: which model a feature calls, how often, for whom, and what happens to price when the vendor's price drops. Anthropic's S-1 shows how much the classification matters at the top of the market, where the same half-year reads as an 85 percent or roughly 45 percent margin depending on one line.

The full argument, including the pass-through policy for vendor price cuts and the one-page monthly review with the CFO, is Anthropic's S-1 Has No Gross Margin Line. Your AI Product Needs One., part 5 of The CPO Transition. The unit to measure is in Gross Margin Is Your Job Now.

Last reviewed 2026-10-29 · 1 min read

Where should AI inference cost sit on the P&L?

SOURCES

THE LONG VERSION

RELATED ANSWERS