Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date.

Lesson 09 of Becoming Forward Deployed Engineer is the bake-off most teams skip on the way to the model. It is right, and the table it produces has two columns missing. I have the numbers for both, from launches that went the way the table said they would not.

The short version

Lesson 09 sorts Meridian's intake documents into four types three ways, a regex table, TF-IDF with logistic regression, and a 3B local model, on 750 labeled documents, each approach seeing a random 12-word fragment, split by claim so nothing leaks. Rules score 97.5% with three no-answers, the trained classifier 100%, the model 87.5% and about 5,805 times slower per fragment. The model came third. Two columns are missing from that table. A slice: a classifier of mine scored 89% on held-out data and 61% in production in six weeks because the held-out set shared the training set's skew, and a feature of mine scored well across 200 real cases and failed on one slice that was a fifth of traffic. A date: the table re-run on production rows at week six, with the no-answer count as the alarm. The row to fear is the 100% with zero no-answers, because it cannot say it has never seen this before.

What the lesson ran

Meridian's VP wants documents sorted into four types before they reach an adjuster: policy declaration, adjuster notes, contractor estimate, photo manifest. The lesson's argument is that a classification task like this should be run against the simpler options before anyone picks a language model, and then it runs them.

The setup is careful in the ways that matter. The 750 documents already carry labels, so the right answers are known. Each approach sees a random 12-word fragment, not the whole document, because that is what intake looks like. The train and test split is by claim rather than by document, because documents from one claim share an identifier and would leak across the split.

Three rows. Rules, a regex table. Trained, TF-IDF with logistic regression. Model, llama3.2:3b. Rules and trained are scored on 120 held-out fragments; the model on 48, because it is slow on CPU.

The sample run:

Rules, 97.5%, three no-answers, 0.023 ms at the median, $0 per thousand documents.

Trained, 100%, zero no-answers, 0.935 ms, $0.

Model, 87.5%, zero no-answers, 133.5 ms, $0.159 per thousand. About 5,805 times slower per fragment than the rules.

The model came third on accuracy, cost, and speed at once. And the lesson makes a second point I'd put above the first: rules that return no answer instead of guessing are useful, because the abstention is what a router can escalate on. The public section ends at that table. The repository behind it is paid, and I haven't read it, so nothing past the table is claimed here.

The column the split by claim cannot see

A split by claim stops leakage. It does nothing about skew.

I had a trained classifier once that would have looked like the middle row. Sentiment, not document type, but the same shape: a held-out set, a good score, a go decision. It scored 89% on held-out data. Six weeks into production it was at 61%. The training set was English-heavy, and non-English customers were scored as angry no matter what they wrote. The held-out set was cut from the same pile, so it had the same skew, so it had no way to show the problem. That story is number two in 10 AI Agents I Built That Failed. The Honest Retrospective., and the reason it matters here is that the split was clean. Nothing leaked. The table would have been right and the row would have been wrong.

The other number is from Which Slice Holds the 3%?. A feature scored well on average across 200 real cases. On one slice, about a fifth of our actual traffic, it failed badly. By failure mode that was a small share of unsupported claims. By slice it was one customer in five getting the broken version.

So the first missing column is a slice. Cut the 120 held-out fragments by whatever the training data is skewed on. Language. Customer size. Which system the document came from. The thing you already suspect, because you always suspect something. Then score each row per slice, and the number that goes in the table is the worst slice, not the average.

The column with a date on it

The second missing column is when.

The lesson's table is scored once, on day one, on held-out rows. My 89% became 61% in six weeks, which is before any user had time to stop checking the output, let alone before anyone scheduled a re-run. The Prompt Commit argues that a score is nothing without the rows it was scored on. The companion rule is that a score is nothing without the date it was scored, because the rows that matter at week six are production rows, and they are not the rows in the split.

Which is where the lesson's own best idea does the work. Three of the rules' 120 answers were no-answer. That cell is the only one in the table that moves before the accuracy does. When the intake mix shifts, a regex that matched nothing says so, and the router escalates. Put that count on a dashboard and you have a drift alarm that costs nothing and needs no labels.

Then look at the row with the best score. 100%, zero no-answers. A regex that matches nothing abstains. A classifier with four classes always picks one of the four. The day a fifth document type shows up at intake, the trained row files it as one of the four, confidently, in under a millisecond, and nothing in the table can tell you. The row to fear is the perfect one.

The drop

This is the harness, the version you can ship this week.

Run the lesson's three rows on the same fragments, split by claim. Don't skip the model row because you already know it loses; the latency and cost cells are the ones you'll need in the conversation with whoever wanted the model.

Add a slice column. Cut the held-out set by the skew you suspect, and report the worst slice beside the average.

Add a date. At week six, re-run the whole table on production rows, not the held-out set, by slice. Chart the no-answer rate from day one, because it moves first.

Route rules first. Escalate abstentions to the next row. The trained classifier earns the second seat when its worst slice holds at week six, not when it hits 100% on day one.

Run the Oracle Before You Swap the Model is the same instinct from Lesson 03, one level up: before anyone proposes a bigger model, split the loss into the shares that are retrieval, model, and version. This is the version of that for the step before the model exists.

This is a standalone post alongside The FDE Transition. The closest part is The FDE Writes the Eval: Almost Right Has a Number, which argues the eval is the deployment artifact. The bake-off table is that eval's first page, and the slice and the date are what keep it from being the only page.

It belongs to the argument on AI Product Management: running the comparison got cheap, and knowing which row is still right in week six did not.

This week: find the last classification task your team settled with a model. Run the three rows on 120 real fragments. Then cut them by the skew you already suspect and look at the worst slice. If you don't have a week-six date in the calendar, that is the date.

Related answer: How do you choose between rules, a trained classifier, and an LLM?

Sources: "Lesson 09: The Cases Where a Model Is the Wrong Tool," Becoming Forward Deployed Engineer, October 30, 2026, public section read in full; the post is paid past its sample-run table and the private repository is not claimed. Falk Gottlob, "Which Slice Holds the 3%?" (September 29, 2026), "10 AI Agents I Built That Failed. The Honest Retrospective." (May 4, 2026), "The Prompt Commit" (October 26, 2026), and "Run the Oracle Before You Swap the Model" (October 9, 2026).