As lesson 6 defines it, a small fixed set of prompts with a machine-checkable pass rule. It does not ask whether a model is good, only whether it does the specific things your system needs. Five probes at ten trials each is the lesson's budget, kept small so the suite is rerun rather than filed.

Why run probes per harness configuration?

Because the harness moved more points this week than the lesson's two candidates differed by, and because the harness is the layer the FDE changes on site. Under the codebase contract a harness change ships by pull request with the customer's eval rows attached, so which config is a question the review can answer with a diff it already has.

Why is a probe not enough to ship?

Forty calls are not a distribution. When I ran one AI feature against 200 real cases split into named slices, the average was fine and the long-input slice, about a fifth of real traffic, failed outright and would have reached customers. The probe picks the candidate. The sliced eval decides whether it ships.

Architecture Review: Pick the Model or the Harness?

Should an architecture review pick the model or the harness?

THE SHORT ANSWER

The harness, and the model once. Lesson 6 of Becoming Forward Deployed Engineer (October 17, 2026) opens on a review asking which model, with two candidates three points apart on a leaderboard that measured a different task; its fix is five probes at ten trials each on the customer's own documents. The same week, Vivek Trivedy at LangChain moved one model 13.7 points on Terminal Bench 2.0 by changing only the harness, Google's Harness^2 reported double-digit gains with the model never retrained, and UC Berkeley's HarnessTax had the same model within about a point across three harnesses at twice the cost in one. Run the probes per harness configuration, with the diff attached, and keep a 200-row sliced eval as the ship gate, because forty calls cannot see a slice that is a fifth of traffic.

The model is the knob everyone in the room can name, and in every measurement that landed this week it was the one that moved the fewest points. The lesson's instrument is right. Point it at the thing the FDE actually changes.

The four numbers on one line, the two-instrument rule, and the version column neither instrument can see are in Which Model Is the Smallest Knob in the Architecture Review. The contract that makes a config change a diff with rows attached is Same Model, Twice the Bill: The FDE Codebase Contract, and the sliced eval is in The Eval That Caught What the Demo Did Not.

Last reviewed 2026-10-19 · 1 min read

Should an architecture review pick the model or the harness?

SOURCES

THE LONG VERSION

RELATED ANSWERS