Lesson 6 of Becoming Forward Deployed Engineer dropped on Saturday, and its opening scene is one I have sat in more times than I want to count. An architecture review at an insurance customer. One question on the agenda: which model. Somebody printed a leaderboard, and the two candidates are three points apart.
I could only read the preview; the lesson is paid past it and I am not going to pretend to the rest. The preview is enough to argue with, and the argument is not with the lesson. It is with the agenda.
The short version
Lesson 6 of Becoming Forward Deployed Engineer, published October 17, 2026, opens on an architecture review that asked which model, with two candidates three points apart on a leaderboard that measured a different task. Its fix is ninety minutes of your own probes, five of them at ten trials each, on the customer's documents. The same week, three other measurements landed: Vivek Trivedy at LangChain moved one model 13.7 points on Terminal Bench 2.0 by changing only the harness, Google's Harness^2 (via Basia Kubicka) edits the harness mid-task for double-digit gains with the model never retrained, and UC Berkeley's HarnessTax had already shown the same model within about a point across three harnesses at twice the cost in one of them. Three points from the model, thirteen from the harness, two times the bill from the harness. The review is asking the question that moves the fewest points. Run the probes per harness configuration, keep the 200-row sliced eval as the ship gate, and ask the model question once.
What the preview says
The customer is an insurer. The review asked which model. The lesson's answer, in its own words: "Three points on what, though. The benchmark that produced them did not extract claim fields from adjuster notes, did not have to say 'I don't know' when a reserve was never set, and did not run on the hardware their CISO will allow. The score is real. It is a measurement of a different task."
The fix it proposes is not a better leaderboard. It is "ninety minutes of your own measurement, on your own documents, against the four or five things this engagement actually depends on." The instrument is a probe: "a small fixed set of prompts with a machine-checkable pass rule. Not a benchmark: it does not ask whether a model is good, only whether it does the specific things your system needs." Five probes, ten trials each. Small on purpose, because "a probe suite you only run once because it takes an afternoon is a document, not a tool."
The subtitle promises that four of the five things you measure will tell you nothing and the fifth will decide the engagement. Which one is behind the paywall, and I am not guessing.
Everything in the preview I would do. Probes on the customer's documents, machine-checkable, small enough to rerun. The lesson is right that the leaderboard measures someone else's task. I want to change what the probes are pointed at.
Three measurements from the same week
The reason is that three other posts in my feed this week measured the other side of the same review, and the numbers do not favour the agenda.
Vivek Trivedy at LangChain took one model, gpt-5.2-codex, from 52.8 to 66.5 on Terminal Bench 2.0, from outside the top 30 to the top 5, and changed only the harness. (The post is from February; the LangChain feed resurfaced it this week, and it reads as current.) The knobs were a system prompt that forces a build-verify loop, context injected about the environment, a middleware that detects edit loops, and a reasoning budget split. Running everything at the highest reasoning setting scored 53.9 because of timeouts; the next level down scored 63.6. Thirteen points and change, model untouched.
Google published Harness^2, which Basia Kubicka broke down on Sunday. The loop watches the agent run, proposes an edit to its prompt, tools, or guardrails, tests the edit, and feeds what it learned into a running playbook for the next task. Ruiyao Xu, Yanfei Chen, and their co-authors report gains of up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench, and 11.8 on LAB, with the underlying model never retrained. Kubicka's line: fix the harness, not the model.
And the one I already built a contract on. In HarnessTax, six researchers at UC Berkeley ran the same models through three coding harnesses. Claude Fable 5 solved 97.8% of attempts in Claude Code and 96.7% in both Codex and Pi, and Claude Code cost about twice what Pi did per attempt. Same Model, Twice the Bill: The FDE Codebase Contract is what I did with that.
Put the four numbers on one line. Three points between the lesson's two models. Thirteen points from LangChain's harness. Double digits from Google's. Two times the bill from Berkeley's.
The review is asking the question that moves the fewest points.
Probe the harness, not the model
So here is the change. Run the lesson's five probes per harness configuration, not per model. Model A in config 1. Model A in config 2. Then, if you must, model B in the better config.
This is not a trick to avoid the model question. It is where the FDE's leverage actually is. Under the codebase contract, the customer layer (prompts, rules, thresholds, connector settings, the customer's eval rows) belongs to the FDE and changes the same day, versioned, diffing against the shipped default in one command. The harness belongs to engineering, and an FDE changes it by pull request with the customer's eval rows and the cost per case attached. Every one of those artifacts is evidence the review could be reading. "Which config" is a question the room can answer with a diff. "Which model" is a question the room answers once, from the probe, and then stops asking.
The ninety minutes the lesson budgets is enough for this. Five probes at ten trials is the lesson's forty calls per model. Two configs is eighty. The leaderboard took longer to print.
Two instruments, not one
The second change is to keep the probe and the eval apart, because they answer different questions and the review tends to collapse them.
A probe answers can it. Five things, machine-checkable, forty calls. It is the right tool for picking a candidate and the wrong tool for deciding to ship, because forty calls are not a distribution.
A sliced eval answers does it, on this customer's traffic. The Eval That Caught What the Demo Did Not is the number I keep for this. We demoed an AI feature on three hand-picked cases and it was flawless. I ran it against 200 real cases, split into named slices: short inputs, long inputs, a second language, ambiguous requests. The average was fine. The long-input slice, about a fifth of what customers actually sent, failed outright, with confidently wrong structured output that looked right and was not. We did not ship. We fixed the slice and shipped a week later on a number.
Forty calls cannot see a fifth. Two hundred rows with named slices can. The probe picks. The sliced eval ships.
And there is a column neither instrument can see. Run the Oracle Before You Swap the Model has it: an internal docs agent of mine retrieved the 2021 page and the 2024 page, weighted them the same, and produced two HR incidents. A probe scores that as a pass, because the fact was in the context. So was its predecessor. Before anyone proposes a bigger model, count the documents in the corpus that answer the same question differently. That number was most of my problem, and no leaderboard has a column for it.
Where this sits
This is a standalone piece next to The FDE Transition, and the closest part is the codebase contract, because the contract is what turns "which config" from an opinion into a diff with rows attached. It belongs to the argument on AI Product Management: the build got cheap, and the scarce work is knowing which part to build. The model is the part everyone can name. The harness is the part that moves.
This week: before your next customer architecture review, pull the agenda. If "model selection" is item one, add "harness configuration" above it, bring the diff between your two configs, and run the five probes on both. Report the spread. If it is bigger than the gap between the two models on the leaderboard, you have your answer, and so does the room.
Related answer: Should an architecture review pick the model or the harness?
Sources: Becoming Forward Deployed Engineer, "Lesson 6: Measuring a Model Instead of Trusting a Leaderboard," Substack, October 17, 2026, the preview above the paywall read and nothing past it cited. Basia Kubicka, LinkedIn post on Harness^2, October 18, 2026, read in full. Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, and Tomas Pfister, "Harness^2: Recursive Agent Harnessing for an Open World," arXiv 2610.14224, October 9, 2026, abstract read. Vivek Trivedy, "Improving Deep Agents with harness engineering," LangChain, February 17, 2026, read in full. Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia, "HarnessTax: How Much Does the Harness Matter for Coding Agents?," UC Berkeley via Arena, September 16, 2026, as cited in the codebase contract post. Falk Gottlob, "Same Model, Twice the Bill: The FDE Codebase Contract," "The Eval That Caught What the Demo Did Not," and "Run the Oracle Before You Swap the Model."
Frequently asked
What does lesson 6 of Becoming Forward Deployed Engineer argue?+
In the preview above its paywall, published October 17, 2026: an insurance customer's architecture review asked which model, with a printed leaderboard putting two candidates three points apart, and the benchmark behind the score never extracted claim fields from adjuster notes, never had to say I don't know when a reserve was unset, and never ran on the hardware the CISO allows. The score is real but measures a different task. The fix is ninety minutes of your own measurement with probes, a small fixed set of prompts with a machine-checkable pass rule, five probes at ten trials each, kept small so the suite gets rerun. The body of the lesson is paid and I do not cite it.
What is a probe, and how is it different from a benchmark?+
As the lesson defines it, a probe is a small fixed set of prompts with a machine-checkable pass rule. It does not ask whether a model is good, only whether it does the specific things your system needs. A benchmark ranks models on someone else's task. A probe checks your task, on your documents, in your environment.
Why is the model the smallest knob?+
Because the measurements that landed the same week all moved more points by changing the harness than the lesson's two candidates differed by. Vivek Trivedy at LangChain took gpt-5.2-codex from 52.8 to 66.5 on Terminal Bench 2.0 by changing only the harness. Google's Harness^2 reports gains of up to 11.5 points on WorkBuddy-Bench and 11.8 on LAB with the model never retrained. UC Berkeley's HarnessTax found Claude Fable 5 solving 97.8% of attempts in Claude Code and 96.7% in Codex and Pi, with Claude Code costing about twice Pi per attempt. Three points from the model, thirteen from the harness, two times the bill from the harness.
What should the architecture review measure instead?+
Run the probes per harness configuration rather than per model: the same model in config 1 and config 2, with the diff attached. The harness is the layer an FDE changes on site, and under the codebase contract a harness change ships by pull request with the customer's eval rows, so which config is a question the review can answer with evidence it already produces. Ask which model once, then move on.
Can a probe suite replace the sliced eval?+
No, they are two instruments. A probe answers can it, on five things, in forty calls. A sliced eval answers does it, on this customer's traffic. When I ran one AI feature against 200 real cases split into named slices, the average was fine and the long-input slice, about a fifth of real traffic, failed outright and would have reached customers. Forty calls cannot see a fifth. The probe picks the candidate; the sliced eval decides whether it ships.
What can neither instrument see?+
Version. Run the Oracle Before You Swap the Model covers the case from an internal docs agent of mine that retrieved the 2021 page and the 2024 page, weighted them the same, and produced two HR incidents. A probe and an oracle both score that as a hit, because the fact was in the context. So was its predecessor. Count the documents that answer the same question differently before anyone proposes a bigger model.
Comments (0)
Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn