The FDE Writes the Eval: Almost Right Has a Number

Aneel Bhusri has one line in the Accenture and Workday announcement that I'd put at the top of every FDE statement of work: "When AI is helping run payroll and close the books, almost right isn't good enough."

He said it on October 13. Nine days later Colin Jarvis published what OpenAI's FDE team does, and the verb that repeats is build, as in built evals. On the semiconductor engagement his engineers "built evals with the engineers to test whether the AI could generate useful tests and correctly diagnose chip failures," and they did it before they built the system. A tax deployment's rental-property automation took "six weeks of engineering and practitioner review to reach 90% precision and recall." When the team extended a rebuilt deployment to the next team, the work was swapping connectors and "building fresh evals (which we also had a skill for)." I wrote about his numbers in a field note the day the post came out. This part is about the artifact under the numbers.

Then the hiring data. On October 23 Akshita Singh at Business Standard reported a Scaler and CyberMedia Research survey of more than 1,000 Indian engineers and 200 enterprise hiring decision-makers. When they evaluate FDE candidates, 51 percent prioritize production deployment and troubleshooting assessments and 24 percent prioritize coding tests. Amar Srivastava of Scaler calls it "one of the most defining engineering careers of this decade."

And the buyers. Lindsay Clark at The Register, October 22: FDEs are "consultants-without-portfolio," expected to build something valuable, at speed, with their employer's technology. The only outcome number in her piece is Marc Benioff's claim from a December earnings call that customers working with Salesforce FDEs deployed its AI 33 percent faster, and she notes nobody has verified it. Gartner's advice to buyers, via Mukul Saha, lists six things a well-governed engagement needs: governance, business value delivery, IP ownership, project co-ownership, knowledge transfer, and an exit strategy. Craig Guarente of Palisade Compliance adds that the engineers "are not even checking licensing."

A CEO says almost right isn't good enough, a lab says its engineers build the test, hiring managers want to test for production, and the buyers get a speed number. The thing that connects all four is the eval. I've treated it as a line item in eleven parts of this series. It gets the part.

This is part 12 of The FDE Transition. Part 11 was when not to hire. This one is for the engineer on site and the three people who sign what the engineer builds.

The short version

The eval is the deployment artifact. Not the demo, not the dashboard, not the deck. One versioned set of the customer's real cases with agreed-correct answers, a target set against the human baseline, and a pass rule a machine can check. The FDE writes it before building, with the customer's experts supplying the labels. It's the statement of work, because done is the row set passing. It's the spine of the handback, because the rows travel to product. It's the independence test at month two, because the customer can add a row and re-run it without you. And it's what the renewal is priced against. Almost right has a number, and the forward deployed engineer is the person who writes it down.

Almost right has a number

Bhusri's line is about payroll, where an error has a name attached and lands on a Friday. Every deployment has a version of it. The question is whether anyone wrote down what almost means before the build started.

The product changes chapter said the eval is the contract, and the eval is the spec has the general case. What's new this week is the order of operations from the one team that published its own. Jarvis's sequence is embed, learn the domain, build the eval with the customer's engineers, then build the system. On the telecom voice case the eval is how the team knew the model was at 60 percent at scoping, and how they could say so to the customer instead of finding out at go-live. The number existed before the work did.

Now hold that next to the buyer's page. Gartner's six criteria are good, and none of them is a test. Benioff's 33 percent is a speed number, and faster only counts if you know what you arrived at. If the eval is the artifact, faster means the rows passed sooner, and the customer can check that without taking the vendor's word. Clark's argument is that buyers need controls over engineers whose employer they're unsure of. The eval is the control that doesn't require trusting anyone. It runs.

Who writes it

The FDE drives it. The customer's experts supply the substance. Saurabh Singh's Evals for Client Work, in the Vibe Engines handbook, is the clearest method I've found for the first week. The job, in his words, is to "get people who disagree with each other to write down what correct means." Two or three of the customer's experts label about 40 real cases independently. Their disagreements get resolved into a written rubric. That rubric becomes the labeling guide, the LLM-judge prompt, and the acceptance language in the statement of work. One document, three uses.

The disagreement is the discovery. I've sat in enough of these sessions to say the eval meeting is where the policy nobody wrote down comes out. Two senior people grade the same case differently, and the reason is a rule that lives in one of their heads. That rule is what the FDE came to find, and it would have surfaced in month four as a reversal if nobody had labeled the row in week one. Part 3 made the handback packet the FDE's main deliverable to product. The rows are the packet's spine.

Lesson 6 of Becoming Forward Deployed Engineer, published October 18, makes the same move one level down. Two models, three leaderboard points apart, and the leaderboard "is a measurement of a different task." The fix is about ninety minutes measuring each model on the customer's own documents: five probes, ten trials each, forty calls per model, and the author says the fifth probe decides the engagement. An eval written by the engineer on site, from the work.

There's a second eval, and it isn't the FDE's. The loop chapter gave the PM the editor's job: read what the field built, decide what generalizes, and write the eval the generalized version has to pass. Two evals, one row set. If the second can't be written from the first one's rows, the first wasn't an eval. It was a demo with a score.

What's in it

Five things, and I'd refuse to call it an eval with fewer. A frozen golden set of the customer's real cases, labeled from the customer's historical decisions where they exist, versioned, with the date the rows were drawn. A target set against the human baseline, not against 100. Singh is right that the invented number is how deployments stall. An error budget for the high-cost mistakes, which is Bhusri's point in operational form: a payroll error and a formatting error aren't the same error, so the eval says which ones are forbidden at any rate. The agreement rate between the judge and the human graders, reported with every result, so nobody mistakes a confident judge for a correct one. And cost and latency limits, because part 4 already learned that the same model can cost twice as much in a different harness.

The date matters more than it looks. I wrote the prompt commit because a classifier of mine scored 89 percent on one row set and 61 percent six weeks later on another, same prompt, same score field. A score without its rows and their date is a number with no claim attached.

Where it goes after

An eval that stays in the FDE's notebook is a test. An eval that moves is an operating model. It goes into the handback packet from part 3. It rides on the harness pull request from part 4, where engineering's obligation is to run every customer's rows on every harness change. It becomes the independence test on the month-two page: a named person at the customer changes a threshold and watches the rows run. And it's written into the buyer's contract as the customer's property, the one thing that lets them switch vendors and know on day one whether the new system is worse. Singh has the sentence for the moment it crosses over: "Once it blocks a merge, it is a control."

Then the line from Jarvis that got the least attention. A skill for writing evals. The first FDE on the semiconductor account wrote the eval by hand, with the customer's engineers. By the time the rebuilt deployment reached the next team, the eval-writing itself was a skill in the harness, next to the connectors. That's the loop closing on the artifact rather than on a feature. The scorecard counts generalization rate as the share of what came back from the field that shipped in the product. I'd count the eval skill as the highest-value item that ever came back, because every deployment after it starts with a test instead of a demo.

What this means in the seat, and at the top

If you're the FDE: write the eval before the kickoff deck. If the customer's experts won't sit for 40 cases, you've learned in week one what would have killed month two, and you can take that to sales under the handshake.

If you run product: the rows are your input, and they arrive before the packet does. A PM who can write the cross-customer eval from one customer's rows has found the generalization. One who can't has found the reason not to.

If you're hiring: half of the hiring managers in the Scaler survey want to assess production work and a quarter still run coding tests. Interview for the eval. Give the candidate twenty real cases and two experts who disagree, and watch what gets written down.

If you're the buyer: Bhusri's standard is yours to enforce. Ask for the row set in the statement of work, with the date and the baseline, and make it the exit test.

One thing to try this week: pull the eval on your live deployment and check five things. Are the rows the customer's real cases? Is the target set against a human baseline? Is there an error that's forbidden outright, at any rate? When were the rows drawn? Can a named person at the customer add one and re-run? Any no is a deployment that's almost right, and you don't know by how much.

Sources: Colin Jarvis, "What FDE means at OpenAI," OpenAI (October 22, 2026) · Accenture, "Accenture Launches Forward Deployed Engineering Program with Workday to Reinvent Work for the Agentic AI Era," Accenture Newsroom (October 13, 2026), with Aneel Bhusri quoted · Lindsay Clark, "Vendors asking forward deployed engineers to make good on trillion-dollar AI bet," The Register (October 22, 2026), with Mukul Saha, Craig Guarente, and Marc Benioff quoted · Akshita Singh, "Only 1% of engineers work in FDE roles despite rising AI demand: Report," Business Standard (October 23, 2026), on the Scaler and CyberMedia Research "India's FDE Opportunity Report 2026," with Amar Srivastava quoted · Saurabh Singh, "Evals for Client Work," Vibe Engines Technical Handbook (July 31, 2026) · Becoming Forward Deployed Engineer, "Lesson 6: Measuring a Model Instead of Trusting a Leaderboard," Substack (October 18, 2026), author unnamed

Part of the running argument on AI Product Management: once building is cheap, the scarce work is deciding what gets built for everyone, and the eval is the one artifact that carries that decision from one customer's building to the roadmap.

Related answer: Who writes the eval in a forward deployed engineering deployment?