LangChain's Eval Loop Starts With a Spec. For PMs, the Spec Is an Eval Now.

LangChain's eval-driven development starts with a spec of good and an eval written before the build. That's the PM's artifact for the next five years.

Falk Gottlob · · 9 min read

Product-green Falkster cover: a small figure in a red cap feeds a long paper blueprint through a row of five turnstile gates, four cream gates with lamps lit above them and the fifth gate dark, the paper buckling against it.
3/3
SeriesThe PM Transition

On November 9, LangChain published a guide to eval-driven development, written by Divya Santhanam. LangChain sells the tooling engineers use to test agents, so I expected a post about harnesses and judges.

Step one is a spec.

In her loop the work starts with a short written definition of what good looks like for one behavior: when it applies, what the agent should and shouldn't do, and what has to keep working. Step two is to write the eval before anyone builds. Step three is to make the change and compare.

Read that as a PM. The company whose business is the test harness just said the job starts with a product definition, and that the definition has to be scoreable, pass or fail. The spec isn't going away over the next few years. It's turning into something that runs.

This is part 3 of The PM Transition. Part 2 said the PM's day becomes a queue of decisions made against a bar. This part is about the bar.

The short version

  • The PM's spec is becoming an eval: a definition of good written as examples and checks that run on every build, each one passing or failing.
  • LangChain's Divya Santhanam laid out the loop on November 9, 2026, and it opens with a short spec for one behavior, then an eval written before the build and confirmed to fail, then the change and a comparison read row by row.
  • Her worked example shows why the reading is product work, because the average went up while one case broke, and only someone who knows which case matters would catch it.
  • Jayesh Govindarajan's refund example, as Aakash Gupta reports it, shows why prose can't do the job: a policy doc changed, no code changed, and the refunds came out too big. Starting is small, and Lenny Rachitsky's episode with Tulsee Doshi of Google's Gemini team spends a chapter on the first eval.
  • The rows come from design, product marketing, and the field as well as from the PM, so a product area needs one eval file with one owner, inside one function under the CPO.

A spec you can run

Santhanam's loop is worth following step by step, because it's a product process with a test runner attached.

Each behavior gets a few lines. Then examples, ideally pulled from real runs, plus the edge cases. Then a scorer, which is code where the check is deterministic and a model judge where it isn't. And then you run the whole thing before you change anything. Her instruction: "Write it first and confirm it fails on the current broken version."

That one line retires most of what I used to put in a requirements doc. A row that fails today and passes after the change is the requirement, the acceptance test, and the proof it shipped, all in one object. A paragraph of prose can't fail. So nobody knows whether it's been met, and an agent has nothing to build toward.

She notes that the per-behavior requirements add up to a PRD for the agent. Fine by me. Call it whatever you like, as long as every line of it can be scored. I've written up the mechanics in The Eval Is The Spec and in a five-row template, so I won't repeat them. What's new is who's saying it. The harness vendor now opens its own method with the PM's work.

The mistakes are product mistakes

Her worked example is small and it's the part I'd show every PM.

A team runs an internal docs agent with two tools, one to search the docs and one to search the web. They add a third that files IT tickets. They rerun the eval. In her words: "The average score went up, but one case broke." Someone asking how to connect to the VPN now gets a ticket filed for them. They wanted the answer from the docs. The cause was a tool description written too broadly.

A dashboard would have shown green. You only catch that by reading the rows, and by knowing that a how-to question answered with a ticket is a worse product. Nobody learns that from a trace.

Then she lists five common mistakes: defining success too loosely, writing the eval after the fix, only checking the average score, adding evals you don't need, and trusting an unchecked judge. I count one that's mostly engineering, the judge that needs calibrating against human scores, and even there the human scores are somebody's judgment. The other four are decisions about what good means and which cases matter. If the PM isn't in the eval file, those decisions get made by whoever is nearest the keyboard.

Jayesh Govindarajan, who runs Salesforce's Agentforce org, gave Aakash Gupta the example that explains why prose can't carry this. An agent answers refund questions from a policy doc. Someone rewrites the doc, the update isn't clear, the code doesn't change, and the refunds come out too big.

Nothing shipped. There was no pull request to review. The behavior moved because a document moved, and a written spec has no way to notice. Ten refund rows that run every night do. Gupta's line for PMs is that "generic evals really don't cut it," and this is why. The rows have to come from someone who knows what the right refund is.

Small is fine, as long as it's yours

On November 8, Lenny Rachitsky published an episode with Tulsee Doshi, who leads product for Google's Gemini models. The show notes list a chapter on how to build your first eval, one on getting better at judging a model's vibes, and one on evaluating the full experience, not just the model.

I've read the show notes and not the transcript, which sits behind the paywall. Scuttlebutt's summary of the episode reports that a first eval can start with five prompts, and that a 20-prompt mini eval is already powerful. Treat those numbers as secondhand. The direction matches everything else this week, and it removes the usual excuse. A PM doesn't need eval infrastructure to start. A PM needs five examples and an opinion about each.

The chapter title I'd underline is the one about the full experience. Benchmarks tell you about a model. Salesforce's 1,500-task agent benchmark came with adjustable weights, and I argued last week in kill the global score that the weights pick the winner. Your product's eval is the part nobody else can write for you.

1 case

broke while the average score went up in LangChain's worked example. Reading the rows is the PM job.

Hiring points the same way. In his late 2026 job market report on November 10, Rachitsky shows open PM roles at their highest in nearly four years, up 75% from the March 2023 low on TrueUp's data, with "LLM" now the most mentioned skill in PM job descriptions. I covered the gap between openings and hires already. My read here is narrower. Companies are hiring for the person who can do step one.

One eval file, four authors

So far this sounds like a PM skill. It's also an org question, and that's where most teams will get stuck.

The rows don't all come from the PM. Design's states and repairs arrive as tests, which is the argument in hand off the tests. Product marketing's claim now gets a message eval suite of its own. The forward deployed engineer writes the eval at the customer and hands the rows back.

Leave those as four suites in four orgs and you have four definitions of good, with an agent building toward whichever one ran last. One product area needs one eval file where all four seats can add a row, and one person who decides which rows gate a release. The Product Builder edits that file. Engineering owns the harness that runs it, the split I laid out in who owns the eval harness. And the CPO owns the fact that there's only one, because the eval is where the one outcome number gets translated into behavior an agent can be held to.

That's the PM's piece of the case The Product OS makes as a whole, and the one The CPO Transition argues from the top. One loop needs one definition of good.

The decision for this quarter is plain. Who's allowed to add, change, or delete a row, and who approves it? If the answer is engineers, in a repo the PM can't open, fix that before you buy another tool.

What I expect

These are my predictions, not facts.

By the end of 2027, at AI-native software companies, most rows in a product area's eval file will have been written by people outside engineering: PMs, designers, product marketers, and the field.

By 2028, the spec review is a diff review. The weekly product meeting reads which rows were added, changed, and deleted since last week, and why. Deleting a row will take more justification than adding one, because a deleted row is a promise withdrawn.

One thing to try this week

Pick the next change on your list. Before anyone builds it, write the behavior in four lines: when it applies, what the product should do, what it shouldn't, and what has to keep working. Then write five examples, and make one of them a case that should stay exactly as it is today.

Run all five against the current product. The new-behavior rows should fail. If they pass, either you don't need the change or the rows don't describe it.

After the change ships, read every row, not the average. I'd start with the one that was supposed to stay the same.

Sources: Divya Santhanam, "What Is Eval-Driven Development? How to Improve AI Agents," LangChain, November 9, 2026. Lenny Rachitsky with Tulsee Doshi, "Why PMs are suddenly valuable again," Lenny's Podcast, November 8, 2026, show notes only. Scuttlebutt, "Tulsee Doshi interview on Lenny's Podcast," November 9, 2026, an unsigned summary, for the five-prompt and 20-prompt figures as it reports them. Lenny Rachitsky, "State of the tech job market: late 2026," Lenny's Newsletter, November 10, 2026, free portion, citing TrueUp. Aakash Gupta with Jayesh Govindarajan, "What an AI-Native Product Team Actually Looks Like at Salesforce," Product Growth, November 3, 2026, for the refund example as Gupta reports it.

Part of the running argument on AI Product Management: the job did not get harder, the parts we trained people for got cheap, and what's left is writing down what good means in a form that runs, inside one loop under the CPO.

Related answer: Which of LangChain's three eval-driven steps is the PM's?

Share this post

Frequently asked

What does it mean that the spec is an eval?+

It means the PM's definition of good is written as examples and checks that run on every build, where each one passes or fails. A row that fails today and passes after the change is the requirement, the acceptance test, and the proof it shipped. A paragraph of prose can't fail, so nobody knows whether it's been met, and an agent can't build toward it.

What are the three steps of LangChain's eval-driven development?+

As Divya Santhanam describes them in LangChain's guide of November 9, 2026: define what good looks like for one behavior in a short spec, write the eval before you build and confirm it fails on the current version, then make the change and compare experiments, reading per-example results and not only the average.

Why is reading the rows a PM job and not an engineering job?+

Because the failures that matter are product judgments. In LangChain's worked example a team adds a ticket-filing tool to a docs agent, the average score goes up, and one case breaks: a VPN how-to question now files a ticket. Catching that takes someone who reads each row and knows that a how-to answered with a ticket is a worse product.

How big does a PM's first eval need to be?+

Small. Lenny Rachitsky's November 8 episode with Tulsee Doshi, head of product for Google's Gemini models, has a chapter on building your first eval. I've read the show notes and not the transcript. Scuttlebutt's summary of the episode reports that a first eval can start with five prompts and that a 20-prompt mini eval is already powerful. Treat those numbers as secondhand.

Why does the eval file belong under the CPO?+

Because its rows come from four seats: design's states, product marketing's claim, the field's handback, and the PM's behavior spec. Four suites in four orgs give an agent four definitions of good. One product area needs one eval file and one person who decides which rows gate a release. Engineering owns the harness that runs it, and the CPO owns the fact that there's only one file.

What are Falk's predictions in part 3?+

Two, labeled as predictions. By the end of 2027, at AI-native software companies, most rows in a product area's eval file will have been written by people outside engineering. And by 2028, the spec review is a diff review: the weekly product meeting reads which rows were added, changed, and deleted, and deleting a row takes more justification than adding one.

product managementPMProduct BuilderCPOPM TransitionProduct OSAI PMevals
Helpful?

Keep reading

The short answer

Part of

AI Product Management

About the author

Falk Gottlob

Falk Gottlob

Product Executive · Founder, Falkster.AI

Thirty years shipping product, from Microsoft Research and Adobe to Salesforce, where he grew Quip into what became Slack Canvas. Four startup exits, including a $6.5B healthcare merger and a company Microsoft bought. Four-time Chief Product Officer. Now founder of Falkster.AI, building Heidi, the company brain for AI-native businesses. This notebook is written from inside the build, not above it.

Sign in with LinkedIn to leave a comment.

Sign in with LinkedIn

Also on Medium

Full archive →