Which of LangChain's three eval-driven steps is the PM's?

Answered by Falk Gottlob · Reviewed · 1 min read

THE SHORT ANSWER

Step one, and the reading in step three. As Divya Santhanam describes LangChain's eval-driven development on November 9, 2026, the loop is: write a short spec of what good looks like for one behavior, write the eval before you build and confirm it fails, then make the change and compare results row by row. The spec and the examples are the PM's work, because they say what the product should do and which cases matter. Engineering owns the harness that runs them. My prediction: by the end of 2027, at AI-native software companies, most rows in a product area's eval file will have been written by people outside engineering.

The short version: step one is the PM's, and so is the reading in step three. Divya Santhanam's guide for LangChain opens the loop with a short spec of what good looks like for one behavior, then an eval written before the build and confirmed to fail, then the change and a comparison. The spec and the examples are product work. The harness is engineering's.

The reading matters as much as the writing. In her worked example the average score went up while one case broke, and nothing on a dashboard would have shown it. Catching it takes someone who reads each row and knows which one a customer would care about. That's the job the PM's spec is turning into: a definition of good that runs on every build.

The rows also come from design, product marketing, and the field, which is why one product area needs one eval file and one owner. The full argument is in part 3 of The PM Transition, every part is on the series hub, and the whole system is The Product OS.

SOURCES

THE LONG VERSION

RELATED ANSWERS