
Lesson 12 of Becoming Forward Deployed Engineer, published November 12, opens with the sentence every forward deployed engineer has heard in some form from a customer's security team: "Recommending a new datastore costs six weeks of security review." This is an Agent Drops entry, and the drop is a week of work that belongs before that conversation.
The short version
- Lesson 12 runs a retrieval bake-off on the Postgres the course's insurer already trusts rather than recommending a vector database that costs six weeks of security review: the existing keyword index, pgvector inside that Postgres, sqlite-vec as a file-based store, and a hybrid, against the 36 golden questions whose answers survived Lesson 11.
- Top-5 accuracy: keyword 63.9%, pgvector 69.4%, hybrid 86.1%, so vectors win by 5.5 points. Then it splits the questions by kind, and plain-language questions go from 23.5% to 82.4% while questions quoting a claim number fall from 100% to 57.9%.
- The split is the finding and the 5.5 points are not, and what the public section never says is how often an adjuster asks each kind.
- A classifier of mine scored 89% held out and 61% in production because the test set shared the training set's skew, and a feature of mine passed 200 real cases and failed on a slice that was a fifth of traffic.
- The drop: pull the last 200 real questions, label each identifier or prose, weight the table by that mix, run the three stores per kind on the Postgres they trust, sign a floor per kind, and re-run at week six.
What the lesson did
The setup, in the lesson's words. "Meridian's adjusters ask two kinds of question," ones that quote a claim number and ones that describe an incident, and "Lesson 1 measured the search they have today at 100% on the first kind and 15% on the second." The obvious fix is a vector database, and the obvious fix brings procurement, security review, and operational overhead before anyone knows whether it helps. So the lesson asks the better question: can the Postgres Meridian already runs do it?
The test set is inherited from the lesson before. "Lesson 11 left 178 readable files out of 204, and 36 of the 42 golden questions have their answer in one of them." Three candidates run against those 36, plus a hybrid: the existing keyword index, pgvector as an extension inside the current Postgres, and sqlite-vec as a file-based store. Vectors are normalized so distance and cosine similarity rank the same way.
The results table: keyword 63.9%, which is the section-aware chunking result that won Lesson 11; pgvector 69.4%; sqlite-vec 69.4%; hybrid 86.1%. "Look at the top-5 column on its own and vectors won by 5.5 points."
Then the lesson does what most bake-off tables never do. It splits the 36 by kind. "Plain-language questions went from 23.5% to 82.4%." "Questions quoting a claim number went from 100% to 57.9%," because a vector store retrieves a passage when "the surrounding words feel similar," and "four out of ten times the words around some other claim feel just as similar." That is the last sentence before the paywall, and I am citing nothing past it.
100% to 57.9%
The split is the finding
The instinct to start with the database they already trust is right, and I would add it to the list of things an FDE should say in the first week. But look at what the two numbers in the table do once you split them. Vectors won the aggregate by 5.5 points. On plain-language questions they won by about 59 points. On claim-number questions they lost by 42, from a score of 100.
A bake-off that reports 69.4% against 63.9% has told you the test set's opinion. The split tells you the adjusters' opinion, except for one thing it cannot tell you, because the public section never says it: how many of the 36 are of each kind, and how often an adjuster asks each kind in a working day.
That second number decides everything. If nine of ten real questions quote a claim number, the store that won by 5.5 points is a regression on the floor, and the hybrid's 86.1% is a fact about the composition of 36 golden questions, not about anyone's Tuesday. If nine of ten are plain-language, the aggregate understates the win. Either way the table as published cannot be signed, because the weights are missing.
I have shipped the unweighted table
I wrote about the two missing columns on this course's Lesson 09 in Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date., and both apply here. A classifier of mine scored 89% on held-out data and 61% in production within six weeks, because the held-out set shared the training set's skew; nobody had weighted it by what production would actually send. A feature of mine passed 200 real cases and failed on one slice that was a fifth of traffic, because the 200 underweighted that slice.
Lesson 12's two kinds of question are a slice, and the lesson deserves credit for cutting it. What it has not done, in the part I can read, is weight the slice by the day. That is the one input only the customer has, and getting it is the FDE's job, which is the argument of The FDE Writes the Eval: Almost Right Has a Number: the eval is the deployment artifact, and its rows come from the customer's real cases in the customer's real proportions.
The drop
One week, before anyone books the security review.
- Pull the last 200 questions the adjusters actually typed into today's search. Not golden questions. The log.
- Label each one identifier or prose, the lesson's own two kinds. The proportion you get is the weight on every row of the bake-off table.
- Run keyword, pgvector on the Postgres they already trust, and hybrid, and report per kind, never only the aggregate.
- Write the floor the customer signs per kind. The claim-number floor is the 100% they have today; a store that cannot hold it does not ship, whatever it does to the aggregate. The plain-language floor is a number above the 15% they have today that the customer's experts agree is worth the change.
- Put a date on the table. At week six, re-run it on the production questions from those six weeks, not the original 200, and compare the mix as well as the scores.
Step 4 is where the hybrid earns its place, if it does. The lesson's hybrid scored 86.1% in aggregate, and the public section does not give its per-kind numbers; if it holds claim-number lookups at or near 100% while lifting prose, it is the answer, and the per-kind floor is how you would know. That is the AI Product Management thesis in one table: AI collapsed the cost of running a bake-off, and left the part nobody trained for, which is knowing what to weight it by.
This post is a standalone companion to The FDE Transition, not a numbered part of it.
This week
Open the search log of the system you are about to replace. Take the last 200 queries, label each by kind, and write the two percentages on the first line of your bake-off doc. If you cannot get the log, that is the first conversation to have, before the one about the datastore.
Related answer: How do you weight a retrieval bake-off by the questions customers actually ask?
Sources: "Lesson 12: Start With the Database They Already Trust," Becoming Forward Deployed Engineer (byline AI Engineering, author otherwise unnamed), November 12, 2026, public section read in full; the six weeks, 36, 42, 178, 204, 63.9%, 69.4%, 86.1%, 5.5, 23.5%, 82.4%, 100%, 57.9%, and "four out of ten" are the lesson's, and nothing past the paywall is cited. The 89 and 61, the 200 and the fifth, are my own, from the post linked above.
Frequently asked
What does Lesson 12 of Becoming Forward Deployed Engineer test?+
Whether the insurer's existing Postgres can close the gap on plain-language questions without a new datastore, since recommending one costs six weeks of security review. It runs the existing keyword index, pgvector inside that Postgres, sqlite-vec as a file-based store, and a hybrid against 36 golden questions. Top-5 accuracy in its table: keyword 63.9%, pgvector 69.4%, sqlite-vec 69.4%, hybrid 86.1%.
What happened when the lesson split the questions by kind?+
Plain-language questions went from 23.5% to 82.4%. Questions quoting a claim number went from 100% to 57.9%, because a vector store retrieves on similar surrounding words and, in the lesson's phrase, four out of ten times the words around some other claim feel just as similar. The aggregate 5.5-point win hid a 42-point loss on the kind of question that used to be perfect.
What is missing from the bake-off table?+
The mix. The public section does not say how often adjusters ask each kind of question in a working day. If nine of ten real questions quote a claim number, the store that won the aggregate is a regression on the floor, and the hybrid's 86.1% describes the test set rather than the adjusters' work.
Why weight the eval by the production mix?+
Because an unweighted set shares whatever skew its author had. A classifier of mine scored 89% on held-out data and 61% in production within six weeks for that reason, and a feature of mine passed 200 real cases and failed on one slice that was a fifth of traffic. The mix is the weight, and the last 200 real questions are where it comes from.
What is the one-week drop?+
Pull the last 200 questions adjusters typed into today's search. Label each identifier or prose. Use that mix as the weights on the bake-off table. Run keyword, pgvector on the Postgres they already trust, and hybrid, reported per kind. Write the floor the customer signs per kind, with claim-number lookups not allowed below the 100% they have today. Re-run on production questions at week six.


Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn