How do you weight a retrieval bake-off by the questions customers actually ask?

Answered by Falk Gottlob · Reviewed · 1 min read

THE SHORT ANSWER

Pull the last 200 questions people actually typed into the search you are replacing, label each by kind, and use that proportion as the weight on every row of the table. Lesson 12 of Becoming Forward Deployed Engineer (November 12, 2026) shows why: on 36 golden questions pgvector beat the keyword index 69.4% to 63.9%, but plain-language questions went from 23.5% to 82.4% while claim-number lookups fell from 100% to 57.9%, and the public section never says how often adjusters ask each kind. A classifier of mine scored 89% held out and 61% in production because the test set shared the training set's skew. Report per kind, sign a floor per kind with the identifier floor at the 100% they have today, and re-run on production questions at week six.

Weight it by the log, not by the golden set. Label the last 200 real questions by kind, use that proportion on every row, report per kind, and sign a floor per kind, with the identifier floor at whatever the existing search scores today.

The Lesson 12 numbers, the 36-question split, and the one-week drop are in Lesson 12 Beat Keyword Search by 5.5 Points and Broke Claim Lookups. Split the Questions Before the Store.. The slice and the date, from the same course's Lesson 09, are in Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date., and why the rows come from the customer is in The FDE Writes the Eval: Almost Right Has a Number.

SOURCES

THE LONG VERSION

RELATED ANSWERS