Lesson 11 Lost 83 Documents in 26 Files. Count the Questions Too, and Keep the Copies.

Lesson 11 of Becoming Forward Deployed Engineer starts where retrieval demos never do: with the files the pipeline could not read. This is an Agent Drop: the manifest I would hand a customer on top of what the lesson builds.

The short version

• Lesson 11, published November 8, 2026, gives the course's insurer, Meridian Mutual, claim files as PDFs from four systems over eleven years, some two-column, some scans with no text layer, seven behind a password nobody remembers setting.
• Its public section triages every file with a verdict and a reason, finds 26 unreadable, and reports the loss as 83 documents, not 26 files, after subtracting four near-duplicate copies.
• It then fixes the denominator at 42 questions before scoring four chunking strategies, because its first scorer let fixed-size chunking score 100% on seven questions. Two changes.
• The lesson can subtract the near-duplicates because it generated them; in a customer's folder an unread copy stays in the loss, and my docs agent lost on a current page and its 2021 twin with both readable.
• And the loss gets counted a third time, in the customer's questions. My own scan this morning reported 0 broken on a run that opened 75 of 149 sources. The drop is a manifest with one row per unreadable file: verdict, reason, documents lost, questions lost, who can unblock it, and the date they were asked.

What the lesson builds

The course runs one forward deployed engagement, at an insurer it calls Meridian Mutual, a lesson at a time. I have written about two of them: Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date. on Lesson 09 and Lesson 10 Tripled the Window and Accuracy Held at 60%. Add the Cited Doc and the Twin Column. on Lesson 10. Lesson 11 is paid, and what follows is from its public section. The results table sits behind the paywall and I did not read it.

The setup: Meridian's claim files are PDFs from four systems over eleven years. Some are two columns. Some are photographs of paper with no text in them. Seven are password protected "by a process nobody remembers setting up." The lesson's opening claim is the one I would put on a wall: "The pipeline you write will read most of them and say nothing about the rest. That silence is the defect." The customer's first question in the review, it says, is not about accuracy. It is "did you look at all of it."

So the lesson renders its 200 claims and 750 documents into 204 PDFs, each with a shape that breaks a different part of the pipeline, and triages them. Every file is read once and gets a verdict with a reason: encrypted, no text layer, too little text, no section markers. The too-little-text branch is the careful one. A scanned photo page with a caption yields about eleven characters, and eleven characters "looks like a document to any code that only checks for empty."

Twenty-six files come back unreadable. Then two moves most retrieval tutorials skip.

It counts the loss in documents, not files. The 26 files hold 83 lost documents, and the lesson's reason is blunt: counting files instead of documents is the version of the number a customer catches you on.

And it fixes the denominator. Four chunking strategies are scored on the same 42 questions from an earlier lesson. The first version of the scorer skipped any question whose answering document no chunk claimed. Fixed-size chunking scored 100%, on seven questions out of 42, and looked like the winner. The rule the lesson draws: the denominator is fixed before any strategy runs.

Its subtitle says the chunk size you took from a blog post is the worst of the four options by 44 points. I believe it. I cannot show you the table.

The copy is not free

One line I would change. "Not every unreadable file costs you anything." Four of the 26 are near-duplicate copies, the same claim file with one figure changed, and since every document in them is reachable in the original, the lesson subtracts them. That is how 26 files becomes 83 documents and not more.

The lesson can do that because it wrote the files. It knows the four are copies and it knows which figure moved.

In a customer's folder you know neither. An unreadable file is a filename and a size. You cannot tell it is a copy of something you did read. And if it is a copy with one figure changed, then the changed figure is the reason the copy exists. Usually somebody corrected an amount or a date and saved a second file. That figure is the question a claims handler will ask.

I lost on exactly this pair with both versions readable. My internal docs agent found the page and the 2021 version of the page, gave them equal weight, and answered from the wrong one. That was the twin column I added to Lesson 10's sweep. Lesson 11 is where the twin first shows up, at ingestion, and an unreadable twin is worse than a readable one, because you cannot even diff it.

So two verdicts change. An unreadable file stays in the loss until someone reads it, copy or not. And a readable near-duplicate gets its own verdict, with the field that differs written beside it.

Count a third time

Files, then documents. The lesson stops there in the part I could read. The unit the customer cares about is one further on.

The lesson already has a fixed set of 42 questions with known answering documents. So the loss has a third count: of those 42, how many have their answer in the unreadable pile. That is the real reply to "did you look at all of it." Not 26 files. Not 83 documents. Some number of the questions you told me mattered, which this system cannot answer today, and here is why for each.

It is also the count that makes the AI Product Management point for me: the model work got cheap, and what is left is the unglamorous judgment about what the system does not know.

The seven passwords are an ask

Seven files sit behind a password "by a process nobody remembers setting up." In the lesson that is a verdict. In an engagement it is a request to the customer, and requests need the four fields from Agents Prompt People Now, Says Arca. Put the Default on Silence in the Ask.: an owner, the evidence, what happens if nobody answers, and a date it expires. Without those, the seven files are still locked at go-live and nobody decided that.

My own scan

I ran the lesson on the pipeline closest to hand, the creator scan that feeds this blog. This morning's report opens: 78 relevant posts from 30 of 149 sources, 6 silent, 12 published only off-thesis, 76 need the browser, 0 broken.

The run opened two of the 76. One loaded. One showed a signup form. The other 74 are LinkedIn profiles it never opened.

So the run looked at 75 of 149 sources and reported zero failures. Both statements are true, and "0 broken" is the one in the header's last position, where the eye lands. The scan does have a triage, a verdict per source. What it does not have is the lesson's second count. It knows 74 sources went unread. It does not know how many posts that is. Half the roster by count could be a tenth of the week's writing or most of it.

And Lesson 11 itself is in this post on a partial read. I said so at the top, which is the same rule applied to me.

The drop

One row per file the pipeline could not fully use.

• Verdict and reason. The lesson's four, plus "near-duplicate, differs in" for readable twins.
• Identifiers kept. Enough to find the file again and to price what it cost.
• Twin of. The file it may be a copy of, or blank if unknown. Blank does not reduce the loss.
• Documents lost.
• Questions lost, out of the fixed set.
• Who can unblock it, on the customer's side, by name.
• Date asked, and the default if nobody answers.

Then one line on the first page of the review, built from the rows: read N of M files, D documents and Q of 42 questions out of reach, K rows waiting on you.

The rest of the series is in The FDE Transition, and The FDE Writes the Eval: Almost Right Has a Number is why the question set exists before the pipeline does.

This week, open the last ingestion run you trust and find the number of files it skipped. If the log does not have one, that is your first row.

Related answer: How do you report the documents a retrieval pipeline could not read?

Sources: "Lesson 11: Documents That Fight Back," Becoming Forward Deployed Engineer, November 8, 2026, public section only; every figure and quote from the lesson is from that section, the 44 points is from its subtitle, and the results behind the paywall were not read. Falk Gottlob, "Lesson 10 Tripled the Window and Accuracy Held at 60%. Add the Cited Doc and the Twin Column." (the docs agent and the 2021 page) and "Agents Prompt People Now, Says Arca. Put the Default on Silence in the Ask.." The 149, 76, 75, and 74 are from the creator scan digest in this repository dated November 9, 2026.