Lesson 10 Tripled the Window and Accuracy Held at 60%. Add the Cited Doc and the Twin Column.

Lesson 10 of Becoming Forward Deployed Engineer tripled the context window, measured what it bought, and found the answer was nothing. This is the Agent Drop that goes with it: the same sweep with two more columns, because the gap it found is where my own agent failed.

The short version

Lesson 10, published November 4, 2026, replaces a top-5 document default that nobody chose and that survived seven lessons with a budgeted context assembler: four sources, a placement order for attention and a separate keep priority for eviction, instructions pinned, documents admitted whole or not at all. Its sweep runs ten golden cases through llama3.2:3b at four budgets. Accuracy 50, 50, 60, 60. Right document present 60, 60, 70, 90. Documents per question 4.8 to 36.5. Seconds per question 3.91 to 19.25. The oracle ceiling, Lesson 03's fake model that answers whenever the fact is in the prompt, goes 60, 60, 70, 90. The lesson's reading is that the window grew about threefold, the right document arrived half again as often, and the model converted none of it. My internal docs agent lost in exactly that gap: it found the page and the 2021 version of the page, gave both equal weight, and answered from the wrong one. The oracle would have counted a hit. So the drop is two columns the lesson leaves out, cited doc and twin present, which split the 30-point gap at 3,200 tokens into distraction, stale twin, and model, three different fixes. And the budget step from 1,600 to 3,200 is a two-times re-quote for zero accuracy, which belongs on the cost-bearing key list before it ships.

The default nobody chose

Since lesson 3, Meridian Mutual's answerer has sent the top five documents to the model. The lesson says it plainly: that number was a first-version default, nobody chose it, and it persisted through seven lessons. The new model has a 128,000-token window, and the question is what to do with it. The lesson's answer is to treat filling the window as a design decision that deserves measurement, since every token is paid for whether or not it helps.

The assembler it builds has two rules where most have one. A placement order decides what the model attends to: instructions, then documents, then tool results, then memory. A keep priority decides what survives when space runs out: instructions never, documents next, then tool results, then memory last. The lesson's argument for keeping them separate is that the item you place first is not the item you drop last, and one sort cannot say both. Instructions are pinned; if they do not fit, the assembler sets over_budget and the caller decides, because dropping the instructions produces a different product, one where the model no longer knows it may say "I don't know."

Documents go in whole or not at all. The lesson's reason is the one that matters for what follows: truncation is worse than dropping, because the key sentence, a reserve amount in an adjuster's note, can be cut while the document still looks present. Retrieval metrics count it as a hit. The answer is wrong and the dashboard is green.

The sweep, and the ceiling over it

The worked example is the question "What is the reserve amount on claim C-1062?" At a 400-token budget, 328 tokens were used, instructions and four documents kept, eight evicted. At 1,600 tokens, 955 used, all twelve candidates kept. In both runs the answering document sat first after the instructions. The extra 627 tokens bought eight more adjuster notes, none about C-1062.

Then the sweep: ten of the 42 golden cases, taken at a stride, five identifier lookups and five plain-language questions, through llama3.2:3b at four budgets.

At 400 tokens: 50% accuracy, right document present 60% of the time, 4.8 documents per question, 3.91 seconds. At 800: 50%, 60%, 10.7 documents, 4.54 seconds. At 1,600: 60%, 70%, 22.3 documents, 9.09 seconds. At 3,200: 60%, 90%, 36.5 documents, 19.25 seconds.

Over it sits the oracle from lesson 3, the fake model that answers correctly whenever the fact is in the prompt and never reasons. Its score is the ceiling: 60, 60, 70, 90. At 400 tokens the model scored 50 under a ceiling of 60. At 3,200 it scored 60 under a ceiling of 90. The lesson's sentence is that the window grew about threefold and the right document arrived half again as often, yet the model converted none of the extra retrieval into correct answers. From 1,600 to 3,200 the prompt grew by 1,118 tokens per question, time doubled, accuracy held, and the lesson notes that whether the extra context distracts the model or is ignored, the same answers cost twice as much.

Its five takeaways are right as far as they go: placement and eviction are different rules; pin instructions; admit whole items only; make the candidate list larger than the budget so you measure the budget rather than top-k; and read accuracy against the ceiling, because a 60% model under a 90% ceiling has a different problem from one under a 60% ceiling.

The gap is where my agent died

Run the Oracle Before You Swap the Model added a column to Lesson 03's oracle, and Lesson 10 is the case for it. Number five in 10 AI Agents I Built That Failed. The Honest Retrospective. was an internal docs agent, classic retrieval with citations. Retrieval worked. It found the page. It also found the 2021 version of the page, gave the two equal weight, and answered confidently from the wrong one. Two HR incidents traced back to it.

Run that case through Lesson 10's oracle. The fact was in the window. The oracle scores a hit. The model answers wrong. The gap between accuracy and ceiling grows by one case, and the lesson's reading files it as the model's problem, distracted or ignoring. It was neither. The model did exactly what a model does with two documents that answer the same question differently and no signal about which one is current.

Lesson 10's own example shows where the twin would sit. The extra 627 tokens at 1,600 bought eight more adjuster notes about other claims. In a window with 36 documents in it, the odds that one of them is a second note on the same claim are not the odds in a window with five. The lesson measures how often the right document is present. It does not measure how often the wrong version of the right document is present too, because the oracle cannot see that.

The drop: two columns

Run the sweep the lesson ships, with two columns added to the table. Both are cheap, and both come from data the harness already has.

Cited doc. For each answer, which document it drew on, matched against the document that holds the expected fact. The answerer already cites; the sweep already knows the answering document. The column is a match or a miss. It makes two cases visible that accuracy hides: a correct answer from the wrong document, which is luck, and a wrong answer from the right document, which is the model.

Twin present. For each case, whether the assembled window held two or more documents that answer the question differently. This is step four of the oracle post, the manual one that hurts, done per budget instead of once. Old policy versions, a superseded price list, the note from before the reserve was changed. At 400 tokens with 4.8 documents it is rare. At 3,200 with 36.5 it is the thing to count.

With those two, the 30-point gap at 3,200 splits three ways. Cases where the twin was present and the cited doc was the wrong version are stale twin, and the fix is a version signal or an archive policy, not a model. Cases where the cited doc was the right one and the answer was still wrong are the model. Cases where the cited doc was an unrelated note are distraction, and the fix is the budget. Three shares, three owners, before anyone proposes a bigger model.

The budget is a cost-bearing key

Salesforce, Snowflake, and Atlassian Shipped the Agent Budget Cap. Kill It as the Control. put context size on the short list of settings that move the price per task: model, top-k, retries, context size, tool fan-out, schedule. Lesson 10's fourth takeaway, measure the budget rather than top-k, is the same list from the other side.

Its own table is the re-quote. From 1,600 to 3,200 tokens: 1,118 more tokens per question, 9.09 seconds to 19.25, accuracy flat. That is a two-times re-quote per completed task for nothing, and the sweep found it in six minutes. The step that follows is to route it. The budget change goes to the name on the cost line before it ships, written per completed task, with the sweep's row attached. The lesson made the cost a number instead of an assumption. The re-quote makes it a decision instead of a default, which is where the whole lesson started.

Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date. adds the last line: the sweep has a date. The twin column changes as the corpus ages, and a window that held no twins in November holds them by March. Re-run the sweep on a schedule, per budget, and keep the date on the row.

Under AI Product Management the claim is that AI collapsed the cost of the parts PMs were trained for and left the parts nobody trained for. Deciding what the model gets to see is one of those. The lesson makes it a measured decision. The two columns make it one you can assign.

This week, if you run a retrieval-backed assistant: take ten golden cases, run them at your current budget and at twice it, and for each answer write down which document it cited and whether a second version of that document was in the window. If the twin column is not empty, the next fix is not the model.

Related answer: Does a bigger context window fix a retrieval-backed assistant?

Sources: Becoming Forward Deployed Engineer, "Lesson 10: Deciding What the Model Gets to See," November 4, 2026, read in full; every budget, accuracy, retrieval, document, time, and oracle figure is as the lesson reports it, and the author is unnamed on the publication. Falk Gottlob, "Run the Oracle Before You Swap the Model" (October 9, 2026), "10 AI Agents I Built That Failed. The Honest Retrospective.," "Salesforce, Snowflake, and Atlassian Shipped the Agent Budget Cap. Kill It as the Control." (November 3, 2026), and "Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date." (October 30, 2026).