Does a bigger context window fix a retrieval-backed assistant?

THE SHORT ANSWER

Not on its own, and the sweep that proves it is six minutes long. Lesson 10 of Becoming Forward Deployed Engineer ran ten golden cases at four context budgets: accuracy 50, 50, 60, 60; right document present 60, 60, 70, 90; documents per question 4.8 to 36.5; time per question 3.91 to 19.25 seconds. The window grew about threefold, the right document arrived half again as often, and the model converted none of it, so from 1,600 to 3,200 tokens the same answers cost twice as much. The lesson reads the gap against the oracle ceiling as the model's problem. My internal docs agent failed in exactly that gap with the fact in the window, because the 2021 version of the same page was in the window too, and the oracle counts that as a hit. Before raising the budget, add two columns to the sweep: cited doc, which document the answer drew on, and twin present, whether the window held two versions of the same record. They split the gap into distraction, stale twin, and model, three different fixes. And the budget step is a cost-bearing key, so a two-times bill for zero accuracy is a re-quote to the budget owner before it ships.

Not on its own. Lesson 10's own sweep tripled the window, got the right document into it 90% of the time, and answered 60%. The gap is where my docs agent died, next to the stale twin of the page it needed. Add the cited doc and twin columns before you blame the model, and treat the budget step as a re-quote.

The sweep, the two columns, and the worked example are in Lesson 10 Tripled the Window and Accuracy Held at 60%. Add the Cited Doc and the Twin Column.. The oracle and the version column come from Run the Oracle Before You Swap the Model, and the cost-bearing key list from Salesforce, Snowflake, and Atlassian Shipped the Agent Budget Cap. Kill It as the Control..

SOURCES

THE LONG VERSION

RELATED ANSWERS

Last reviewed 2026-11-04 · 1 min read