Lost in the Middle Is an Argument for Retrieving Less

Everyone quotes the U-shaped curve from the lost in the middle paper. Almost nobody quotes the number underneath it, and that number changes what you should do.

GPT-3.5-Turbo scored 52.9% when the answer sat 10th out of 20 documents. With no documents at all, the same model scored 56.1%. So the retrieval step left it worse off than memory.

Key takeaways#

• Middle-position accuracy fell below the closed-book baseline: 52.9% against 56.1%.
• One correct document alone scored 88.3%, so precision buys far more than volume.
• Past 20 documents, more retrieval bought about 1.5%.
• A 16K model tracked its 4K version almost exactly, so a bigger window fixed nothing.
• The reordering trick most posts recommend is not the one the paper validated.

What does lost in the middle actually measure?#

It measures position, not content. The team moved a single correct document through every slot in a list of distractors and re-asked the same question each time. Accuracy was highest at the start and end of the list, and lowest in the middle, which draws a U.

So the input never changed. Only the order did.

That design matters when you read the result. The model had the answer in front of it every single time.

The number that reframes it#

Read the U-curve alone and it suggests a layout problem. Add the baselines and it becomes a quantity problem.

Three figures from the same model and dataset settle it. Closed-book, meaning no documents, scored 56.1%. The oracle setting, meaning only the one correct document, scored 88.3%. Twenty documents with the answer in the middle scored 52.9%.

Line those up and the ranking is uncomfortable. One good document beats twenty documents by 35.4 points. Twenty documents lose to zero documents by 3.2.

The authors say it plainly: performance in the 20- and 30-document settings is "lower than performance without any input documents". Nineteen distractors cost more than the correct document gave.

Why more documents stop helping#

They also ran the practical version of this, wiring a real retriever to a reader and sweeping the number of documents.

Their finding is quotable and rarely quoted. "Model performance saturates long before retriever performance levels off, indicating that readers are not effectively using the extra context."

Concretely, going past 20 documents bought about 1.5% for GPT-3.5-Turbo and about 1% for claude-1.3. Your recall curve keeps climbing, so the dashboard looks fine. The answer quality has already stopped moving.

That gap between the two curves is where most RAG tuning effort goes to die. Improving recall from 85% to 92% feels like progress, and the reader cannot spend it.

A worked example#

Take a typical setup: 20 chunks of about 500 tokens each, so roughly 10,000 tokens of context per question.

Now price the two options. Cutting to the top 5 chunks saves 7,500 tokens per question, which is a 75% cut in retrieved context on every single call. Against that, the paper's own sweep says the documents you dropped were worth around 1.5%.

Then look at what precision is worth. Moving from twenty documents to the single right one was worth 35.4 points in the same experiment.

So the arithmetic points one way. Spending on a reranker that promotes the right chunk into the top 5 targets a 35-point prize. Spending on a retriever that widens k from 20 to 50 targets a 1.5-point one, and pays for it in tokens and latency on every request.

This is why reranking beats retrieving more, and it is also why fusing several retrievers helps: better candidates at small k. We covered one way to do that in reciprocal rank fusion.

Reranking is not free either, though, and I should price it rather than wave at it. A cross-encoder scores every query and document pair together, so 50 candidates means 50 forward passes before the reader even starts. That is real latency on the critical path, plus either a model to host or a per-call bill.

Weigh it against what it replaces. Those 50 scoring passes are short, since they read one chunk each, while the 7,500 tokens you removed were paid by a much larger model on every request. For most setups the reranker is cheaper than the context it saves, and it arrives as one added step rather than a rewrite.

Choosing k, with the trade written down#

Small k is not always right. Here is how I decide, with the criteria that actually move the answer.

Your situation	Reader budget	Why
Reranker in place, factoid questions	Top 3–5	Precision is worth 35 points, volume about 1.5
No reranker yet, weak embeddings	Top 10–15	You are buying recall insurance until you have one
Multi-hop or comparison questions	Top 8–12	Several documents are the task, not noise
Short, redundant, templated chunks	Top 15–20	Distractors overlap, so each adds little confusion
Latency-critical path	Top 3	Fewer tokens is the cheapest speed-up you have

The column that decides most cases is the middle one, and notice it is a reader budget. Your retriever can still fetch 50 candidates in every row. The question is only how many survive to the prompt.

I picked these criteria because each one changes the cost of a distractor. A reranker lowers the odds a distractor outranks the answer. Redundant chunks make distractors harmless. Multi-hop questions turn extra documents from noise into evidence.

Count documents, or count tokens?#

One caveat before you copy any k from a paper, including this one.

Their experiment varied the number of Wikipedia chunks while holding chunk type fixed. Your chunks are probably not that size. Twenty chunks of 100 tokens and twenty chunks of 1,000 tokens are the same k and a tenfold difference in context.

So treat their document count as a stand-in for context length rather than a number to copy. If your chunks run long, you may hit the same wall at k=6.

A practical move is to record tokens alongside k in your evaluation, and plot accuracy against tokens. That curve transfers between setups, and a k curve does not.

The fix people recommend, and the one that was tested#

Standard advice is to reorder your chunks so the best one lands first or last. It follows logically from the U-curve, and the paper did not test it.

What the authors did test is query-aware contextualization: putting the question both before and after the documents. In key-value retrieval it worked dramatically well.

In multi-document question answering it mostly did not. The paper reports that it "improves performance when the relevant information is located at the very beginning of the input context, but slightly decreases performance in other settings".

Read that carefully before shipping it. A change that helps in one position and hurts in the others is not a fix, and averaged over real traffic it can be a small loss. Reordering may still help you, but treat it as a hypothesis to measure rather than a result to cite.

Longer windows did not fix it#

A tempting response is to wait for bigger context windows. That was already tested, and it failed in 2023.

The paper compared GPT-3.5-Turbo against its 16K variant, and reports the curves as "nearly superimposed". Claude-1.3 and Claude-1.3-100k behaved the same way. A model with four times the window used the overlapping part no better.

The lesson generalises past those specific models. Window size is a capacity limit, and using the middle of a context well is a separate skill. If you want the mechanics of why the limit exists at all, we wrote about context windows separately.

The failure mode, and how to detect it#

This failure is dangerous because it produces confident wrong answers. Your retrieval metrics stay green, because the right chunk really was retrieved. Only the reader dropped it.

So recall cannot detect this. You need a position test instead, and the paper hands you the protocol.

Build a small set of questions where you know the answer chunk. Then run each question several times, moving that chunk through first, middle and last position while keeping the other chunks fixed. Score accuracy per position.

A flat line means your model and your k are fine. A U means you are paying for context the reader cannot use. Run the closed-book condition too, with no documents, since that is the baseline that turns an interesting curve into a decision.

Automate it against your own corpus rather than trusting a public benchmark. Chunk size, domain and prompt all move the curve, and only your numbers apply to you.

What retrieving more gets right#

A case for large k is real, and I do not want to wave it away.

High k buys recall insurance. When your embedding model is weak, or a query is phrased unusually, the correct chunk may sit at rank 30, and a top-5 cut throws the answer away entirely. Missing the answer is a worse failure than burying it, because no amount of reader skill recovers it.

Low-traffic services get a pass too. Running a reranker to serve a handful of questions an hour is operational weight for little gain, so a smaller k on its own captures most of the benefit there.

Multi-hop questions also genuinely need several documents at once. There the extra context is not distraction, it is the task, and a small k breaks it by design.

An honest framing is a two-stage split. Retrieve wide to protect recall, then rerank hard and pass only the top few to the model. That way large k serves the retriever, and small k serves the reader, and the filtered vector search work happens where it belongs.

Where this advice is wrong#

The measurements are from 2023. The models tested were GPT-3.5-Turbo, Claude-1.3, MPT-30B and LongChat-13B. Frontier models have improved markedly at long-context recall since, and quoting 52.9% as if it describes a 2026 model would be dishonest. What survives is the method, not the number.

Position effects have shrunk, not vanished. Treat the U as something to measure rather than assume. Running the test above on your own stack costs an afternoon and settles the question for your model, and that is a better answer than either citing a three-year-old figure or ignoring it.

Small k is wrong for some corpora. Where documents are short, highly redundant, or heavily templated, twenty chunks may carry barely more distraction than five. The saturation point is a property of your data, so find yours rather than adopting mine.

The short version#

The lost in the middle result is usually told as a story about ordering. Its own numbers tell a story about volume. Twenty documents with the answer in the middle scored 52.9%, below the 56.1% the model managed with nothing, while the single correct document scored 88.3%. Past twenty documents, more retrieval bought about 1.5%. So retrieve wide, rerank hard, and hand the model few enough documents that it can still use them.

Sources: Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172), the same paper in TACL