The BM25 Versus Embeddings Argument Measures the Wrong Gap

Every few months the argument comes round again. Keyword search or embeddings, BM25 or vectors, lexical or semantic. Both camps cite benchmarks, and both are arguing about the smaller of the two numbers on the table.

On a 2026 financial retrieval benchmark, switching first stage moved Recall@5 by 0.057. Adding a second stage moved it by 0.172.

Key takeaways#

• BEIR's own best zero-shot result was reranking, not any single retriever.
• On BEIR, cross-encoder reranking scored +11% against BM25, while dense models mostly scored below it.
• A 2026 financial benchmark repeats the shape: two-stage beats every single-stage method.
• First-stage choice bought 0.057 Recall@5 there. The second stage bought 0.172.
• Modern embeddings have since passed BM25 on BEIR, and the two-stage conclusion survived that.

What did BM25 actually win?#

Robustness, not the top score. BEIR tested 18 datasets across 10 systems and found the lexical baseline held up remarkably well out of domain, where several dense retrievers did not. That is a real result. It is also not the same as BM25 being the best retriever, and the paper never says it is.

The system that actually won is the one nobody argues about, and it is not a retriever at all.

What BEIR really concluded#

The abstract states the finding in one sentence, and it names three things rather than two.

BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches.

Read the structure. BM25 is called robust. The best performers are reranking and late interaction. Dense retrieval is called efficient but often worse.

So the paper describes a cost-quality frontier with three points on it. Coverage flattened that into "BM25 beat the neural models". That version drops the winner.

The numbers underneath#

BEIR reports each system as a percentage against BM25, averaged over the 18 datasets. The spread is wide.

DPR came in at −47.7%, which is the number people remember. ANCE managed −7.4%, GenQ −3.6%, and TAS-B −2.8%. So the better dense models of that era landed just under the lexical baseline rather than far below it.

Two systems finished above. ColBERT, using late interaction, reached +2.5%. BM25 with a cross-encoder reranker reached +11%.

That last figure is four times the best single-stage result. The real headline sat in the same table. It was about pipelines, not retrievers.

Five years later, on a different corpus#

A 2026 paper benchmarked ten retrieval strategies on financial documents mixing text and tables, across 23,088 queries over 7,318 documents. Different decade, different domain, same shape.

a two-stage pipeline combining hybrid retrieval with neural reranking achieves Recall@5 of 0.816 and MRR@3 of 0.605, outperforming all single-stage methods by a large margin

They also found the lexical baseline still ahead of a current commercial embedding model.

BM25 outperforms state-of-the-art dense retrieval on financial documents, challenging the common assumption that semantic search universally dominates

Concretely, BM25 reached Recall@5 of 0.644 against 0.587, MRR@3 of 0.411 against 0.351, and nDCG@10 of 0.515 against 0.466. The paper notes dense retrieval led on Recall@20 alone.

The two gaps, side by side#

Now put those three systems in one column and subtract, because the comparison people skip is the interesting one.

Dense alone scored 0.644 minus 0.587, so choosing BM25 over dense bought 0.057 Recall@5. Going from BM25 alone to hybrid retrieval plus reranking bought 0.816 minus 0.644, or 0.172.

That second gain is three times the first. The same ratio holds on MRR@3, where the first-stage choice is worth 0.060 and the second stage another 0.194, so roughly 3.2 times.

Both benchmarks therefore point the same way across five years and two domains. Your first-stage choice is a rounding error next to whether you have a second stage at all.

Why the second stage can do better#

It helps to know why a reranker wins, because the same fact explains why it cannot be your first stage.

A retriever is a bi-encoder. It embeds the query and the document separately, which is what makes it fast. Every document vector gets computed once at index time, and a search becomes a nearest-neighbour lookup over vectors that already exist.

Separately is the operative word. The document vector is fixed before anyone asks a question, so it has to summarise the whole document for every possible query at once.

A cross-encoder does not work that way. It feeds the query and one document through the model together and returns a relevance score. The model can therefore attend to how a specific query term relates to a specific passage, which is information a precomputed vector never had.

That extra context is where the accuracy comes from. It is also the reason nothing can be cached.

Count the work. A bi-encoder does one model pass per query, no matter how large your corpus. A cross-encoder does one pass per candidate, so reranking 50 documents means 50 passes before you answer.

So the architecture forces the pipeline. You cannot cross-encode a million documents per query, and you can cross-encode fifty. BEIR said exactly this when it put reranking at the top "at high computational costs", and the cost is what confines it to a second stage over a short list.

A table for the decision#

Given that, here is how I would spend effort, in order.

If you have	Do this next	Expected payoff
One retriever, no reranker	Add a cross-encoder reranker	The largest single gain available
Reranker already, one retriever	Fuse a second retriever into stage one	Better candidates at small k
Both stages, tuning the first	Stop, and tune k and the reranker depth	The first-stage gap is small
Domain-specific jargon corpus	Keep lexical in the mix	It wins on exact terminology
Hard latency budget, no reranker possible	Pick the first stage carefully	Now the 0.057 is all you get

The last row is the honest exception. If a reranker genuinely does not fit your latency budget, then the first-stage argument becomes your whole decision, and it deserves the attention it normally gets by default.

Fusing two retrievers is the other cheap move, and reciprocal rank fusion does it without training anything.

Why lexical wins where it wins#

The financial result is not a fluke, and the authors explain the mechanism rather than leaving it as a curiosity.

They attribute BM25's advantage to "precise, domain-specific terminology (company names, ticker symbols, standardized metric labels) that lexical matching captures effectively".

That generalises. Exact-match retrieval excels wherever query and document share rare, literal strings. Ticker symbols, error codes, SKUs, API method names, legal citations, part numbers.

Embeddings are built to bridge vocabulary gaps, which is exactly the wrong instinct when the vocabulary gap is zero and the token is the answer. A model that knows AAPL relates to Apple will helpfully return documents about Apple. You wanted the one containing that literal ticker.

The failure mode, and how to detect it#

The characteristic failure is silent and organisational. A team spends a quarter evaluating embedding models, ships the winner, and gains a point or two, having never built the stage worth ten.

Measure your ceiling before you optimise your floor. Take your current first stage, retrieve a generous k, then rerank offline with any competent cross-encoder. Compare Recall@5 before and after.

That number is your available headroom. If reranking moves it by a lot, no first-stage change will catch up, and you now know where the quarter should go.

Run the reverse check too. Swap your first stage for plain BM25, keep everything else fixed, and see how much you actually lose. Teams are regularly surprised by how small that number is, and a small number means the first stage is not your problem.

Here is the whole experiment, in four steps. Collect 100 real queries with known answers. Retrieve the top 50 for each. Rerank those 50 offline. Then score Recall@5 before and after.

It needs no production change and no new service. A laptop and an afternoon are enough. The output is one number, and that number tells you where the next quarter should go.

One warning on evaluation sets. Both papers here measured on out-of-domain data. An evaluation set drawn from your embedding model's training distribution will flatter dense retrieval, so judge on your own queries against your own corpus.

What dense retrieval genuinely gets right#

None of this is an argument for keyword search alone, and the cases where embeddings win are the cases lexical search cannot reach at all.

BM25 fails completely when the query and document share no terms. A user asking about "cancelling my plan" will not match a document titled "subscription termination", and no amount of tuning fixes that, because the words are simply different.

Recall is the other advantage, and the financial paper found it. Dense retrieval led on Recall@20 even while losing on the precision-oriented metrics. That is precisely what you want from a first stage feeding a reranker, since the reranker supplies the precision and the retriever only has to supply the candidate.

That is the real argument for hybrid. Lexical catches literal matches, dense catches paraphrases, and fusion keeps both sets of candidates for the reranker to sort out.

Where this advice is wrong#

BEIR's first-stage verdict has expired. It is a 2021 result, and the field moved. Microsoft's E5 paper claimed in 2022 to be "the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data", and stronger models followed. Quoting BEIR today to argue that embeddings lose would be citing a stale table.

The two-stage finding is what survived. Note which half of this article the 2026 paper reinforced and which half it revised. That asymmetry is the reason to trust the pipeline conclusion more than any particular retriever ranking.

Rerankers are not free, and I have skipped that. A cross-encoder scores every query-document pair, so latency scales with candidate count and you host another model. On a low-traffic internal tool, a slightly better first stage may genuinely beat the operational weight of a second one, and retrieving fewer documents is the cheaper lever to try first.

The short version#

BEIR is cited to prove BM25 beats embeddings, and its own best zero-shot result was cross-encoder reranking at +11%, four times the best single-stage system. A 2026 financial benchmark found the same shape, with two-stage retrieval at 0.816 Recall@5 against 0.644 for BM25 and 0.587 for dense. The first-stage choice was worth 0.057 there and the second stage 0.172. Modern embeddings have since passed BM25 on BEIR, which revised the first-stage answer and left the pipeline answer intact. So add the reranker first, fuse retrievers second, and argue about BM25 versus vectors last.

Sources: Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv:2104.08663), Akarsu et al., From BM25 to Corrective RAG (arXiv:2604.01733), Wang et al., Text Embeddings by Weakly-Supervised Contrastive Pre-training (arXiv:2212.03533)