A reranker sorts the results of a search a second time, this time thoroughly: it reads the question and the result together and scores how well they match. That makes RAG systems much more accurate, but costs time and compute on every request. A memory that learns from use sorts better by itself over time and needs the reranker less and less.
What is a reranker?
A reranker is the second stage of an AI search. The first stage, usually a vector or keyword search, pulls a few dozen candidates from a large collection in milliseconds. Fast, but rough. Then comes the reranker: it takes each candidate individually, reads it together with the question and assigns a score. Only the best land in the language model's context.
Think of it like a job application: the first stage sorts by keywords in the résumé. The reranker conducts the interview.
Bi-encoder and cross-encoder
- Bi-encoder (first stage): Question and text are turned into vectors separately. The texts can be precomputed, and search is lightning fast, even across millions of entries. But subtle relationships between question and text fall by the wayside.
- Cross-encoder (reranker): Question and text go through the model together. This is far more accurate, but must be recomputed for every request and every candidate. That's why it only pays off for a small shortlist.
This division of labor, broad and fast first, then narrow and precise, has become the standard of modern text search since 2019.
What reranking delivers
A lot. Anthropic systematically compared different methods for RAG systems. An additional reranking step cut the rate of searches where the relevant passage was missing by a further wide margin: 67% overall, from 5.7% to 1.9%. The takeaway there: reranking beats no reranking.
And because only the best results are passed on, the context stays lean. A win, because full contexts make models worse.
The catch: every request costs
A reranker starts from scratch on every single request. That costs:
- Time: An extra compute step lengthens every search.
- Money or compute: Hosted rerankers are billed per request, your own need processing power. If you use a language model as a reranker, every scoring costs tokens.
- A trade-off: Scoring more candidates means better results, but more waiting time and cost.
Above all: a reranker learns nothing. Tomorrow it scores the same question exactly as it does today, no matter whether its pick helped yesterday or not.
Reranker or learning memory?
For a static document collection, a reranker is the right choice. But an AI memory isn't static. Every day it sees which memories actually helped your agent and which didn't. That experience is worth more than any one-off score.
A reranker judges anew every time. A memory remembers what was right.
Hebby: reranker on board, rarely needed
Hebby ships with a built-in, economical reranker. But Hebby doesn't depend on it. Following the principle of Hebbian learning, it strengthens the connections between memories that are used together, and so learns by itself what belongs to which kind of work. The longer Hebby runs, the more accurately it sorts on its own and the less often the reranker is needed. Better results, less waiting, lower costs.
Hebby Memories: the second brain for Claude Code, Codex & co. Learns as you work, thinks ahead, never forgets what matters. Get Early Access →
Frequently asked questions
What is a reranker?
A reranker is a model that re-sorts the results of a first, fast search. It scores each candidate together with the question and passes only the most relevant ones on to the language model.
What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder turns question and text into vectors separately and compares them. That is fast and suited to searching large collections. A cross-encoder reads question and text together. That is more accurate but much slower, so it is applied only to a few candidates, as a reranker.
Do I need a reranker for RAG?
For static document collections, a reranker usually brings noticeably better results. A memory that learns from its own use, like Hebby, instead sorts correctly by itself over time and needs one only rarely.