rapidfire.

Contextual Retrieval: How It Works and How to Implement It

Written By:

Matt Tanner

Published on


Contextual retrieval prepends an LLM-written description of an ACME Q2 2023 filing to a revenue chunk before indexing. At query time, embedding and BM25 results combine through rank fusion.

Alt text (deployment only): A contextual prefix identifies the company, reporting period, and source of a revenue statement before embedding and BM25 indexing; ranked results are fused at query time.

Here's a chunk from a release-notes index: "The default for this limit increased to 64 requests per second." It's the exact answer to someone's question, but nothing in it says which product, version, or limit, so a search for "Tidewater Gateway 3.4 rate limit" has almost nothing to match. Chunking cut that information off, and every later stage of your RAG pipeline inherits the loss.

Contextual retrieval puts it back at index time. This guide covers what it is, how to implement contextual retrieval with a short Python example, what it costs at current Claude Haiku 5.5 prices, and how to check whether it helps on your corpus before you rebuild an index.

What is contextual retrieval?

Contextual retrieval is a RAG indexing technique, introduced by Anthropic in September 2024, that adds a short, chunk-specific explanation to every chunk before indexing. An LLM writes each explanation from the full source document, and the combined text feeds both the embeddings and the BM25 index. Anthropic calls the two halves contextual embeddings and contextual BM25.

Compared with traditional RAG, nothing changes at query time. The query still hits a vector index and, with hybrid search, a keyword index, but both now hold text that carries the document's identity, which gives retrieval-augmented generation a better shot at the right chunks.

The phrase has other meanings too. Neo4j uses it for agents that start from a vector or full-text match and follow relationships through a graph of connected records. Box stretches it to cover context-aware querying, where the user's session or profile shapes the search. This guide covers Anthropic's index-time method.

Why traditional RAG loses context

Traditional RAG splits each document into chunks of a few hundred tokens and embeds each one on its own. That's what lets RAG systems scale, and it's also where meaning leaks out. Anthropic's example is an SEC filing chunk that reads "The company's revenue grew by 3% over the previous quarter." The company and the quarter are named elsewhere in the filing, so a question about one company's Q2 growth has no strong match on keywords or semantic similarity.

Three kinds of information go missing most often:

  • Identity: the product, company, or customer name, usually in the title or a header.

  • Version and time: release numbers, quarters, and effective dates. Chunks from consecutive versions can be nearly identical.

  • Referents: "this limit" or "the company", where the noun lives in an earlier paragraph.

Smaller chunks make all three worse, because fewer chunks contain the header. Bigger chunks keep more together but blur the embedding and send more text to the model. Semantic chunking puts boundaries in better places, but it can't copy a page title into every chunk below it.

The damage shows up as retrieval failure: the chunk with the answer exists but isn't in the top k, so the model never sees it. Anthropic measures the retrieval failure rate as 1 minus recall@20, the share of relevant chunks missing from the top 20. That's a cleaner target than answer quality, which mixes retrieval misses with generation mistakes.

How contextual retrieval works

For each chunk, you send a model the whole document plus that chunk. Anthropic's prompt wraps them in <document> and <chunk> tags and asks the model to "give a short, succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk," answering with nothing else. The result is usually 50 to 100 tokens.

You prepend that context to the chunk and index the combined text twice:

  1. Contextual embeddings: embed context plus chunk with your embedding model and store the vectors in your vector database.

  2. Contextual BM25: build the BM25 index over the same combined text, so terms that only appear in the context, such as product names, version numbers, and quarters, become searchable keywords.

Here's one chunk at each stage, using Anthropic's SEC filing example:

Stage

Text

Original chunk

The company's revenue grew by 3% over the previous quarter.

Generated context

This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million.

Indexed for embeddings and BM25

The context, a blank line, then the original chunk

Shown as the citation

The original chunk only

The last row matters. Anthropic's cookbook stores original_content and contextualized_content as separate fields and embeds the two joined. Keep that split, since the original is your evidence and the context is a model's summary that can be wrong. Anthropic also suggests giving the generator the contextualized chunk with a clear line between context and chunk.

The context has to be about this chunk, in this document. Anthropic tried adding generic document summaries to chunks and saw very limited gains.


Contextual Retrieval: How It Works and How to Implement It โ€” diagram 2

Context is added before indexing. At query time, vector and BM25 results are combined, with optional reranking.

Alt text (deployment only): Contextual retrieval pipeline: generate context for each document chunk, index context plus chunk in vector and BM25 indexes, retrieve ranked lists, fuse them, optionally rerank, and send top chunks to the LLM.

All of this happens once, at ingestion. Unlike query-time techniques such as HyDE, there's no extra contextualization model call on the search path. End-to-end latency can still change because the index, retrieved text, and generator prompt are larger; measure it rather than assuming it stays constant.

How to implement it, step by step

Here's how to implement contextual retrieval in six steps. Steps 1, 5, and 6 are standard hybrid RAG, and steps 2 to 4 are the contextual part.

  1. Chunk the documents and keep a document ID and chunk ID on every chunk, since step 2 needs the full document and evaluation needs the IDs.

  2. Generate a context for each chunk with an LLM.

  3. Embed the context plus chunk and write the vectors to your vector database, with the original chunk text in its own field.

  4. Build a BM25 index over the same combined text.

  5. Retrieve from both indexes and merge the ranked lists with rank fusion, then keep the top k.

  6. Optionally rerank the fused candidates before they go to the model.

Generate the context

This function follows the cookbook's request shape and model, Claude Haiku 5.5. Install the anthropic Python package and set ANTHROPIC_API_KEY before running it; real calls incur API charges. The document goes in the first content block with cache_control, because it's identical for every chunk of that document. The instruction and chunk go in the second block, which changes on every call.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

CHUNK_PROMPT = """Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""

def situate(document: str, chunk: str) -> str:
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": [
                # Identical across chunks; cache reuse also needs the minimum length and TTL.
                {"type": "text", "text": f"<document>\n{document}\n</document>",
                 "cache_control": {"type": "ephemeral"}},
                # Changes on every call.
                {"type": "text", "text": CHUNK_PROMPT.format(chunk=chunk)},
            ],
        }],
    )
    usage = response.usage
    print("cache write:", usage.cache_creation_input_tokens,
          "cache read:", usage.cache_read_input_tokens)
    return "".join(block.text for block in response.content if block.type == "text")
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

CHUNK_PROMPT = """Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""

def situate(document: str, chunk: str) -> str:
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": [
                # Identical across chunks; cache reuse also needs the minimum length and TTL.
                {"type": "text", "text": f"<document>\n{document}\n</document>",
                 "cache_control": {"type": "ephemeral"}},
                # Changes on every call.
                {"type": "text", "text": CHUNK_PROMPT.format(chunk=chunk)},
            ],
        }],
    )
    usage = response.usage
    print("cache write:", usage.cache_creation_input_tokens,
          "cache read:", usage.cache_read_input_tokens)
    return "".join(block.text for block in response.content if block.type == "text")
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

CHUNK_PROMPT = """Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""

def situate(document: str, chunk: str) -> str:
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": [
                # Identical across chunks; cache reuse also needs the minimum length and TTL.
                {"type": "text", "text": f"<document>\n{document}\n</document>",
                 "cache_control": {"type": "ephemeral"}},
                # Changes on every call.
                {"type": "text", "text": CHUNK_PROMPT.format(chunk=chunk)},
            ],
        }],
    )
    usage = response.usage
    print("cache write:", usage.cache_creation_input_tokens,
          "cache read:", usage.cache_read_input_tokens)
    return "".join(block.text for block in response.content if block.type == "text")

This example shows the API request shape; the search demo below uses hand-written contexts. When you run it for real, send a document's first chunk on its own and the rest after it, because a cache entry only becomes available once the first response starts. If both usage counters stay at zero, nothing was cached.

Index and search

Before: The default for this limit increased to 64 requests per second.

After: From the Tidewater Gateway 3.4 release notes, Rate limiting section. The default for this limit increased to 64 requests per second.

The next script builds plain and contextual versions of a six-chunk release-notes index, searches each with BM25 and embeddings, and fuses the rankings with reciprocal rank fusion (RRF) using k = 60, the constant from the original RRF paper. The contexts are hand-written in the title-and-section format a model often produces, so no API key is needed. It uses rank-bm25 0.2.2, sentence-transformers 5.7.0 and the all-MiniLM-L6-v2 model (pip install rank-bm25==0.2.2 sentence-transformers==5.7.0).

import re
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

# (chunk_id, original chunk text, hand-written context stand-in)
CHUNKS = [
    ("tw34-rate", "The default for this limit increased to 64 requests per second.",
     "From the Tidewater Gateway 3.4 release notes, Rate limiting section."),
    ("tw34-logs", "Access logs now include the upstream latency field.",
     "From the Tidewater Gateway 3.4 release notes, Logging section."),
    ("tw33-rate", "The default for this limit stays at 32 requests per second.",
     "From the Tidewater Gateway 3.3 release notes, Rate limiting section."),
    ("hc34-up", "This limit increased to 10 parallel uploads.",
     "From the Harbor CLI 3.4 release notes, Uploads section."),
    ("hc34-auth", "Tokens now expire after 12 hours by default.",
     "From the Harbor CLI 3.4 release notes, Authentication section."),
    ("ma21-buf", "This limit increased to 500 queued events.",
     "From the Mooring Agent 2.1 release notes, Buffering section."),
]
QUERIES = [
    ("What is the default rate limit in Tidewater Gateway 3.4?", "tw34-rate"),
    ("What is the default rate limit in Tidewater Gateway 3.3?", "tw33-rate"),
    ("How long do tokens last in Harbor CLI 3.4?", "hc34-auth"),
]

def tokens(text):
    return re.findall(r"[a-z0-9]+(?:\.[0-9]+)*", text.lower())

def rrf(rankings, k=60):
    # Reciprocal rank fusion: each list adds 1 / (k + rank) to a chunk's score.
    scores = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda kv: (-kv[1], kv[0]))

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
ids = [cid for cid, _, _ in CHUNKS]

for variant in ("plain", "contextual"):
    # Contextual: prepend the context before BM25 indexing and embedding.
    texts = [chunk if variant == "plain" else f"{ctx}\n\n{chunk}"
             for _, chunk, ctx in CHUNKS]
    bm25 = BM25Okapi([tokens(t) for t in texts])
    doc_vecs = model.encode(texts, normalize_embeddings=True)
    print(f"== {variant}")
    for query, gold in QUERIES:
        lex = bm25.get_scores(tokens(query))
        sem = doc_vecs @ model.encode(query, normalize_embeddings=True)
        # BM25 only returns chunks that share a weighted term with the query.
        by_bm25 = [ids[i] for i in sorted(range(len(ids)), key=lambda i: (-lex[i], ids[i]))
                   if lex[i] > 0]
        by_vec = [ids[i] for i in sorted(range(len(ids)), key=lambda i: (-sem[i], ids[i]))]
        fused = rrf([by_bm25, by_vec])
        rank = lambda ranking: ranking.index(gold) + 1 if gold in ranking else "-"
        top = ", ".join(f"{cid} {score:.4f}" for cid, score in fused[:2])
        print(f"  {gold:<9} BM25={rank(by_bm25)} vector={rank(by_vec)} "
              f"fused={rank([cid for cid, _ in fused])} | {top}")
import re
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

# (chunk_id, original chunk text, hand-written context stand-in)
CHUNKS = [
    ("tw34-rate", "The default for this limit increased to 64 requests per second.",
     "From the Tidewater Gateway 3.4 release notes, Rate limiting section."),
    ("tw34-logs", "Access logs now include the upstream latency field.",
     "From the Tidewater Gateway 3.4 release notes, Logging section."),
    ("tw33-rate", "The default for this limit stays at 32 requests per second.",
     "From the Tidewater Gateway 3.3 release notes, Rate limiting section."),
    ("hc34-up", "This limit increased to 10 parallel uploads.",
     "From the Harbor CLI 3.4 release notes, Uploads section."),
    ("hc34-auth", "Tokens now expire after 12 hours by default.",
     "From the Harbor CLI 3.4 release notes, Authentication section."),
    ("ma21-buf", "This limit increased to 500 queued events.",
     "From the Mooring Agent 2.1 release notes, Buffering section."),
]
QUERIES = [
    ("What is the default rate limit in Tidewater Gateway 3.4?", "tw34-rate"),
    ("What is the default rate limit in Tidewater Gateway 3.3?", "tw33-rate"),
    ("How long do tokens last in Harbor CLI 3.4?", "hc34-auth"),
]

def tokens(text):
    return re.findall(r"[a-z0-9]+(?:\.[0-9]+)*", text.lower())

def rrf(rankings, k=60):
    # Reciprocal rank fusion: each list adds 1 / (k + rank) to a chunk's score.
    scores = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda kv: (-kv[1], kv[0]))

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
ids = [cid for cid, _, _ in CHUNKS]

for variant in ("plain", "contextual"):
    # Contextual: prepend the context before BM25 indexing and embedding.
    texts = [chunk if variant == "plain" else f"{ctx}\n\n{chunk}"
             for _, chunk, ctx in CHUNKS]
    bm25 = BM25Okapi([tokens(t) for t in texts])
    doc_vecs = model.encode(texts, normalize_embeddings=True)
    print(f"== {variant}")
    for query, gold in QUERIES:
        lex = bm25.get_scores(tokens(query))
        sem = doc_vecs @ model.encode(query, normalize_embeddings=True)
        # BM25 only returns chunks that share a weighted term with the query.
        by_bm25 = [ids[i] for i in sorted(range(len(ids)), key=lambda i: (-lex[i], ids[i]))
                   if lex[i] > 0]
        by_vec = [ids[i] for i in sorted(range(len(ids)), key=lambda i: (-sem[i], ids[i]))]
        fused = rrf([by_bm25, by_vec])
        rank = lambda ranking: ranking.index(gold) + 1 if gold in ranking else "-"
        top = ", ".join(f"{cid} {score:.4f}" for cid, score in fused[:2])
        print(f"  {gold:<9} BM25={rank(by_bm25)} vector={rank(by_vec)} "
              f"fused={rank([cid for cid, _ in fused])} | {top}")
import re
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

# (chunk_id, original chunk text, hand-written context stand-in)
CHUNKS = [
    ("tw34-rate", "The default for this limit increased to 64 requests per second.",
     "From the Tidewater Gateway 3.4 release notes, Rate limiting section."),
    ("tw34-logs", "Access logs now include the upstream latency field.",
     "From the Tidewater Gateway 3.4 release notes, Logging section."),
    ("tw33-rate", "The default for this limit stays at 32 requests per second.",
     "From the Tidewater Gateway 3.3 release notes, Rate limiting section."),
    ("hc34-up", "This limit increased to 10 parallel uploads.",
     "From the Harbor CLI 3.4 release notes, Uploads section."),
    ("hc34-auth", "Tokens now expire after 12 hours by default.",
     "From the Harbor CLI 3.4 release notes, Authentication section."),
    ("ma21-buf", "This limit increased to 500 queued events.",
     "From the Mooring Agent 2.1 release notes, Buffering section."),
]
QUERIES = [
    ("What is the default rate limit in Tidewater Gateway 3.4?", "tw34-rate"),
    ("What is the default rate limit in Tidewater Gateway 3.3?", "tw33-rate"),
    ("How long do tokens last in Harbor CLI 3.4?", "hc34-auth"),
]

def tokens(text):
    return re.findall(r"[a-z0-9]+(?:\.[0-9]+)*", text.lower())

def rrf(rankings, k=60):
    # Reciprocal rank fusion: each list adds 1 / (k + rank) to a chunk's score.
    scores = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda kv: (-kv[1], kv[0]))

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
ids = [cid for cid, _, _ in CHUNKS]

for variant in ("plain", "contextual"):
    # Contextual: prepend the context before BM25 indexing and embedding.
    texts = [chunk if variant == "plain" else f"{ctx}\n\n{chunk}"
             for _, chunk, ctx in CHUNKS]
    bm25 = BM25Okapi([tokens(t) for t in texts])
    doc_vecs = model.encode(texts, normalize_embeddings=True)
    print(f"== {variant}")
    for query, gold in QUERIES:
        lex = bm25.get_scores(tokens(query))
        sem = doc_vecs @ model.encode(query, normalize_embeddings=True)
        # BM25 only returns chunks that share a weighted term with the query.
        by_bm25 = [ids[i] for i in sorted(range(len(ids)), key=lambda i: (-lex[i], ids[i]))
                   if lex[i] > 0]
        by_vec = [ids[i] for i in sorted(range(len(ids)), key=lambda i: (-sem[i], ids[i]))]
        fused = rrf([by_bm25, by_vec])
        rank = lambda ranking: ranking.index(gold) + 1 if gold in ranking else "-"
        top = ", ".join(f"{cid} {score:.4f}" for cid, score in fused[:2])
        print(f"  {gold:<9} BM25={rank(by_bm25)} vector={rank(by_vec)} "
              f"fused={rank([cid for cid, _ in fused])} | {top}")

This illustrative output shows the right chunk's rank in each list, then the top two fused results with their RRF scores:

== plain
  tw34-rate BM25=4 vector=2 fused=4 | hc34-up 0.0323, tw33-rate 0.0323
  tw33-rate BM25=3 vector=1 fused=2 | hc34-up 0.0323, tw33-rate 0.0323
  hc34-auth BM25=1 vector=1 fused=1 | hc34-auth 0.0328, tw33-rate 0.0161
== contextual
  tw34-rate BM25=1 vector=2 fused=2 | tw33-rate 0.0325, tw34-rate 0.0325
  tw33-rate BM25=1 vector=1 fused=1 | tw33-rate 0.0328, tw34-rate 0.0323
  hc34-auth BM25=1 vector=1 fused=1 | hc34-auth 0.0328, hc34-up 0.0323
== plain
  tw34-rate BM25=4 vector=2 fused=4 | hc34-up 0.0323, tw33-rate 0.0323
  tw33-rate BM25=3 vector=1 fused=2 | hc34-up 0.0323, tw33-rate 0.0323
  hc34-auth BM25=1 vector=1 fused=1 | hc34-auth 0.0328, tw33-rate 0.0161
== contextual
  tw34-rate BM25=1 vector=2 fused=2 | tw33-rate 0.0325, tw34-rate 0.0325
  tw33-rate BM25=1 vector=1 fused=1 | tw33-rate 0.0328, tw34-rate 0.0323
  hc34-auth BM25=1 vector=1 fused=1 | hc34-auth 0.0328, hc34-up 0.0323
== plain
  tw34-rate BM25=4 vector=2 fused=4 | hc34-up 0.0323, tw33-rate 0.0323
  tw33-rate BM25=3 vector=1 fused=2 | hc34-up 0.0323, tw33-rate 0.0323
  hc34-auth BM25=1 vector=1 fused=1 | hc34-auth 0.0328, tw33-rate 0.0161
== contextual
  tw34-rate BM25=1 vector=2 fused=2 | tw33-rate 0.0325, tw34-rate 0.0325
  tw33-rate BM25=1 vector=1 fused=1 | tw33-rate 0.0328, tw34-rate 0.0323
  hc34-auth BM25=1 vector=1 fused=1 | hc34-auth 0.0328, hc34-up 0.0323

How to read this example:

  • Plain chunks lost both rate-limit questions. The Tidewater chunks never say "Tidewater", "3.4", or "rate", so BM25 had little besides "limit", which four chunks share. The 3.4 chunk came fourth, and the 3.3 chunk tied for first with a Harbor CLI chunk and lost on ID.

  • Contextual BM25 ranked the right version first both times, because "3.3" and "3.4" are now tokens in the index.

  • The embeddings still couldn't separate versions. They ranked the 3.3 chunk first for both questions, so the 3.4 question ended in an exact RRF tie (0.0325 each) that the script broke the wrong way.

  • The token-expiry question didn't need help, since that chunk already contains the words that matter.

Six chunks show mechanics, not the size of a gain. They do show where to look in your data: identifiers and version strings are where contextual BM25 pays off, and fusion weights decide whether that signal survives. The cookbook's hybrid search defaults to 80% semantic and 20% BM25 weight, which would have tipped this tie the wrong way too. Our guide to hybrid search covers fusion methods and tuning.

Two production notes. The cookbook warns that embedding models with fixed input limits can truncate contextualized chunks and make contextual embeddings look worse than they are, so check the limit against your longest chunk plus context. It also runs BM25 in Elasticsearch, where BM25 is the default similarity; rank_bm25 is fine for a local test.

Reducing the cost

Contextualization costs one LLM call per chunk, and every call includes the whole document, so without caching, a 10-chunk document gets sent ten times. Four levers bring that down:

Lever

What it changes

Watch for

Prompt caching

Document tokens after the first call bill at 0.1x the input price

512-token minimum on Haiku 5.5 (4,096 on Haiku 4.5) and a 5-minute default lifetime

Smaller model

Haiku-class models cost the least per token

Spot-check context quality on a sample before indexing everything

Batch API

50% off input and output, and it stacks with caching

Results can take up to 24 hours and cache hits are best effort, so use the 1-hour cache

One-time indexing

You pay at ingestion, not per query

Editing a document can make every context in it stale

Prompt caching does most of the work. The first call for a document writes it to the cache at 1.25x the base input price, or 2x for the 1-hour cache. Later calls within the cache lifetime read it at 0.1x. A prompt below the model's minimum isn't cached and doesn't raise an error, so a short document quietly costs full price.

To size it, use Anthropic's assumptions: 800-token chunks, 8,000-token documents, 50 tokens of instructions and 100 tokens of context per chunk.

We priced them at Claude Haiku 5.5 list prices on November 13, 2026, which are $0.10 per million input tokens, $0.50 per million output tokens, $0.125 for 5-minute cache writes, and $0.01 for cache reads.

That works out to about $1.17 per million document tokens without caching and $0.38 with the 5-minute cache, roughly a third. Run the same jobs through the Batch API at its 50% discount, and it would drop to about $0.19, if every cache read still hit, which batches don't guarantee.

Anthropic's 2024 figure was $1.02 per million document tokens on an older Haiku model, so treat any published number as a snapshot. Haiku 5.5 also charges more for prompts over 100,000 tokens, which matters if your documents are long.

On Amazon Bedrock Knowledge Bases, AWS's walkthrough runs contextualization in a custom chunking Lambda that calls Claude during ingestion. Check Bedrock's own caching docs for minimums and usage field names, and note that automatic caching isn't available on Bedrock's runtime API, so use an explicit breakpoint like the one above.

The cost people underestimate is maintenance. Each context depends on the whole document, so one edited paragraph can make every context in it stale. Budget for regeneration, and judge spend by cost per successful task rather than per call, as our LLM cost optimization guide recommends.

Adding reranking and hybrid search

Contextual retrieval changes what's in your indexes. Hybrid search and reranking change how you search them, so the three stack.

Anthropic's best setup retrieved the top 150 candidates from contextual embeddings and contextual BM25, reranked them with Cohere's reranker and kept the top 20 for the prompt. Reranking adds a second pass in which a model scores the query against each candidate directly. Two points are specific to contextual retrieval:

  • The reranker can read the context too. Pass it the combined text, as the cookbook does, and it can use the document identity the context adds.

  • Reranking costs latency on every query, unlike contextualization. Anthropic notes the trade-off between reranking more chunks and keeping latency and cost down, and the cookbook puts the added latency at about 100 to 200 ms per query.

Layers don't always add up neatly. In the cookbook, adding contextual BM25 hybrid search lifted Pass@10 from 92.34% to 93.21% but dropped Pass@5 from 88.12% to 86.43%, and its reranking step was built on contextual embeddings alone, not the hybrid setup. Evaluate each layer at the k you actually send to the model.

What improvement to expect

Anthropic's September 2024 numbers get repeated everywhere, so here they are with their conditions: averages across codebases, fiction, ArXiv papers and science papers, with Gemini Text 004 embeddings and the top 20 chunks, from Anthropic's contextual retrieval post.

Setup

Top-20 retrieval failure rate

Reduction vs. baseline

Baseline

5.7%

n/a

Contextual embeddings

3.7%

35%

Contextual embeddings + contextual BM25

2.9%

49%

Both, plus reranking

1.9%

67%

The 49% figure is contextual embeddings plus contextual BM25 without a reranker; reranking is what takes it to 67%.

The Claude Cookbook gives a second, smaller data point: 9 codebases, 248 queries, scored with Pass@k. Contextual embeddings moved Pass@10 from 87.15% to 92.34%, hybrid search took it to 93.21%, and reranking took it to 95.26%. AWS's Bedrock walkthrough reported better context recall, context precision, and answer correctness in RAGAS on a small test set built from a single AWS guide.

Those are vendor results on vendor-chosen data. What moves the gain on your corpus:

  • How self-contained your chunks are. If every chunk names its product and version, as the token-expiry chunk did, there's little to add.

  • How users ask. Questions that name an entity, version or date gain the most.

  • Your embedding model. Anthropic saw gains with every model it tested, larger for some.

  • Chunk size, boundaries and overlap, which Anthropic lists as factors.

  • Your baseline. At 87% Pass@10 there's room, while near-perfect recall@k leaves little.

These results make contextual retrieval worth testing, not a guarantee of improvement on your corpus.

How to test it on your own data

Contextual retrieval adds choices: chunk size, the context prompt and model, the embedding model, the BM25 weight in fusion, whether to rerank, and top k. Benchmarks can't settle them, so test whether the gain on your corpus is worth the indexing cost.

1. Build a question set with gold chunk IDs. Pull real queries from logs where you can, including ones that name a product, version, customer, or period, since those are what contextual retrieval should fix. Label the chunk IDs that hold each answer, and keep unanswerable questions in a separate group so they don't distort recall.

2. Measure recall@k at the k you send to the model. Recall@k is the share of a query's relevant chunks that appear in the top k; compute it per query and average over answerable queries. Score answers on a smaller sample too, because better retrieval doesn't guarantee better answers. Our guide to RAG evaluation metrics covers both sides.

If you use RAGAS, its context recall metric is LLM-graded: it checks whether claims in a reference answer can be attributed to the retrieved text, which is a different measurement from ID-based recall@k.

3. Compare variants that change one thing at a time, holding chunking, the embedding model, and k constant:

Variant

Indexed text

Search

A

Plain chunks

Vectors

B

Context + chunk

Vectors

C

Context + chunk

Vectors + BM25, fused

D

Context + chunk

C, then a reranker

E

Title and section prefix, no LLM

Same as C

B against A isolates contextual embeddings, and C against B isolates contextual BM25. E is the cheap control. If a prefix built from headings gets most of the gain, you may not need a model at all, and the six-chunk demo above used exactly that kind of prefix.

We built RapidFire to run comparisons like this on one eval set. The open-source engine runs RAG and context engineering configurations side by side and processes the eval set in shards. As each shard finishes, it shows running metric estimates with confidence intervals, so you can stop a variant that's clearly losing instead of paying to finish it.

In the open-source engine, you do that by hand with IC Ops (stop, resume, clone-modify). Converge stops weak runs and clones strong ones on its own, within the time and spend limits you set.

A few specifics matter for this experiment:

  • Plain and contextual indexes need separate configs from the start. Clone-Modify can change query-side knobs such as search_cfg, the reranker, or the generator mid-run, but document preprocessing is fixed once a run starts, since changing it means rebuilding the index.

  • There's no built-in BM25. The built-in search types are vector-based (similarity, similarity with a score threshold, and MMR), so variant C goes in as a custom retriever, which can be any LangChain BaseRetriever.

  • Generate contexts before the experiment and load the contextualized chunks as documents. Skip the text splitter and the RAG spec embeds each one whole, while document_template formats each retrieved chunk for the prompt, so you can label context and chunk text separately.

  • Type your metrics. Return per-batch sums and counts from compute_metrics_fn and divide in accumulate_metrics_fn. In rapidfireai 0.16.2, without accumulate_metrics_fn the engine calls compute_metrics_fn on the collected dataset-level results at the end; it does not provide the same running per-batch metric estimates. Mark recall as algebraic with a value range of 0 to 1, or the dashboard shows it without running estimates or confidence intervals. Shuffle the eval set first if it's grouped by source or difficulty.

When it is not worth it

Skip contextual retrieval, or test something cheaper first, when:

  • The corpus is small. Anthropic's own advice is that a knowledge base under 200,000 tokens can go straight into the prompt. Our comparison of long context vs. RAG covers how to test that route.

  • Chunks are already self-contained. FAQ entries and product records that name the entity in every chunk gain little.

  • Documents change constantly. An edit can invalidate the contexts for the whole document, and you pay to regenerate them. A heading-based prefix costs nothing to rebuild.

  • Baseline recall is already high at the k you use. Put the effort into reranking or generation instead.

  • The failures are about reasoning, not lookup. If the right chunks come back but answering takes several searches or a decision about what to look up next, agentic RAG is the better fit, and contextual retrieval can sit underneath it.

The cheapest experiment is the one in this post. Take a chunk you know fails, like "The default for this limit increased to 64 requests per second.", prepend its title and section, and see whether it starts showing up.

If that fixes most misses, ship it. If not, generate contexts for a sample, run variants A to E against your question set, and let recall@k and answer quality decide. RapidFire's RAG quickstart runs in Colab with no setup, or locally with pip install rapidfireai on Python 3.12 or newer.

FAQ

What is contextual retrieval?

Contextual retrieval is Anthropic's technique for improving RAG retrieval: an LLM writes a short, chunk-specific context for every chunk, which is prepended before embedding and BM25 indexing. In Anthropic's tests, it cut top-20 retrieval failures by 49%, and by 67% with reranking.

What is an example of contextual?

In retrieval, a contextual chunk carries what it needs to be understood on its own. "The company's revenue grew by 3% over the previous quarter" isn't contextual. Prefix it with "This chunk is from an SEC filing on ACME corp's performance in Q2 2023" and a search for ACME's Q2 revenue can find it.

What does "contextual" mean in research?

"Contextual" means understood in relation to surrounding circumstances rather than in isolation. In information retrieval, it usually describes representations or searches that account for surrounding text, a passage's source document, or the user's situation. That's why the phrase covers Anthropic's chunk enrichment, session-aware search, and graph-based agent memory.

What is contextual recall?

Contextual recall, usually called context recall, is a RAG evaluation metric for how much of a reference answer's information appears in the retrieved context. In RAGAS, it's LLM-graded: the reference is split into claims, and each claim is checked against the retrieved text. That's related to recall@k over labeled chunk IDs, but it's a different measurement.

What is the main purpose of chunking in contextual retrieval?

Chunking still sets the unit you retrieve and send to the model, which keeps prompts small and lets search target specific passages. Contextual retrieval doesn't replace it. It adds a short context to each chunk so smaller chunks stay findable, and Anthropic notes that chunk size, boundaries, and overlap still affect retrieval performance.