rapidfire.

RAG Query Rewriting: Techniques, Examples and Evaluation

Written By:

Matt Tanner

Published on


Example BM25 rankings: rewriting 'what about the pricing?' as 'pricing for team plans' moves the Team plan chunk from outside the top three to first place.

Alt text (deployment only): Example BM25 rankings: rewriting โ€œwhat about the pricing?โ€ as โ€œpricing for team plansโ€ moves the Team plan chunk from outside the top three to first.

A user asks your docs assistant whether the Team plan includes SSO, gets a yes, and follows up with "what about the pricing?" Your retriever searches for exactly that string and comes back with the API pricing page, the enterprise pricing page and a pricing FAQ. The one chunk that lists the Team plan's per-user price never shows up, because nothing in "what about the pricing?" says which plan the user means.

RAG query rewriting fixes that class of miss by changing the user query before retrieval runs. This guide covers the main query rewriting techniques, from paraphrasing and query expansion to HyDE and routing, plus a worked example with a model prompt and illustrative rewrite. Then it shows how to implement query rewriting with rank fusion, where rewriting makes a RAG system worse, and how to test it on your own questions before you ship.

What is query rewriting in RAG?

Query rewriting in RAG is a step that transforms the user's question into one or more search queries before retrieval runs. A model reads the user query and any chat history. It produces a rewritten query that's standalone, specific, and closer to how your documents are phrased, and retrieval runs on it, often alongside the original.

The user query and the retrieval query don't have to be the same string. The user writes for a person who remembers the conversation; the retriever needs something it can match against chunks with no memory at all. In our example, the original query is "what about the pricing?" and the rewritten query is "pricing for team plans". The generator still answers the user's actual question. Only the search changes.

You'll also see this called query transformation or query optimization. Those terms usually cover the whole family: rewriting one query, expanding it, splitting it, or generating several. Query optimization in this sense is about retrieval, which is different from the query planner in your database.

Why user queries fail in retrieval

Vector search and keyword search both compare the user query against chunks as they were indexed. Neither can ask a clarifying question, so four patterns cause most of the cases where relevant documents never come back.

  • Missing context in follow-ups. "What about the pricing?", "How do I configure it?" and "Is that on the free plan?" depend on earlier turns. The pronoun or the implied subject never reaches the retriever.

  • Conversational phrasing. "Hey, so I was wondering if there's any way to get my money back" carries one useful term (money back) among filler words that dilute an embedding and add noise to a keyword match.

  • Very short queries. "rate limits" or "SSO" match dozens of chunks equally well. The retriever has nothing to rank on beyond the one term.

  • Vocabulary mismatch. Users say "cancel my plan" while the docs say "downgrade to Free". Our Team plan chunk says "$12 per user per month" and never uses the word "pricing", so a keyword search for "pricing" scores it at zero.

Chunking makes the last problem worse. Chunk boundaries, as our guide to semantic chunking puts it, "determine which facts stay together and which get separated." If the plan name and its price land in different chunks, a well-phrased query has less to match on and a vague one has almost nothing. Rewriting can't repair a bad chunk, but it gives the retriever better terms to work with.

Where rewriting fits in a RAG pipeline

Rewriting sits between the user and the retriever. In a typical RAG pipeline, the order is:

  1. Take the user query and apply the chat history (in conversational RAG, the last few turns).

  2. Send both to a rewriter LLM, which returns one or several rewritten queries.

  3. Run retrieval for each query, keep the original query in the set, and merge the result lists with rank fusion.

  4. Rerank the merged candidates.

  5. Pass the top chunks and the user's original question to the generator.


RAG Query Rewriting: Techniques, Examples and Evaluation โ€” diagram 2

Rewrite after applying chat history, keep the original query in retrieval, and pass the original question to the generator.

Alt text (deployment only): Query rewriting pipeline: combine the question with chat history, produce rewritten queries, retrieve and merge results, rerank, then generate an answer. The original query is also retrieved, and the original question goes to the generator.

Two placement details matter more than they look. First, the rewrite has to happen after chat history is applied and before any search call, including the keyword half of hybrid search. If you only rewrite the vector query, the BM25 side still searches for "what about the pricing?". Second, rewriting and reranking solve different problems. Rewriting decides which candidates enter the pool. A reranker can only reorder chunks that were already retrieved, so it can't rescue a chunk the query never matched.

Query rewriting techniques

Each query transformation below changes the user query in a different way. They aren't mutually exclusive: a common production setup is a conversational rewrite followed by multi-query generation.

Simple LLM rewrite (paraphrase)

Ask a model to turn the user's message into one clean, standalone rewritten query. This is the technique most people mean by query rewriting, and it's the one in our worked example.

  • Before: "what about the pricing?" (after a turn about the Team plan)

  • After: "pricing for team plans"

It helps most with follow-ups and chatty phrasing, which makes it the usual first step in conversational RAG. Cost is one short LLM call per user turn, and you can skip it when there's no chat history. The research version goes further: Ma et al. proposed a Rewrite-Retrieve-Read pipeline in which a small trainable rewriter learns from the LLM reader's feedback through reinforcement learning.

Query expansion

Query expansion adds related terms, synonyms, or a short pseudo-answer to the original query instead of replacing it. Classic IR did this with thesauri and relevance feedback. LLM-based versions such as Query2doc (Wang et al.) have the model write a pseudo-document and append it to the query, which they tested with both BM25 and dense retrievers.

  • Before: "cancel subscription"

  • After: "cancel subscription downgrade plan end billing"

Expansion helps keyword search with vocabulary mismatch. It costs one LLM call and can add noise if the model invents terms your docs don't use.

Multi-query generation

Generate several differently worded queries from one question, retrieve for each, and merge the results. The multi-query approach widens recall because each phrasing can match relevant documents the others miss.

  • Before: "SSO setup"

  • After: "configure SAML single sign-on", "connect identity provider", "SSO admin settings"

The cost scales with the number of queries: N rewrites mean N retrieval calls plus a merge step.

Query decomposition

Split a compound question into sub-questions and retrieve for each. "Does the Team plan include audit logs, and how long are they kept?" becomes two queries: "Team plan audit logs" and "audit log retention period". Decomposition helps with multi-part and comparison questions, where no single chunk answers everything. Sub-questions sometimes depend on each other's answers, which pushes you toward multi-step retrieval and more latency.

Step-back prompting

Ask the model for a more general question first, then retrieve for both. Zheng et al. introduced step-back prompting so LLMs could "derive high-level concepts and first principles from instances containing specific details." They reported gains on reasoning-heavy benchmarks including TimeQA and MuSiQue. For RAG, the step-back query pulls in background chunks the specific question misses.

  • Before: "Why did my SSO login fail after we rotated the IdP certificate?"

  • Step-back: "How does SAML certificate rotation work?"

HyDE (hypothetical documents)

HyDE, from Gao et al., has the model write a hypothetical answer, embeds that answer, and searches with the embedding. The paper is explicit that the generated document "is unreal and may contain false details". That's acceptable because the text is only used to find a neighborhood in embedding space, and real documents are what get retrieved.

HyDE helps vector search when questions and documents are phrased very differently. It generates a full passage per query, so it spends more output tokens and time than a one-line rewrite.

Query routing

Routing doesn't change the words. It decides where the query goes: the docs index, a SQL table of plans and prices, the API reference, or no retrieval at all. Routing is a core part of agentic RAG, where an agent picks sources and can retry with a new query. The cost is a classification call, and a wrong route means retrieving from the wrong source entirely.

Technique

What changes

Helps most with

Extra cost per turn

Simple rewrite

One standalone query replaces the message

Follow-ups, chatty phrasing

1 short LLM call

Query expansion

Terms or a pseudo-answer added to the query

Vocabulary mismatch, keyword search

1 LLM call

Multi-query

Several phrasings, results merged

Low recall on varied wording

1 LLM call + N searches

Decomposition

Compound question split into parts

Multi-part and comparison questions

1 LLM call + N searches

Step-back

Adds a more general question

Questions needing background

1 LLM call + 1 search

HyDE

Searches with an embedded fake answer

Question/document phrasing gap in vector search

1 longer generation + 1 embedding

Routing

Picks the index or tool

Mixed sources (docs, SQL, APIs)

1 classification call

Worked example

We ran the follow-up from the intro through Qwen2.5-0.5B-Instruct on CPU with greedy decoding. The index holds six short chunks from a made-up project-tracking SaaS: a pricing FAQ, API pricing, enterprise pricing, the Team plan, the Free plan, and an SSO setup page. Retrieval is a plain BM25 keyword scorer.

The worked example teaches the transformation: combine a follow-up question with its conversation topic to produce a standalone search query. Its CSV-export topic is deliberately different from the Team-plan question. This is the prompt we ran:

Rewrite the user's last message as a standalone search query.
Use the conversation to fill in what the message refers to.
Keep product names, plan names, IDs and numbers exactly as written.
Do not answer the question. Return only the query.

Example
Conversation:
User: How do I export a board to CSV?
Assistant: Open the board menu and choose Export.
Last message: is there a row limit?
Standalone query: board CSV export row limit

Conversation:
User: Does the Team plan include SSO?
Assistant: Yes, SAML SSO is included on the Team plan.
Last message: what about the pricing?
Standalone query

Rewrite the user's last message as a standalone search query.
Use the conversation to fill in what the message refers to.
Keep product names, plan names, IDs and numbers exactly as written.
Do not answer the question. Return only the query.

Example
Conversation:
User: How do I export a board to CSV?
Assistant: Open the board menu and choose Export.
Last message: is there a row limit?
Standalone query: board CSV export row limit

Conversation:
User: Does the Team plan include SSO?
Assistant: Yes, SAML SSO is included on the Team plan.
Last message: what about the pricing?
Standalone query

Rewrite the user's last message as a standalone search query.
Use the conversation to fill in what the message refers to.
Keep product names, plan names, IDs and numbers exactly as written.
Do not answer the question. Return only the query.

Example
Conversation:
User: How do I export a board to CSV?
Assistant: Open the board menu and choose Export.
Last message: is there a row limit?
Standalone query: board CSV export row limit

Conversation:
User: Does the Team plan include SSO?
Assistant: Yes, SAML SSO is included on the Team plan.
Last message: what about the pricing?
Standalone query

Recorded rewrite:

pricing for team plans
pricing for team plans
pricing for team plans

And what that does to retrieval (top 3, BM25 scores):

Query

Rank 1

Rank 2

Rank 3

Original: "what about the pricing?"

api-pricing (0.88)

enterprise-pricing (0.79)

pricing-faq (0.79)

Rewritten: "pricing for team plans"

team-plan (1.32)

api-pricing (0.88)

enterprise-pricing (0.79)

The Team plan chunk goes from absent to first because the rewrite adds "team", which the original query lacks. That improves retrieval in this example, but the output is not fully compliant: it changes "Team plan" to "team plans" despite the exact-name instruction. "Team plan pricing" would preserve the name; it is not the output this run produced.

The rewrite isn't perfect, either. The prompt asked for plan names exactly as written, and the model still turned "Team plan" into "team plans". Our keyword scorer doesn't stem, so "plans" matched nothing and "team" alone carried the match. Small drift like that is one reason to keep the original query in the mix, which the next section does.

How to implement it

Here's the script for the worked example. To implement query rewriting against your own index, swap bm25_search for your retriever and the Hugging Face pipeline for whatever model you call. This local demo has no timeout or exception handling around generation. In production, bound that call and fall back to the original query if it fails.

# pip install torch transformers
import math, re
from collections import Counter
from transformers import pipeline

# A tiny docs index for a made-up project-tracking SaaS.
DOCS = {
    "pricing-faq": "Pricing FAQ: refund policy, taxes and currency.",
    "api-pricing": "API pricing: usage-based pricing per 1,000 API calls, billed monthly.",
    "enterprise-pricing": "Enterprise pricing is custom. Contact sales for a quote.",
    "team-plan": "Team plan: $12 per user per month billed annually. Includes SAML SSO and audit logs.",
    "sso-setup": "Set up SAML SSO: upload your identity provider metadata in Admin settings.",
    "free-plan": "Free plan: up to 3 users. SSO is not included.",
}
STOP = {"a", "an", "the", "what", "about", "is", "for", "of", "and", "how", "in", "does", "to", "your", "up", "per"}

def tokens(text):
    return [t for t in re.findall(r"[a-z0-9$]+", text.lower()) if t not in STOP]

def bm25_search(query, k=3, k1=1.2, b=0.75):
    docs = {d: tokens(t) for d, t in DOCS.items()}
    avg = sum(map(len, docs.values())) / len(docs)
    scores = {}
    for d, toks in docs.items():
        tf, s = Counter(toks), 0.0
        for q in set(tokens(query)):
            n = sum(q in t for t in docs.values())
            if tf[q]:
                idf = math.log(1 + (len(docs) - n + 0.5) / (n + 0.5))
                s += idf * tf[q] * (k1 + 1) / (tf[q] + k1 * (1 - b + b * len(toks) / avg))
        if s > 0:
            scores[d] = round(s, 2)
    return sorted(scores.items(), key=lambda x: (-x[1], x[0]))[:k]

REWRITE_PROMPT = """Rewrite the user's last message as a standalone search query.
Use the conversation to fill in what the message refers to.
Keep product names, plan names, IDs and numbers exactly as written.
Do not answer the question. Return only the query.

Example
Conversation:
User: How do I export a board to CSV?
Assistant: Open the board menu and choose Export.
Last message: is there a row limit?
Standalone query: board CSV export row limit

Conversation:
{history}
Last message: {question}
Standalone query:"""

llm = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")

def rewrite(history, question):
    prompt = REWRITE_PROMPT.format(history=history, question=question)
    out = llm([{"role": "user", "content": prompt}], max_new_tokens=32, do_sample=False)
    return out[0]["generated_text"][-1]["content"].strip().strip('"')

history = "User: Does the Team plan include SSO?\nAssistant: Yes, SAML SSO is included on the Team plan."
question = "what about the pricing?"

rewritten = rewrite(history, question)
print("Rewritten:", rewritten)
print("Original top 3: ", bm25_search(question))
print("Rewritten top 3:", bm25_search(rewritten))
# pip install torch transformers
import math, re
from collections import Counter
from transformers import pipeline

# A tiny docs index for a made-up project-tracking SaaS.
DOCS = {
    "pricing-faq": "Pricing FAQ: refund policy, taxes and currency.",
    "api-pricing": "API pricing: usage-based pricing per 1,000 API calls, billed monthly.",
    "enterprise-pricing": "Enterprise pricing is custom. Contact sales for a quote.",
    "team-plan": "Team plan: $12 per user per month billed annually. Includes SAML SSO and audit logs.",
    "sso-setup": "Set up SAML SSO: upload your identity provider metadata in Admin settings.",
    "free-plan": "Free plan: up to 3 users. SSO is not included.",
}
STOP = {"a", "an", "the", "what", "about", "is", "for", "of", "and", "how", "in", "does", "to", "your", "up", "per"}

def tokens(text):
    return [t for t in re.findall(r"[a-z0-9$]+", text.lower()) if t not in STOP]

def bm25_search(query, k=3, k1=1.2, b=0.75):
    docs = {d: tokens(t) for d, t in DOCS.items()}
    avg = sum(map(len, docs.values())) / len(docs)
    scores = {}
    for d, toks in docs.items():
        tf, s = Counter(toks), 0.0
        for q in set(tokens(query)):
            n = sum(q in t for t in docs.values())
            if tf[q]:
                idf = math.log(1 + (len(docs) - n + 0.5) / (n + 0.5))
                s += idf * tf[q] * (k1 + 1) / (tf[q] + k1 * (1 - b + b * len(toks) / avg))
        if s > 0:
            scores[d] = round(s, 2)
    return sorted(scores.items(), key=lambda x: (-x[1], x[0]))[:k]

REWRITE_PROMPT = """Rewrite the user's last message as a standalone search query.
Use the conversation to fill in what the message refers to.
Keep product names, plan names, IDs and numbers exactly as written.
Do not answer the question. Return only the query.

Example
Conversation:
User: How do I export a board to CSV?
Assistant: Open the board menu and choose Export.
Last message: is there a row limit?
Standalone query: board CSV export row limit

Conversation:
{history}
Last message: {question}
Standalone query:"""

llm = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")

def rewrite(history, question):
    prompt = REWRITE_PROMPT.format(history=history, question=question)
    out = llm([{"role": "user", "content": prompt}], max_new_tokens=32, do_sample=False)
    return out[0]["generated_text"][-1]["content"].strip().strip('"')

history = "User: Does the Team plan include SSO?\nAssistant: Yes, SAML SSO is included on the Team plan."
question = "what about the pricing?"

rewritten = rewrite(history, question)
print("Rewritten:", rewritten)
print("Original top 3: ", bm25_search(question))
print("Rewritten top 3:", bm25_search(rewritten))
# pip install torch transformers
import math, re
from collections import Counter
from transformers import pipeline

# A tiny docs index for a made-up project-tracking SaaS.
DOCS = {
    "pricing-faq": "Pricing FAQ: refund policy, taxes and currency.",
    "api-pricing": "API pricing: usage-based pricing per 1,000 API calls, billed monthly.",
    "enterprise-pricing": "Enterprise pricing is custom. Contact sales for a quote.",
    "team-plan": "Team plan: $12 per user per month billed annually. Includes SAML SSO and audit logs.",
    "sso-setup": "Set up SAML SSO: upload your identity provider metadata in Admin settings.",
    "free-plan": "Free plan: up to 3 users. SSO is not included.",
}
STOP = {"a", "an", "the", "what", "about", "is", "for", "of", "and", "how", "in", "does", "to", "your", "up", "per"}

def tokens(text):
    return [t for t in re.findall(r"[a-z0-9$]+", text.lower()) if t not in STOP]

def bm25_search(query, k=3, k1=1.2, b=0.75):
    docs = {d: tokens(t) for d, t in DOCS.items()}
    avg = sum(map(len, docs.values())) / len(docs)
    scores = {}
    for d, toks in docs.items():
        tf, s = Counter(toks), 0.0
        for q in set(tokens(query)):
            n = sum(q in t for t in docs.values())
            if tf[q]:
                idf = math.log(1 + (len(docs) - n + 0.5) / (n + 0.5))
                s += idf * tf[q] * (k1 + 1) / (tf[q] + k1 * (1 - b + b * len(toks) / avg))
        if s > 0:
            scores[d] = round(s, 2)
    return sorted(scores.items(), key=lambda x: (-x[1], x[0]))[:k]

REWRITE_PROMPT = """Rewrite the user's last message as a standalone search query.
Use the conversation to fill in what the message refers to.
Keep product names, plan names, IDs and numbers exactly as written.
Do not answer the question. Return only the query.

Example
Conversation:
User: How do I export a board to CSV?
Assistant: Open the board menu and choose Export.
Last message: is there a row limit?
Standalone query: board CSV export row limit

Conversation:
{history}
Last message: {question}
Standalone query:"""

llm = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")

def rewrite(history, question):
    prompt = REWRITE_PROMPT.format(history=history, question=question)
    out = llm([{"role": "user", "content": prompt}], max_new_tokens=32, do_sample=False)
    return out[0]["generated_text"][-1]["content"].strip().strip('"')

history = "User: Does the Team plan include SSO?\nAssistant: Yes, SAML SSO is included on the Team plan."
question = "what about the pricing?"

rewritten = rewrite(history, question)
print("Rewritten:", rewritten)
print("Original top 3: ", bm25_search(question))
print("Rewritten top 3:", bm25_search(rewritten))

Our CPU run printed:

Rewritten: pricing for team plans
Original top 3:  [('api-pricing', 0.88), ('enterprise-pricing', 0.79), ('pricing-faq', 0.79)]
Rewritten top 3: [('team-plan', 1.32), ('api-pricing', 0.88), ('enterprise-pricing', 0.79)]
Rewritten: pricing for team plans
Original top 3:  [('api-pricing', 0.88), ('enterprise-pricing', 0.79), ('pricing-faq', 0.79)]
Rewritten top 3: [('team-plan', 1.32), ('api-pricing', 0.88), ('enterprise-pricing', 0.79)]
Rewritten: pricing for team plans
Original top 3:  [('api-pricing', 0.88), ('enterprise-pricing', 0.79), ('pricing-faq', 0.79)]
Rewritten top 3: [('team-plan', 1.32), ('api-pricing', 0.88), ('enterprise-pricing', 0.79)]

Two habits reduce the risk of query rewriting. Keep the original query in the retrieval set so its matches enter the candidate pool, then merge with rank fusion instead of trusting one list. The final top-k cutoff can still drop original-query results, so fusion is not a guarantee against retrieval regressions.

Reciprocal rank fusion, from Cormack, Clarke and Bรผttcher, scores each document as the sum of 1/(k + rank) across lists, with k = 60. It's the same fusion step our guide to hybrid search uses to merge keyword and vector results, and it works the same way for an original query and its rewrites.

Add this below the script:

import re

def safe_rewrite(history, question, max_words=16):
    if not history:                # nothing to resolve, so skip the LLM call
        return None
    candidate = rewrite(history, question)
    # Preserve short uppercase-prefix IDs and dotted numbers matched below.
    literals = re.findall(r"\b[A-Z]{1,5}-?\d+\b|\b\d+(?:\.\d+)+\b", question)
    if not candidate or len(candidate.split()) > max_words:
        return None
    if any(lit not in candidate for lit in literals):
        return None
    return candidate

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, (doc_id, _) in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=lambda d: (-scores[d], d))

def retrieve(history, question, k=3):
    queries = [question]                      # always keep the original
    candidate = safe_rewrite(history, question)
    if candidate and candidate != question:
        queries.append(candidate)
    return queries, rrf([bm25_search(q, k=k) for q in queries])[:k]

print(retrieve(history, question))
import re

def safe_rewrite(history, question, max_words=16):
    if not history:                # nothing to resolve, so skip the LLM call
        return None
    candidate = rewrite(history, question)
    # Preserve short uppercase-prefix IDs and dotted numbers matched below.
    literals = re.findall(r"\b[A-Z]{1,5}-?\d+\b|\b\d+(?:\.\d+)+\b", question)
    if not candidate or len(candidate.split()) > max_words:
        return None
    if any(lit not in candidate for lit in literals):
        return None
    return candidate

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, (doc_id, _) in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=lambda d: (-scores[d], d))

def retrieve(history, question, k=3):
    queries = [question]                      # always keep the original
    candidate = safe_rewrite(history, question)
    if candidate and candidate != question:
        queries.append(candidate)
    return queries, rrf([bm25_search(q, k=k) for q in queries])[:k]

print(retrieve(history, question))
import re

def safe_rewrite(history, question, max_words=16):
    if not history:                # nothing to resolve, so skip the LLM call
        return None
    candidate = rewrite(history, question)
    # Preserve short uppercase-prefix IDs and dotted numbers matched below.
    literals = re.findall(r"\b[A-Z]{1,5}-?\d+\b|\b\d+(?:\.\d+)+\b", question)
    if not candidate or len(candidate.split()) > max_words:
        return None
    if any(lit not in candidate for lit in literals):
        return None
    return candidate

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, (doc_id, _) in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=lambda d: (-scores[d], d))

def retrieve(history, question, k=3):
    queries = [question]                      # always keep the original
    candidate = safe_rewrite(history, question)
    if candidate and candidate != question:
        queries.append(candidate)
    return queries, rrf([bm25_search(q, k=k) for q in queries])[:k]

print(retrieve(history, question))

Output:

(['what about the pricing?', 'pricing for team plans'], ['api-pricing', 'enterprise-pricing', 'team-plan'])
(['what about the pricing?', 'pricing for team plans'], ['api-pricing', 'enterprise-pricing', 'team-plan'])
(['what about the pricing?', 'pricing for team plans'], ['api-pricing', 'enterprise-pricing', 'team-plan'])

The fused top three include team-plan at rank 3, replacing pricing-faq from the original query's top three. Keeping the original query preserves its candidates before truncation, not every result in the final list. A downstream reranker can reorder the candidates it receives, but cannot recover candidates already cut off.

The guard skips the LLM call when there's no chat history, rejects empty or long rewrites, and falls back when a rewrite drops a literal matched by its regex. That regex covers short uppercase-prefix IDs and dotted numbers, not every identifier or amount; adapt it to your data and test it. It checks literal retention, not whether the rewrite preserves meaning.

If you're on a framework, you don't need to write this loop to implement query rewriting. LangChain, LlamaIndex and Haystack each ship multi-query or conversational rewriting components, covered in the frameworks section below.

When rewriting hurts

Rewriting adds a model between the user and your index, and that model can be wrong in ways that are hard to see in a demo.

  • Over-specific rewrites. A rewrite that adds a plan name, product version or date the user didn't mean narrows retrieval and drops relevant documents. Over-general rewrites fail the other way and pull in noise.

  • Drift from user intent. We also ran the same prompt without conversation history or the no-history check. The model turned "enterprise pricing" into "enterprise pricing row limit", borrowing "row limit" from the CSV-export example. The example helped resolve the follow-up but also contaminated a query that did not need rewriting.

  • Added latency and cost. Every rewrite is an extra model call on the critical path before retrieval. Measure that overhead on your target hardware and model. Hosted models add network time, and multi-query or HyDE multiply both calls and tokens.

  • Domain jargon. General-purpose models can "correct" terms they don't recognize. An internal product name, a ticket prefix or a legal term of art can get swapped for a common word, which sends retrieval to the wrong documents.

  • Exact identifiers. Microsoft's Azure AI Search docs warn that rewritten queries "might not contain all of the exact terms the original query had", which matters for IDs and product codes. That's the reason for the literal check in safe_rewrite.

Elastic's search team reached a similar conclusion in their own experiments. They kept the original query as a required clause and used LLM-generated terms only as optional boosts, after finding that replacing the original with a rewrite hurt their vector search results.

How to evaluate query rewriting

To evaluate query rewriting, run the same question set through retrieval with and without the rewrite. Then compare four things: recall@k (did a relevant chunk make the top k), answer quality, latency, and cost. Our guide to RAG evaluation metrics covers recall@k, MRR and answer-level scoring in more depth.

Build the question set from real traffic and label which chunk answers each one. Make sure it includes follow-ups that need history, standalone questions that are already fine, and queries with IDs or codes. Rewriting should help the first group, leave the second alone and never break the third. Here's a six-question version against the toy index that covers the first two groups, continuing the script. Add your ID queries the same way.

import statistics, time

EVAL = [  # (history, question, relevant doc)
    ("User: Does the Team plan include SSO?\nAssistant: Yes, SAML SSO is included on the Team plan.",
     "what about the pricing?", "team-plan"),
    ("User: How do I set up SSO?\nAssistant: Upload your identity provider metadata in Admin settings.",
     "is it on the free plan?", "free-plan"),
    ("User: Do you charge for API calls?\nAssistant: Yes, API usage is usage-based.",
     "how is that billed?", "api-pricing"),
    ("User: Is SAML supported?\nAssistant: Yes, on the Team plan.",
     "how do I configure it?", "sso-setup"),
    ("", "enterprise pricing", "enterprise-pricing"),
    ("", "refund policy", "pricing-faq"),
]

def recall_at(ranked, gold, k):
    return int(gold in ranked[:k])

totals = {"original": [0, 0], "rewrite only": [0, 0], "original + rewrite (RRF)": [0, 0]}
latencies = []
for hist, q, gold in EVAL:
    start = time.perf_counter()
    rw = safe_rewrite(hist, q) or q
    if hist:                                   # time only the turns that call the model
        latencies.append(time.perf_counter() - start)
    ranked = {
        "original": [d for d, _ in bm25_search(q, k=3)],
        "rewrite only": [d for d, _ in bm25_search(rw, k=3)],
        "original + rewrite (RRF)": rrf([bm25_search(q, k=3), bm25_search(rw, k=3)])[:3],
    }
    print(f"{q!r:28} -> {rw!r}")
    for name, docs in ranked.items():
        totals[name][0] += recall_at(docs, gold, 1)
        totals[name][1] += recall_at(docs, gold, 3)

for name, (r1, r3) in totals.items():
    print(f"{name:26} recall@1={r1}/{len(EVAL)} recall@3={r3}/{len(EVAL)}")
print(f"rewrite latency over {len(latencies)} turns: median {statistics.median(latencies):.2f}s, max {max(latencies):.2f}s")

Illustrative rewrites and their retrieval scores:
'what about the pricing?'    -> 'pricing for team plans'
'is it on the free plan?'    -> 'is it on the free plan?'
'how is that billed?'        -> 'how is that billed?'
'how do I configure it?'     -> 'configure SAML'
'enterprise pricing'         -> 'enterprise pricing'
'refund policy'              -> 'refund policy'
original                   recall@1=4/6 recall@3=4/6
rewrite only               recall@1=6/6 recall@3=6/6
original + rewrite (RRF)   recall@1=5/6 recall@3=6/6
import statistics, time

EVAL = [  # (history, question, relevant doc)
    ("User: Does the Team plan include SSO?\nAssistant: Yes, SAML SSO is included on the Team plan.",
     "what about the pricing?", "team-plan"),
    ("User: How do I set up SSO?\nAssistant: Upload your identity provider metadata in Admin settings.",
     "is it on the free plan?", "free-plan"),
    ("User: Do you charge for API calls?\nAssistant: Yes, API usage is usage-based.",
     "how is that billed?", "api-pricing"),
    ("User: Is SAML supported?\nAssistant: Yes, on the Team plan.",
     "how do I configure it?", "sso-setup"),
    ("", "enterprise pricing", "enterprise-pricing"),
    ("", "refund policy", "pricing-faq"),
]

def recall_at(ranked, gold, k):
    return int(gold in ranked[:k])

totals = {"original": [0, 0], "rewrite only": [0, 0], "original + rewrite (RRF)": [0, 0]}
latencies = []
for hist, q, gold in EVAL:
    start = time.perf_counter()
    rw = safe_rewrite(hist, q) or q
    if hist:                                   # time only the turns that call the model
        latencies.append(time.perf_counter() - start)
    ranked = {
        "original": [d for d, _ in bm25_search(q, k=3)],
        "rewrite only": [d for d, _ in bm25_search(rw, k=3)],
        "original + rewrite (RRF)": rrf([bm25_search(q, k=3), bm25_search(rw, k=3)])[:3],
    }
    print(f"{q!r:28} -> {rw!r}")
    for name, docs in ranked.items():
        totals[name][0] += recall_at(docs, gold, 1)
        totals[name][1] += recall_at(docs, gold, 3)

for name, (r1, r3) in totals.items():
    print(f"{name:26} recall@1={r1}/{len(EVAL)} recall@3={r3}/{len(EVAL)}")
print(f"rewrite latency over {len(latencies)} turns: median {statistics.median(latencies):.2f}s, max {max(latencies):.2f}s")

Illustrative rewrites and their retrieval scores:
'what about the pricing?'    -> 'pricing for team plans'
'is it on the free plan?'    -> 'is it on the free plan?'
'how is that billed?'        -> 'how is that billed?'
'how do I configure it?'     -> 'configure SAML'
'enterprise pricing'         -> 'enterprise pricing'
'refund policy'              -> 'refund policy'
original                   recall@1=4/6 recall@3=4/6
rewrite only               recall@1=6/6 recall@3=6/6
original + rewrite (RRF)   recall@1=5/6 recall@3=6/6
import statistics, time

EVAL = [  # (history, question, relevant doc)
    ("User: Does the Team plan include SSO?\nAssistant: Yes, SAML SSO is included on the Team plan.",
     "what about the pricing?", "team-plan"),
    ("User: How do I set up SSO?\nAssistant: Upload your identity provider metadata in Admin settings.",
     "is it on the free plan?", "free-plan"),
    ("User: Do you charge for API calls?\nAssistant: Yes, API usage is usage-based.",
     "how is that billed?", "api-pricing"),
    ("User: Is SAML supported?\nAssistant: Yes, on the Team plan.",
     "how do I configure it?", "sso-setup"),
    ("", "enterprise pricing", "enterprise-pricing"),
    ("", "refund policy", "pricing-faq"),
]

def recall_at(ranked, gold, k):
    return int(gold in ranked[:k])

totals = {"original": [0, 0], "rewrite only": [0, 0], "original + rewrite (RRF)": [0, 0]}
latencies = []
for hist, q, gold in EVAL:
    start = time.perf_counter()
    rw = safe_rewrite(hist, q) or q
    if hist:                                   # time only the turns that call the model
        latencies.append(time.perf_counter() - start)
    ranked = {
        "original": [d for d, _ in bm25_search(q, k=3)],
        "rewrite only": [d for d, _ in bm25_search(rw, k=3)],
        "original + rewrite (RRF)": rrf([bm25_search(q, k=3), bm25_search(rw, k=3)])[:3],
    }
    print(f"{q!r:28} -> {rw!r}")
    for name, docs in ranked.items():
        totals[name][0] += recall_at(docs, gold, 1)
        totals[name][1] += recall_at(docs, gold, 3)

for name, (r1, r3) in totals.items():
    print(f"{name:26} recall@1={r1}/{len(EVAL)} recall@3={r3}/{len(EVAL)}")
print(f"rewrite latency over {len(latencies)} turns: median {statistics.median(latencies):.2f}s, max {max(latencies):.2f}s")

Illustrative rewrites and their retrieval scores:
'what about the pricing?'    -> 'pricing for team plans'
'is it on the free plan?'    -> 'is it on the free plan?'
'how is that billed?'        -> 'how is that billed?'
'how do I configure it?'     -> 'configure SAML'
'enterprise pricing'         -> 'enterprise pricing'
'refund policy'              -> 'refund policy'
original                   recall@1=4/6 recall@3=4/6
rewrite only               recall@1=6/6 recall@3=6/6
original + rewrite (RRF)   recall@1=5/6 recall@3=6/6

For the illustrative rewrites shown, retrieval recovers the two missed follow-ups: "what about the pricing?" becomes "pricing for team plans", and "how do I configure it?" becomes "configure SAML". The other queries stay unchanged. These recall values are calculated from those fixed rewrites, not a model-quality benchmark. Run the script to measure your model's actual rewrites and latency.

On real traffic, you'd expect some rewrites to drift, which is one risk that keeping the original query in the fusion pool can reduce, but not eliminate. Report results per slice, since an average across all questions can hide a regression on the ID queries.

The rewrite isn't one knob, though. The prompt, the rewriter model, the number of generated queries, whether the original is kept, the retriever and its k, and the reranker all interact. A rewrite that helps a pure vector retriever might do nothing once you add hybrid search, and a better reranker might make three generated queries unnecessary. That's a grid of configurations, and running them one after another on the full question set gets slow.

This is where RapidFire fits. Our open-source engine runs multiple RAG configurations side by side on the same evaluation data. Each config's preprocess_fn receives the batch and the RAG spec before generation, and your code calls rag.get_context(batch_queries=...) inside it for retrieval. A rewrite drops in right before that call. Wrap a baseline function and a rewrite function in a List([...]) knob, and RFGridSearch expands them into separate runs. The spec accepts any LangChain BaseRetriever, so the same comparison works with your own retriever.

With online aggregation, metrics show up as running estimates with confidence intervals while the runs progress. You declare each metric's type for that, and shuffle the data if its order correlates with the metric.

You can then stop a weak config early or clone a promising one with Interactive Control Ops from the dashboard. In the open-source engine, those are manual actions. Converge automates them: it stops what isn't working, clones what is, and holds the experiment to the time and spend limits you set.

Frameworks that support it

Most RAG frameworks ship query rewriting components. Check the defaults, because they differ on whether the original query is kept.

Framework

Components

Original query kept by default?

LangChain (langchain-classic 1.0.8)

MultiQueryRetriever, create_history_aware_retriever, HypotheticalDocumentEmbedder

No for MultiQueryRetriever (include_original=False)

LlamaIndex (llama-index-core 0.14.25)

QueryFusionRetriever, HyDEQueryTransform, decompose transforms, condense-question chat engines, routers

Yes for QueryFusionRetriever and HyDEQueryTransform

Haystack (haystack-ai 3.5.0)

QueryExpander with MultiQueryTextRetriever or MultiQueryEmbeddingRetriever

Yes (include_original_query=True)

Elasticsearch

bool queries for expansion terms, RRF retriever for merging

You build the query, so you decide

Azure AI Search

Generative query rewrite (preview) in semantic ranking

Yes, original plus up to 10 rewrites

LangChain. The multi-query retriever, MultiQueryRetriever, generates queries, retrieves for each and returns the unique union, without rank fusion. For chat, create_history_aware_retriever passes the input straight to the retriever when there's no chat history and only calls the LLM when there is, the same gate as our safe_rewrite.

LlamaIndex. QueryFusionRetriever queries with the original plus generated queries (four in total by default) and supports reciprocal rank fusion with k = 60. HyDE, decomposition, and routing live in its query transformation and router modules.

Haystack. QueryExpander generates four expansions by default and pairs with the multi-query retrievers, which run every query, deduplicate the documents, and sort them by score.

Elasticsearch and Azure AI Search. With Elasticsearch, you call the model from your application. Then you either put the original and generated terms in a bool query, or run them as separate retrievers and merge with its RRF retriever, whose rank_constant defaults to 60. Azure AI Search does generate them: set queryRewrites to generative|count-N on a semantic query, and it searches with the original plus the rewrites. The feature is in preview and requires the semantic ranker.

Whichever framework you use, the question that matters is whether the rewrite helps on your traffic. To test it, pip install rapidfireai, follow the RAG walkthrough, and put a baseline and a rewrite config in the same experiment. If you're planning query optimization for a production RAG system, talk to our team.

FAQ

What is query rewriting in RAG?

Query rewriting in RAG transforms a user query into one or more search queries before retrieval. It resolves follow-ups using chat history, removes conversational filler, and adds terms your documents use. The generator still answers the original question, and only the retrieval query changes.

How to improve the accuracy of RAG?

Fix retrieval first, since the generator can't use relevant documents it never sees. Measure recall@k on labeled questions, then test changes one at a time: chunking, hybrid search, query rewriting, and reranking. Keep the changes that improve your metrics on real traffic, and watch latency and cost alongside quality.

How does RAG retrieve relevant data?

A RAG system splits documents into chunks and indexes them, usually as embeddings for vector search and often with a keyword index too. At query time, it searches with the user query or a rewritten query, takes the top k chunks, optionally reranks them, and passes them to the LLM as context.

What is query expansion in information retrieval?

Query expansion adds related terms to a query, such as synonyms, alternate spellings, or words from relevant documents, so it matches documents that describe the same thing differently. LLM-based expansion generates those terms or a short pseudo-answer and adds them to the original query instead of replacing it.