What Is a RAG Pipeline? Stages, Components, and How to Tune It

Suppose you're building a chatbot that answers questions from policy documents. You split the documents into chunks, index them for search, and pass retrieved passages to a language model. The prototype answers your test questions, but a question about a coverage limit gets an answer from the wrong section or misses the table containing the limit. Now you need to trace the answer back through the pipeline: what was parsed, what was retrieved, and what reached the model.

This guide covers the stages of a RAG system, a short Python example, failure points, and how to compare configurations using an eval set.

What Is a RAG Pipeline?

A RAG pipeline is the sequence of steps that turns your documents into searchable chunks and finds the most relevant ones for each user query. It then passes those chunks to a large language model, so the answer is grounded in your data instead of only in what the model memorized during training.

The name comes from the 2020 retrieval-augmented generation paper by Lewis et al., which paired a language model's built-in memory with a dense vector index searched by a neural retriever. Knowledge baked into weights is hard to update and can't reliably show where an answer came from, while external data in a knowledge base can be refreshed tonight and cited line by line. That separation lets you update the knowledge base without retraining the generator.

How a RAG Pipeline Works: Indexing and Querying

The dense-retrieval pipeline below has two phases that run at different times.

Indexing runs offline. You start with documents, parse them into clean text, chunk the text, embed each chunk, and write the vectors to a vector store. It reruns when your data changes.

Querying runs in real time. When a user query arrives, you embed it with the same model and retrieve the top-k most similar chunks. Then you rerank them, build a prompt around the best few, and generate the answer.

The split matters later, when you tune. Changing the parser, chunk size, or embedding model means rebuilding the index, while k, reranker depth, the prompt, and the generator are cheap to retest.

The Stages of a RAG Pipeline

Seven stages, each with its main decision and a common mistake. The linked guides go deeper.

Ingestion and Parsing

Ingestion pulls unstructured data from your data sources: PDFs, wiki pages, and support tickets. Parsing turns each file into text plus the metadata you'll need later, such as source, page, section, and who's allowed to see it. The main decision is which parser to trust for each file type. The trap is treating parsing as solved. A PDF table flattened into a string of numbers still embeds fine but retrieves badly, and no later stage can recover what the parser threw away.

Chunking

Text splitters break documents into pieces small enough to be retrieved individually and to fit in the model's context window. You choose chunk size, overlap, and where to split (characters, tokens, headings, or meaning). Too small and a chunk loses its context; too large and one chunk covers three topics, which blurs its embedding. Don't just keep whatever size the tutorial used. See our guide to chunking strategies for RAG.

Embedding

An embedding model turns each chunk into a vector so that text with similar meaning lands close together. Queries have to go through the same model that embedded the documents, or similarity scores stop meaning anything. You choose the model and number of dimensions, balancing quality against cost. The usual mistake is assuming a general-purpose model understands your jargon and part numbers. See our guide to choosing an embedding model.

Indexing and Storage

Your vector embeddings can live in a library like FAISS, in Postgres via pgvector, or in one of the managed vector databases such as Pinecone. The main decision is exact versus approximate search, since approximate indexes like HNSW trade some recall for speed. Some teams also build a knowledge graph, as GraphRAG does, to answer corpus-wide questions that chunk retrieval handles poorly. Storing vectors without the metadata you'll need for filtering and access control is the error that bites later.

Retrieval

Dense retrieval compares vectors, so it catches paraphrases. Lexical search such as BM25 matches the literal query words, which misses synonyms but finds exact product codes and error strings. The knobs are the search type (plain similarity, a minimum score threshold, or MMR for more diverse results) and k, the number of chunks returned. Dense search alone often misses when users type exact identifiers. Our guide to hybrid search covers running both together.

Reranking

First-pass retrieval is built for speed across millions of chunks, so it returns near misses. A reranker, usually a cross-encoder, reads the query and each candidate together and scores relevance more accurately, but it's too slow to run over the whole corpus. So you fetch a wide candidate set (k) and keep only the best few (top_n). A common shortcut is skipping reranking and shrinking k instead, which buys precision by giving up recall. More in our guide to cross-encoder reranking.

Prompt Assembly and Generation

The last stage wraps the retrieved chunks, instructions, and question into a prompt and calls the LLM. You decide how many chunks go in, their order, and the generator. Order matters: in the Lost in the Middle study, language models were often much better at using information at the start or end of the context than in the middle. Stuffing in ten chunks "just in case" can bury the one that answers the question.

A Simple RAG Pipeline in Python

Here's the full flow using LangChain, an in-memory vector store, and a sentence-transformers cross-encoder for reranking.

1. Use Python 3.12+ in a virtual environment and run the pip command below before saving the code as rag_demo.py.

2. Create docs/refunds.md containing "Annual plans may be refunded within 30 days of purchase."

3. Set OPENAI_API_KEY in your environment and use an OpenAI chat model your account can access.

Running python rag_demo.py embeds the files, retrieves and reranks chunks, and prints an answer with [refunds.md] as its source; the exact wording can vary.

Embedding and generation calls send the supplied text to OpenAI and incur API charges. The reranker runs locally and downloads its weights on first use. Treat this as a small demonstration, not production code.

# pip install langchain langchain-openai langchain-text-splitters sentence-transformers
from pathlib import Path

from langchain.chat_models import init_chat_model
from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from sentence_transformers import CrossEncoder

# 1. Load: one Document per file, keeping the source for citations
docs = [
Document(page_content=p.read_text(encoding="utf-8"), metadata={"source": p.name})
for p in Path("docs").glob("*.md")
]

# 2. Chunk (sizes below are characters, not tokens)
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(docs)
if not chunks:
raise ValueError("Add at least one non-empty Markdown file to docs/ before running.")

# 3. Embed and store (the same model embeds chunks and queries)
store = InMemoryVectorStore(OpenAIEmbeddings(model="text-embedding-3-small"))
store.add_documents(documents=chunks)

# 4. Retrieve a wide candidate set
question = "What is the refund window for annual plans?"
candidates = store.similarity_search(question, k=15)

# 5. Rerank and keep the best three
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
ranked = reranker.rank(question, [c.page_content for c in candidates], top_k=3)
context = "\n\n".join(
f"[{candidates[r['corpus_id']].metadata['source']}]\n"
f"{candidates[r['corpus_id']].page_content}"
for r in ranked
)

# 6. Generate a grounded answer
llm = init_chat_model("gpt-5.5")
prompt = (
"Answer using only the context below and cite sources in brackets. "
"If the answer isn't in the context, say so.\n\n"
f"Context:\n{context}\n\nQuestion: {question}"
)
print(llm.invoke(prompt).text)

Count the decisions hiding in that script: chunk_size, chunk_overlap, the embedding model, k=15, top_k=3, the reranker, the prompt, and the generator. They're sensible starting points, but nothing in the script tells you whether they suit your documents.

Where RAG Pipelines Break

When a RAG system gives wrong answers, work backward through the stages to check what evidence reached the generator.

Domain mismatch is well documented: on the 18 datasets in the BEIR benchmark, BM25 held up as a strong baseline while dense retrievers often underperformed outside their training domain. Leaked permissions are a security bug, and OWASP's 2025 Top 10 for LLM applications recommends permission-aware vector stores to prevent them.

How to Evaluate and Tune a RAG Pipeline

The stages are coupled. Chunk size changes what the embedder sees, the embedder changes what retrieval returns, and k changes what the reranker can work with. Testing isolated changes on a few outputs can miss interactions between those choices.

Use a repeatable eval loop:

1. Build a labeled eval set of real user queries, each with the correct answer and the chunks or pages that support it.

2. Score retrieval and answers separately. Precision, recall, NDCG, and MRR show whether the right chunks came back and how high they ranked. Correctness and faithfulness (whether every claim is supported by the retrieved context) show what the model did with them. Track latency and token cost too. Our guide to RAG evaluation metrics explains each one.

3. Sweep data preparation and retrieval first. Chunking, the index, search type, k, and reranking are where knowledge of your own data pays off, so test those before swapping generators or rewriting prompts.

4. Compare full configurations on the same eval set. Two chunk sizes × two embedding models × two rerank depths is already 8 configs.

5. Confirm the winner on held-out questions, and if your generator samples at nonzero temperature, run it more than once before trusting a small gap.

We built RapidFire AI for this workflow. The open-source LangChain RAG spec exposes chunking, embedding, indexing, retrieval, and reranking as configuration choices, and almost every choice can take a list of options instead of one value. The resulting configs run side by side on the same eval set, scored with metric functions you write.

Running estimates with confidence intervals update as each shard (slice) of the eval set finishes, so you can do more than stop early. Stop the weak configs, then clone the strong ones and change query-side knobs like search settings, reranker depth, the prompt, or the generator. Built-in search covers similarity, score threshold, and MMR; for hybrid search, you plug in your own LangChain retriever.

In our travel-insurance chatbot case study, we held the generator and embedder fixed and compared 40 retrieval configs. The winner used 512-token chunks, retrieved 10 candidates, and reranked down to 2, lifting the retrieval composite score from 0.439 to 0.627, about 43 percent on that benchmark. The 76 test questions were synthesized from the policy itself: this was a controlled demonstration.

In the second case, a global automotive manufacturer's team spent a couple of months benchmarking its own stacks. After a handful of days learning the data, we evaluated a 432-config grid in two days. On the team's 28-question eval set, the winner beat the stronger incumbent on 5 of 7 shared metrics but ranked about 282nd of 432 on faithfulness. No system cleared every internal pass target. It was the best config on that eval set by a composite score weighted toward answer correctness.

RAG Pipeline vs. Fine-Tuning vs. Agentic RAG

RAG isn't the only way to adapt a model.

A standard RAG pipeline always retrieves before it generates, which keeps it simple and fast. In agentic RAG, an LLM-powered agent decides when and how to retrieve as it reasons, trading predictability for flexibility. The approaches combine well, too: fine-tune for behavior, retrieve for facts. See RAG vs. fine-tuning and our explainer on agentic RAG.

Conclusion

A RAG pipeline is a chain of decisions, and its weakest stage caps the quality of every answer. Because the stages interact, a handful of demo questions won't reveal which one that is. Build an eval set, score retrieval and answers separately, and compare whole configurations, starting with how you prepare and retrieve your data.

To try that loop on real data, open the RapidFire AI RAG notebook on Colab. It builds a FiQA RAG pipeline and compares chunk sizes and reranker depths on retrieval metrics with no local install. You can also run the open-source package locally with pip install rapidfireai, and if you're planning this for a production retrieval stack, talk to our team.

RAG Pipeline FAQ

What Does RAG Stand For?

Retrieval-augmented generation: retrieve relevant information from external data sources, augment the prompt with it, then let the model generate the answer.

Can You Explain RAG to a Beginner?

Think of an open-book exam. A plain large language model answers from memory, while a RAG system looks up the relevant documents first. That makes it more likely to be right about your data and able to show its sources.

Is ChatGPT a RAG LLM?

Not by itself, since the underlying model answers from what it learned in training. But ChatGPT can search the web when a question needs current information and cite the pages it used, which is the same retrieve-then-generate pattern.

Are RAG Systems Outdated?

No. Context windows have grown, so some teams paste whole documents into the prompt instead, which works for small collections. But you pay for those tokens on every call, and this approach breaks down once the corpus outgrows the window. Long-context research has also found that models use information in the middle of the prompt less reliably.

Do You Need a Vector Database for RAG?

No. A prototype can keep vectors in memory, like the example above, and pgvector runs vector search inside Postgres. A dedicated vector database earns its place once you need scale, persistence, filtering, and access control together.