rapidfire.

Graph RAG vs Vector RAG: How to Choose in 2026

Written By:

Matt Tanner

Published on


Vector RAG retrieves similar shipment text; graph RAG follows shipment SHP-1042 through component PCB-7 to supplier Northwind Circuits.

Alt text (deployment only): Vector RAG retrieves similar shipment notes, while a graph follows shipment SHP-1042 through component PCB-7 to supplier Northwind Circuits.

Vector RAG is hard to beat on the questions most teams ship first. Embed your chunks into a vector database and search by similarity, and a question like "How long do customers have to send a unit back?" finds the returns policy. The policy never says "send back," and it doesn't need to. It's cheap to build, quick to refresh, and easy to debug.

Then someone asks, "Which supplier is behind the delay on shipment SHP-1042?" The answer is spread across three records: the shipment, the component it's waiting on, and that component's supplier. Only the first one reads anything like the question. A knowledge graph stores those links explicitly, and graph RAG follows them, but it also brings an extraction pipeline, a schema, and a second datastore beside your vector database that somebody has to keep current.

Like RAG vs fine-tuning, this is a decision you can test instead of argue about. Below: how each approach retrieves, where each one breaks, what the graph really costs, and how to compare both on your own questions.

Graph RAG vs Vector RAG at a Glance

Use vector RAG when the answer lives in a passage that resembles the question: single facts, policies, how-to lookups. Use graph RAG when the answer depends on relationships between records, such as multi-hop questions, dependency chains, and corpus-wide themes, and you can afford to maintain the graph. The two often run together, since many graph designs start from a vector match.

Vector RAG

Graph RAG

Retrieval method

Embed the question, return the nearest chunks by similarity

Match the question to entities, then traverse relationships in a knowledge graph (often seeded by a vector search)

Best for

Single-hop lookups, paraphrased questions, detail-heavy answers

Multi-hop questions, dependency and impact chains, corpus-wide summaries

Setup effort

Chunk, embed, load a vector database

Design the schema, extract entities and relationships (often with an LLM), merge duplicates, load a graph database

Cost

Embeddings at index time, one similarity search per query

LLM extraction at index time for text sources, plus refresh work; query cost depends on the design

Weaknesses

Misses links between records, weak on "across everything" questions

Extraction gaps and errors, upkeep, extra work on simple lookups

Explainability

Shows which chunks came back and their scores

Shows the path of records and edges behind the answer

How Vector RAG Works

A vector RAG system splits documents into chunks, turns each chunk into vector embeddings with an embedding model, and stores them in a vector database or index. At query time, it embeds the question with the same model and runs a similarity search over that index, often an approximate nearest neighbor index such as HNSW. The top k chunks go into the prompt. Our guide to the RAG pipeline walks through each stage.

Each chunk is scored against the question on its own. That's why vector search handles paraphrase so well: "send a unit back" and "return unopened units within 30 days" land close together in embedding space. It's also why the retriever has no idea that two chunks are related unless they happen to sound alike.

Before you blame the architecture, check the knobs. Chunk size, the embedding model, k, maximal marginal relevance (MMR) for diversity, metadata filters, and a cross-encoder reranker all change what reaches the model. In the controlled study we lean on below, reranking and iterative retrieval improved every method tested, vector RAG included. An untuned vector RAG baseline isn't a fair opponent for a graph.

Where Vector RAG Breaks Down

Three question shapes push vector-only RAG past what similarity can do.

Multi-hop questions. The SHP-1042 standup note says the shipment is "on hold until the PCB-7 controller boards arrive." The supplier for PCB-7 sits in a supplier table that never mentions SHP-1042 or a delay. Similarity search returns the note and stops there. Raising k might drag the supplier row in by luck, and it'll drag in a lot of unrelated text with it.

Relationships across documents. "Which open orders depend on parts from Northwind Circuits?" means joining a supplier list, component records, and order records. Vector search returns chunks that mention Northwind. It can't walk from Northwind to the components to the orders, because nothing in the index stores those links.

Global summaries. "What drove late shipments this quarter?" has no single best chunk. The GraphRAG paper from Microsoft Research says RAG fails on global questions such as "What are the main themes in the dataset?" Answering one means summarizing the whole corpus, which top-k retrieval was never built to do.

These are failures of multi-hop reasoning and aggregation, and they aren't universal. In a controlled benchmark, Han et al. found vector RAG did better on single-hop, detail-oriented questions and GraphRAG did better on multi-hop ones. Even on the MultiHop-RAG dataset, 13.6% of queries were answered correctly only by GraphRAG and 11.6% only by RAG with Llama 3.1-8B. Each approach wins questions the other loses.

How Graph RAG Works

Graph RAG retrieves through relationships between records, stored in a knowledge graph, instead of or on top of similarity. The graph comes from one of two places.

Structured data you already have. Shipments, components, and suppliers often live in ERP tables with foreign keys. Load them into a graph database such as Neo4j and query them with Cypher, Neo4j's declarative query language. There's no LLM extraction step, and the edges are as accurate as the source system.

Relationships extracted from text. Microsoft's GraphRAG project shows how this works. Its indexer slices the corpus into TextUnits, uses an LLM to extract entities and relationships (and optionally claims), runs community detection with the hierarchical Leiden technique, and writes summaries for each community from the bottom up. The standard pipeline also embeds text into a vector store.

At query time, a typical graph retriever identifies the entities in the question, matches them to nodes and runs a graph traversal outward from there. Microsoft's design offers four query modes:

  • Local search fans out from specific entities to their neighbors and combines that graph data with raw text chunks.

  • Global search answers corpus-wide questions from the community summaries.

  • DRIFT search adds community context to local search.

  • Basic search is plain top-k vector retrieval.

For SHP-1042, the traversal is short: Shipment SHP-1042, WAITING_ON component PCB-7, SUPPLIED_BY Northwind Circuits. The LLM gets the path plus the passages behind it. Store where every edge came from. In our test graph, the SUPPLIED_BY edge carries suppliers.csv#88, so the answer can cite the supplier record instead of asking the reader to trust the graph.


Graph RAG vs Vector RAG: How to Choose in 2026 โ€” diagram 2

Vector retrieval matches text similarity; graph retrieval follows explicit relationships between the shipment, component, and supplier.

Alt text (deployment only): Two retrieval paths for the same supplier question: vector search returns shipment-related passages; graph traversal follows WAITING_ON and SUPPLIED_BY relationships and supplies the path and source passages to an LLM.

Where Graph RAG Struggles

The graph earns its keep on connected questions. Everything else about it costs money and attention.

Graph construction cost. For Standard GraphRAG, Microsoft estimates graph extraction at roughly 75% of indexing cost. Its cheaper FastGraphRAG method swaps LLM extraction for NLP noun phrases and co-occurrence edges, and the docs say the resulting graph is noisier. Han et al. measured the gap on MultiHop-RAG: 135 seconds to build the RAG index versus 7,702 seconds for KG-GraphRAG and 5,560 seconds for Community-GraphRAG.

Extraction errors. An LLM-built knowledge graph only contains the entities and relationships the extractor caught. In the same study, with graphs built by GPT-4o-mini, only about 65.8% of answer entities made it into the knowledge graph for HotpotQA. A stronger extraction model helped, at a higher price. A missing entity means there's nothing for graph traversal to land on.

Maintenance. When Northwind stops supplying PCB-7, the edge has to change, along with any community summary that mentions it. GraphRAG ships an update command for existing indexes, but someone still has to own the schedule, check the diffs, and handle deletions. Edges without a recorded source are the hardest to fix, because nobody can tell which document put them there.

Latency on simple queries. It depends on the design. In Han et al.'s timing on MultiHop-RAG, KG-GraphRAG's retrieval took 14,434 seconds against 1,724 for RAG, mostly because of LLM-based entity expansion and multi-step traversal. Community-GraphRAG came in under RAG at 1,249 seconds. Microsoft calls its own global search resource-intensive. For "How long do customers have to send a unit back?", any extra hop is cost without benefit.

Key Differences

The table has the short version. Here's what sits behind those rows:

  • Data type: vector RAG works on unstructured text chunks. Graph RAG needs entities and relationships, either loaded from structured systems or extracted from text.

  • Representation: in vector RAG, each chunk becomes one point in embedding space. A property graph stores nodes and typed edges with properties, such as an edge's source record.

  • Retrieval: nearest neighbors by semantic similarity versus graph traversal along edges, often seeded by a vector search.

  • Cost: embedding at index time versus LLM extraction plus ongoing upkeep. Per-query cost varies by graph design.

  • Explainability: a ranked list of chunks versus a path (shipment, component, supplier) where each hop can point to its source.

When to Use Which

Start from the questions people actually ask, not from the architecture.

Question pattern

Example

Start with

Single-fact lookup

"How long do customers have to send a unit back?"

Vector RAG

Exact codes and IDs

"What does error E4012 mean?"

Keyword plus vector search

Multi-hop relationship

"Which supplier is behind the delay on SHP-1042?"

Vector seed plus graph expansion

Supply chain impact

"Which open orders need parts from Northwind Circuits?"

Graph traversal

Compliance mapping

"Which controls cover clause 4.2, and who owns them?"

Graph, if the mapping exists as data

Corpus-wide themes

"What drove late shipments this quarter?"

Community summaries (global search) or map-reduce over filtered records

Data shape matters as much as question shape. If the relationships already exist in a system of record (ERP, CMDB, a control matrix), loading them into a graph database is cheaper and more accurate than extracting them from prose. If they only exist in prose, you're paying for extraction and inheriting its error rate.

So do team skills. Somebody has to own the schema, write and tune Cypher, maintain extraction prompts, and run refresh jobs. If nobody will, vector-only RAG with good chunking and a reranker is the safer bet, even with a few multi-hop questions in the mix.

Hybrid Approaches

This isn't the same thing as hybrid search, which fuses keyword and vector rankings. Hybrid graph RAG combines vector retrieval with graph structure, and you can wire the two together in three ways.

Vector first, then graph expansion. Vector search finds seed passages, then a graph traversal attaches the related records. Here's that pattern as a LangChain retriever over Neo4j:

This is an integration template for an existing graph, not a complete database setup. Install langchain-core and the Neo4j Python driver, create the note_embedding vector index on your Note embeddings, and load the Note, Shipment, Component, and Supplier nodes and relationships shown below. Use the same embedding model and dimensions for the index and query embedder. The second snippet also needs rapidfireai and your loader, embedding configuration, and Neo4j connection settings.

from langchain_core.documents import Document
from langchain_core.embeddings import Embeddings
from langchain_core.retrievers import BaseRetriever
from neo4j import GraphDatabase

HYBRID_QUERY = """
CYPHER 25
MATCH (note:Note)
  SEARCH note IN (VECTOR INDEX note_embedding FOR $embedding LIMIT $k) SCORE AS score
OPTIONAL MATCH (note)-[:MENTIONS]->(s:Shipment)-[:WAITING_ON]->(c:Component)
               -[e:SUPPLIED_BY]->(sup:Supplier)
RETURN note.text AS text, note.source AS source, score,
       collect(s.id + " waits on " + c.name + ", supplied by "
               + sup.name + " [" + e.source + "]") AS links
ORDER BY score DESC
"""

class SupplierPathRetriever(BaseRetriever):
    embedder: Embeddings
    uri: str
    auth: tuple[str, str]
    k: int = 4
    _driver = None  # opened on first query so the retriever can be pickled

    def _get_relevant_documents(self, query, *, run_manager):
        if self._driver is None:
            self._driver = GraphDatabase.driver(self.uri, auth=self.auth)
        rows, _, _ = self._driver.execute_query(
            HYBRID_QUERY, embedding=self.embedder.embed_query(query), k=self.k)
        return [Document(page_content="\n".join([r["text"], *r["links"]]),
                         metadata={"source": r["source"]}) for r in rows]
from langchain_core.documents import Document
from langchain_core.embeddings import Embeddings
from langchain_core.retrievers import BaseRetriever
from neo4j import GraphDatabase

HYBRID_QUERY = """
CYPHER 25
MATCH (note:Note)
  SEARCH note IN (VECTOR INDEX note_embedding FOR $embedding LIMIT $k) SCORE AS score
OPTIONAL MATCH (note)-[:MENTIONS]->(s:Shipment)-[:WAITING_ON]->(c:Component)
               -[e:SUPPLIED_BY]->(sup:Supplier)
RETURN note.text AS text, note.source AS source, score,
       collect(s.id + " waits on " + c.name + ", supplied by "
               + sup.name + " [" + e.source + "]") AS links
ORDER BY score DESC
"""

class SupplierPathRetriever(BaseRetriever):
    embedder: Embeddings
    uri: str
    auth: tuple[str, str]
    k: int = 4
    _driver = None  # opened on first query so the retriever can be pickled

    def _get_relevant_documents(self, query, *, run_manager):
        if self._driver is None:
            self._driver = GraphDatabase.driver(self.uri, auth=self.auth)
        rows, _, _ = self._driver.execute_query(
            HYBRID_QUERY, embedding=self.embedder.embed_query(query), k=self.k)
        return [Document(page_content="\n".join([r["text"], *r["links"]]),
                         metadata={"source": r["source"]}) for r in rows]
from langchain_core.documents import Document
from langchain_core.embeddings import Embeddings
from langchain_core.retrievers import BaseRetriever
from neo4j import GraphDatabase

HYBRID_QUERY = """
CYPHER 25
MATCH (note:Note)
  SEARCH note IN (VECTOR INDEX note_embedding FOR $embedding LIMIT $k) SCORE AS score
OPTIONAL MATCH (note)-[:MENTIONS]->(s:Shipment)-[:WAITING_ON]->(c:Component)
               -[e:SUPPLIED_BY]->(sup:Supplier)
RETURN note.text AS text, note.source AS source, score,
       collect(s.id + " waits on " + c.name + ", supplied by "
               + sup.name + " [" + e.source + "]") AS links
ORDER BY score DESC
"""

class SupplierPathRetriever(BaseRetriever):
    embedder: Embeddings
    uri: str
    auth: tuple[str, str]
    k: int = 4
    _driver = None  # opened on first query so the retriever can be pickled

    def _get_relevant_documents(self, query, *, run_manager):
        if self._driver is None:
            self._driver = GraphDatabase.driver(self.uri, auth=self.auth)
        rows, _, _ = self._driver.execute_query(
            HYBRID_QUERY, embedding=self.embedder.embed_query(query), k=self.k)
        return [Document(page_content="\n".join([r["text"], *r["links"]]),
                         metadata={"source": r["source"]}) for r in rows]

This integration example targets a toy shipment graph on Neo4j Community 2026.10.1. The OPTIONAL MATCH matters: notes with no supplier path still come back as plain passages, so the graph adds context without hiding vector search hits. The SEARCH clause needs Neo4j 2026.01 or later and Cypher 25. The older db.index.vector.queryNodes procedure was deprecated in 2026.04.

The ORAN benchmark tested this shape, with vector chunks first and graph context after, over 74 telecom specification documents. Hybrid had the best factual correctness (0.58 versus 0.50 for graph and 0.48 for vector) but the lowest context relevance (0.45). Most of those gaps are smaller than the reported standard deviations across difficulty levels; the questions were multiple choice, and latency was left for future work.

Graph as a reranking signal. Retrieve with vectors as usual, then boost chunks whose entities sit within one or two hops of the entities in the question. It's the least invasive way to try a graph: the vector index stays the only retrieval path, so a stale graph hurts ranking instead of dropping answers.

Route or run both. Han et al. tested both: routing each query to RAG or GraphRAG (Selection) and running both and combining the context (Integration). On MultiHop-RAG with Llama 3.1-70B, Selection beat the best baseline by 1.1% and Integration by 6.4%. Integration pays for both retrievers on every query.

How to Test Both on Your Data

Published benchmarks on vector RAG vs graph RAG disagree because their corpora, questions, and baselines differ. Your question mix decides, so test on it.

  1. Build a labeled question set. Pull real questions from logs and tag each one as single-hop, multi-hop, or corpus-wide. For each, list the records a correct answer needs: the SHP-1042 note, the PCB-7 component, the supplier row. Include a few questions whose links don't exist in your data, because no retriever fixes a data gap.

  2. Tune the vector RAG baseline first. Sweep k, MMR, and a reranker before you compare anything against it.

  3. Match context budgets. In Han et al.'s setup, Community-GraphRAG used 9,770 retrieved tokens on MultiHop-RAG to RAG's 3,631. With Llama 3.1-70B, giving RAG more chunks lifted its overall score from 65.77 to 71.01, next to GraphRAG's 71.17, though the graph kept a lead on temporal questions (49.06 versus 43.74). Run one comparison as shipped and one with matched budgets: the first tells you what to deploy, the second tells you why it won.

  4. Score retrieval and answers separately. Check whether the needed records reached the context, then whether the answer was right. Our guide to RAG evaluation metrics covers context recall and answer correctness. If an LLM judges summaries side by side, swap the order, since Han et al. found strong position bias.

  5. Track cost. Record build time and extraction tokens, latency and tokens per query, and the time a refresh takes.

RapidFire's open-source engine runs this comparison side by side on one machine. Define each candidate as a RAG spec. The graph candidate passes the retriever above through retriever=, which takes any LangChain BaseRetriever and uses it as is. The baseline keeps the default FAISS store, an HNSW index on CPU:

from rapidfireai.automl import RFLangChainRagSpec

# loader, embedding_cfg, embedder and the Neo4j settings come from your setup
def with_source(doc):
    return f"[{doc.metadata['source']}] {doc.page_content}"

vector_rag = RFLangChainRagSpec(
    document_loader=loader,          # the notes the graph was extracted from
    embedding_cfg=embedding_cfg,     # same embedding model for both candidates
    vector_store_cfg={"type": "faiss"},
    search_cfg={"type": "similarity", "k": 4},
    document_template=with_source,
)

graph_rag = RFLangChainRagSpec(
    retriever=SupplierPathRetriever(embedder=embedder, uri=NEO4J_URI,
                                    auth=NEO4J_AUTH, k=4),
    document_template=with_source,
)
from rapidfireai.automl import RFLangChainRagSpec

# loader, embedding_cfg, embedder and the Neo4j settings come from your setup
def with_source(doc):
    return f"[{doc.metadata['source']}] {doc.page_content}"

vector_rag = RFLangChainRagSpec(
    document_loader=loader,          # the notes the graph was extracted from
    embedding_cfg=embedding_cfg,     # same embedding model for both candidates
    vector_store_cfg={"type": "faiss"},
    search_cfg={"type": "similarity", "k": 4},
    document_template=with_source,
)

graph_rag = RFLangChainRagSpec(
    retriever=SupplierPathRetriever(embedder=embedder, uri=NEO4J_URI,
                                    auth=NEO4J_AUTH, k=4),
    document_template=with_source,
)
from rapidfireai.automl import RFLangChainRagSpec

# loader, embedding_cfg, embedder and the Neo4j settings come from your setup
def with_source(doc):
    return f"[{doc.metadata['source']}] {doc.page_content}"

vector_rag = RFLangChainRagSpec(
    document_loader=loader,          # the notes the graph was extracted from
    embedding_cfg=embedding_cfg,     # same embedding model for both candidates
    vector_store_cfg={"type": "faiss"},
    search_cfg={"type": "similarity", "k": 4},
    document_template=with_source,
)

graph_rag = RFLangChainRagSpec(
    retriever=SupplierPathRetriever(embedder=embedder, uri=NEO4J_URI,
                                    auth=NEO4J_AUTH, k=4),
    document_template=with_source,
)

Then:

  • Keep everything else identical. Attach each spec with rag= to otherwise identical generator configs, use the same preprocess_fn (which calls rag.get_context()) and the same compute_metrics_fn, then pass both configs to run_evals() as a list.

  • Watch it as it runs. Metrics update shard by shard with confidence intervals once your metric functions declare each metric's type and range. With an API generator and MLflow on, RapidFire also logs average query latency, total tokens, and projected spend, and the MLflow trace view lists every retriever call with the documents it returned, so you can see whether the supplier row made it in.

  • Act mid-run. Stop a candidate that's clearly losing, or Clone-Modify the baseline's search_cfg to raise k for the matched-budget run. If RFOptuna drives a knob search, it can prune underperforming runs for you. Clone-Modify only edits query-processing knobs, so define the graph candidate up front.

  • Keep the retriever picklable. RapidFire pickles the retriever for its Ray workers. That's why the driver above opens on first query: in our test, a retriever holding an open driver failed both pickle and deepcopy.

RapidFire doesn't build or host the graph. You bring the graph store and the retriever. If you'd rather not babysit the run, Converge stops weak versions, clones strong ones, and launches new variants within the time and spend limits you set.

To try it, the RAG and context engineering walkthrough starts with a Colab notebook, and if you use only OpenAI APIs for embeddings and generation, you won't need a GPU. If you're planning this for a production system, talk to our team.

FAQ

When to use vector RAG vs graph RAG?

Use vector RAG when answers sit in passages that resemble the question, such as policies, docs, and single facts. Use graph RAG when the right passages come back, but the link between records is missing, and someone can keep those links current. If you can't tell which case you're in, test both on a labeled question set.

What are the key differences between graph RAG, vector RAG, and Agentic RAG?

Vector RAG and graph RAG differ in what they retrieve: similar chunks versus connected records. Agentic RAG differs in who drives retrieval: an agent decides whether to search, what to search for, and which tools to use. An agent can call a vector retriever, a graph retriever, or both.

What is graph RAG vs RAG?

Graph RAG is RAG that pulls context through relationships stored in a knowledge graph or graph database, instead of or on top of similarity search. "RAG" on its own usually means vector retrieval over text chunks, which Microsoft's GraphRAG docs call baseline RAG.

Is vectorless RAG better?

Not by default. "Vectorless RAG" means retrieval without a vector index. PageIndex, for example, builds a tree index of each document and has an LLM reason its way through it. That swaps similarity search for LLM reasoning at query time, so put it through the same question set, matched budgets, and cost tracking as any other retriever.