RAG Metadata Filtering: How It Works and How to Set It Up
Written By:
Matt Tanner
Published on
Consider five short chunks of API documentation and the question, "How do I upload a file with the v2 API?" An unfiltered similarity search can rank v1 instructions or another customer's note above the matching v2 chunk.
That need not mean the embedding is broken: v1 and v2 upload docs can be semantically similar. Vector similarity cannot reliably enforce version, tenant, date, and permission constraints, so if your RAG pipeline needs to respect them, they have to travel with each chunk as metadata and be enforced at query time.
This guide covers the whole loop: what to store, how to attach it at ingestion, how to turn a question into a filter, how pre- and post-filtering change the results, and how to test whether any of it helped.
What Is Metadata Filtering in RAG?
Metadata filtering in RAG limits retrieval to chunks whose stored attributes match a condition, such as version == 2 or tenant == "acme", so vector similarity only ranks chunks that pass. Similarity decides which chunks are closest to the question, and metadata filters decide which ones are allowed in the results at all. Ideally, the search space shrinks to the matching subset, though, as the pre-filtering section shows, not every store applies the filter before the search.
In practice, a filter is a small structured object sent alongside the query vector. Pinecone, Chroma, and LangChain's FAISS wrapper all accept the MongoDB-style form {"$and": [{"tenant": {"$eq": "acme"}}, {"version": {"$eq": 2}}]}, while Milvus and pgvector express the same idea as a boolean expression or a SQL WHERE clause.
Why Vector Search Alone Returns the Wrong Documents
Embeddings compress meaning into a vector, and most of the facts that decide whether a chunk is usable (version, date, department, language, access rights) aren't meaning. "Upload a file with POST /v1/files" and "upload a file with POST /v2/uploads" land close together because they describe the same task. The model has no reason to push them apart.
That gap shows up in a handful of predictable ways in RAG applications:
Version: user queries about v2 retrieve v1 instructions, as in the test above.
Date: a question about the current refund policy pulls the 2023 policy, which is worded almost the same.
Tenant or department: a shared index returns another customer's notes, which is a correctness bug and a data exposure bug at once.
Access rights: a chunk from a document the user can't open in the source system still ranks, because the index never learned who could open it.
Reranking and better chunking can improve ordering inside the candidate set, but they can't know that a chunk belongs to the wrong customer.
Types of Metadata That Help RAG
When people ask about types of metadata, the standard answer comes from library science. NISO's Understanding Metadata primer splits metadata into descriptive, structural and administrative types, with administrative covering technical, preservation and rights information, and it adds markup languages as a final category. For retrieval, those categories map onto fields you can actually filter on:
Type | What it describes | RAG fields that use it | Typical filter |
|---|---|---|---|
Descriptive | What the content is about | title, author, product, topic tags | |
Structural | Where a chunk sits in its source | section, heading path, page number | |
Administrative | How the content is managed | created/updated date, version, owner, permissions |
|
Source | Where the chunk came from | system, URL, file name, document ID |
Source isn't a NISO category, but in RAG it earns its own row because it drives citations and lets you drop a whole system from retrieval when its content goes stale. Custom metadata, like customer tier or jurisdiction, usually fits under descriptive or administrative.
How to Design a Metadata Schema
Start from the questions people ask, not from the fields your documents happen to have. Pull 50 to 100 real user queries and mark every phrase that implies a constraint ("in v2," "for EU customers," "since the March release"). Each recurring constraint becomes a candidate metadata field, and fields nobody filters on are just one more thing that can be wrong at ingestion.
For each of the metadata fields that survive, decide four things:
Type. Store versions as integers and dates as dates or epoch numbers, so range operators work. Pinecone, for example, only accepts numbers for
$gtand$lt.Allowed values. Use a controlled vocabulary (
upload-api, not "Upload API" in one source and "uploads" in another).Who sets it. Some fields come from the source system, some from parsing, some from an LLM. Write that down per field, because it decides how much you trust the value.
Whether a model may filter on it. Relevance fields (product, version, date) can come from the question. Access fields (tenant, group, clearance) must come from the authenticated session.
NVIDIA's RAG Blueprint enforces this with a typed schema per collection: a file containing an undefined metadata field fails ingestion. Rejecting that file early is easier to debug than a filter that silently matches nothing.
Adding Metadata at Ingestion
Metadata has to be on the chunk before it's embedded and stored, and it usually comes from three places:
From the source system. Version, owner, permissions, and timestamps already exist in your CMS, ticketing tool, or file store. Copy them across with the document ID so you can resync later. Amazon Bedrock Knowledge Bases takes this route for S3 data sources, reading metadata from companion
fileName.extension.metadata.jsonfiles.From parsing. Section headings, page numbers, and table captions come out of the document structure. If you split with semantic chunking or any other splitter, check that the parent document's fields land on every child chunk.
From an LLM. When the source has no structure, LLM metadata extraction fills the gap: a model reads the text and returns field values. Haystack's
LLMMetadataExtractordoes this and parks documents it fails on in afailed_documentslist, and LlamaIndex ships extractors such asTitleExtractorandEntityExtractor.
Treat extracted metadata as a guess until it passes the same checks you'd run on a filter: field allowlist, type, and allowed values. A wrong version tag is worse than a missing one, because the filter will confidently hide the right chunk.
Filtering at Query Time
Filters reach a query in three ways, often combined:
Explicit filters from the UI. The user picks a product and version from dropdowns, and the app passes them straight through. This is the most reliable option.
Rule-based filters. The app adds filters it already knows, like the tenant from the login session or
status == "published", plus simple patterns such as a regex for "v2" in the question.LLM-generated filters from natural language. A model reads the natural language question plus a description of the metadata fields and proposes a filter. LangChain's self-querying retriever (
SelfQueryRetriever) and LlamaIndex'sVectorIndexAutoRetrieverwork this way, and Amazon Bedrock Knowledge Bases offers it as implicit filtering, which "generates and applies a retrieval filter based on the user query and a metadata schema."
NVIDIA's RAG docs file this third option under advanced metadata filtering, and it's where most of the risk sits. Treat the model's output as a proposal. Parse it, check every field and value against the schema, and add the access fields yourself from the session. Here's a short, store-agnostic version:
A proposal of {"tenant": "globex"} raises field not allowed: tenant, and {"version": "2"} fails because the string "2" isn't the integer 2. When validation fails, send the error back to the model once or ask the user a clarifying question. Don't quietly drop the bad field and search anyway.
Save the first snippet as filters.py beside the script below. This example uses chromadb 1.5.9 and fastembed 0.9.1; install them with pip install chromadb==1.5.9 fastembed==0.9.1. The BAAI/bge-small-en-v1.5 embedding model downloads on first use. The model's filter proposal is a fixed string standing in for an LLM call:
The example output above illustrates the filter: although the query asks for three results, only two chunks match. Exact similarity ordering can vary. A filter changes how many results you get, not just which ones.
[Image placeholder: Left-to-right flow diagram. A separate left column labeled "Ingestion" shows: Source documents, then Chunk + attach metadata (tenant, product, version, date), then Vector store. Main flow on the right: User question, then Filter extractor (LLM) producing a proposed filter, then Validate against schema (with a side arrow labeled "reject or clarify" pointing back to the user), then two parallel boxes entering the vector store together: "Metadata filter (tenant from session + validated fields)" and "Vector query", then Vector store (the same box fed by the ingestion column), then Filtered chunks, then Generator, then Answer. Alt text: RAG metadata filtering flow from question through LLM filter extraction and validation to a filtered vector search, with metadata tagged at ingestion.]
Pre-Filtering vs. Post-Filtering
Where the filter runs relative to the nearest-neighbor search decides whether you get k results, fewer, or none. Pre-filtering restricts candidates before or during the vector search. Post-filtering runs the vector search first and then drops chunks that fail the filter, so a selective filter can leave you with almost nothing.
| Pre-filtering | Post-filtering |
|---|---|---|
When the filter runs | Before or during the ANN search | After the ANN search returns candidates |
Result count | Can return k matching results; approximate-search limits still apply | Can return fewer than k, or zero |
Recall on selective filters | Holds up | Drops as the filter gets narrower |
Latency | On HNSW, a narrow filter can make the search slower | Skips the cost of filtered graph traversal |
Best fit | Narrow filters such as tenant, version, permissions | Broad filters that match most chunks |
How common stores handle it:
Elasticsearch: a filter inside the
knnclause "is applied during approximate kNN search to ensure that k matching documents are returned," while a filter placed outside the kNN clause post-filters. The filtered kNN docs also note that on HNSW, "filters can make a search slower rather than faster," so don't promise faster responses as a side effect.Milvus: standard filtering applies the filter before the ANN search, with an iterative filtering mode for very complex expressions.
pgvector: with approximate indexes, the
WHEREclause runs after the index scan. The pgvector README gives the numbers: a condition matching 10% of rows at the defaulthnsw.ef_searchof 40 returns about 4 rows. Version 0.8.0 added iterative index scans (SET hnsw.iterative_scan = strict_orderorrelaxed_order), which keep scanning up tohnsw.max_scan_tuples, 20,000 by default.LangChain's FAISS wrapper: similarity search fetches
fetch_kcandidates, 20 by default, and filters those.
That last one is easy to hit in a prototype. For a synthetic example, index 1,000 chunks with deterministic two-dimensional vectors and place one tenant's 20 chunks beyond the first 20 candidates. A k=5 query with that tenant filter has no eligible results among the default 20 candidates; raising fetch_k to 1,000 makes enough matching candidates available.
This demonstrates candidate starvation, not its frequency on real embeddings. There's no error or warning, just an empty context, so log the returned-chunk count for every filtered query.
Metadata Filtering With Hybrid Search and Agents
Hybrid search runs a keyword retriever such as BM25 next to vector search and fuses the two ranked lists. If only the vector side is filtered, the keyword side can pull a v1 chunk or another tenant's chunk back into the fused list, and fusion will happily rank it. Apply the same filter to both retrievers before fusion, then test with a question whose exact keywords also appear in an out-of-scope version.
Agentic RAG moves the filter decision into a tool call. The agent calls a search tool whose arguments are the filter fields (product, version, date range), and the tool builds the store query from them in code. The argument schema doubles as your allowlist, and when an entity is ambiguous, the agent can ask a follow-up question instead of guessing. Keep tenant and permission fields out of the tool arguments, though, and have the tool read them from the session.
Common Pitfalls
Over-filtering that returns nothing. Stack enough extracted conditions on a small corpus and nothing matches, and post-filtering stores get there sooner. Decide the fallback per field ahead of time: relax date first, keep version if the user said it, never relax tenant.
Wrongly extracted filters. Models pick the wrong field, invent values, or turn "last year" into a hard-coded date, and a wrong filter looks just as valid as a right one. Some tools fail open by design: when generation is invalid, NVIDIA's RAG Blueprint "returns None, continues without filtering." That's fine for a relevance filter and a serious bug if access control ever rode on the same generated filter.
Inconsistent values. "2" vs. 2, upload-api vs. Upload API, dates as strings in one source and epochs in another. Filters are exact, so each mismatch silently hides chunks, which is why the validator above checks types.
Stale metadata. Permissions and versions change in the source system after you index. Keep the source document ID on every chunk and resync metadata whenever the source changes, including permission changes that never touch the text.
Access-control mistakes. A metadata filter only enforces what your code puts in it. Build tenant and permission filters from the authenticated session on every request and deny by default when they're missing, as OWASP's Authorization Cheat Sheet advises for any authorization check. For hard tenant isolation, Pinecone's guidance recommends a namespace per tenant and calls one shared namespace scoped by per-user metadata filters an anti-pattern. Part of the reason is that it reduces the risk of a bug that passes the wrong filter value.
How to Test Whether Filtering Helps
Every choice in this guide (fields, filter source, pre- or post-filtering, fallback strictness) changes what reaches the generator, and each one interacts with chunk size, k and reranking. A filter that looks right in a demo can still return empty context in production.
Test it like any other retrieval change:
Build a question set that depends on filters. Include version-specific, date-bounded, and tenant-scoped questions, with the expected chunk or document IDs for each. Add control questions that need no filter, and a few whose correct answer is "nothing in scope."
Score retrieval and answers separately. Precision@k and recall@k against the labeled IDs show whether the filter kept the right chunks (see our guide to RAG evaluation metrics). Add two counts of your own: queries that returned zero chunks, and chunks from outside the allowed tenant or version, which should be zero everywhere.
Compare configurations on the same questions. No filter vs. rule-based vs. LLM-generated, strict vs. relaxed fallback, and at least one k above your default.
This is the workflow we built RapidFire for. The open-source engine runs multiple RAG configurations side by side on the same eval set, and in its LangChain RAG spec, the filter lives in search_cfg next to k and the search type.
RFGridSearch accepts a list of config dictionaries, so each filter variant can be its own config, and List-valued knobs such as k inside search_cfg expand into more runs. metadata_func attaches your fields at load time, so your eval functions can read tenant and version off every retrieved chunk.
When your metrics declare their type, online aggregation shows running estimates with confidence intervals as shards complete. From the dashboard or notebook panel, you can then stop a weak configuration, or Clone-Modify a promising one with a different search_cfg. Converge automates those stop and clone decisions and keeps the experiment inside the time and spend limits you set.
Two details matter here. The default vector store is FAISS, so the 20-candidate behavior above applies: a selective tenant filter can exhaust the retrieved candidates, so log returned-chunk counts per question rather than assuming k matching chunks will arrive. And get_context takes only the batch of queries, so the filter in search_cfg is fixed per configuration. Group tenant-scoped questions into per-tenant configs, or, with the Pinecone store, point each config at that tenant's namespace through index_namespace.
To try it on real data, the RAG walkthrough runs a FiQA pipeline end to end (pip install rapidfireai, Python 3.12+). If you're planning metadata filtering for a production retrieval stack, talk to our team.
FAQ
What is metadata filtering in RAG?
It's a retrieval step that only lets chunks with matching stored attributes, such as a version, date range, or tenant ID, into the similarity ranking. It fixes problems vector similarity can't see in a RAG system, like a v1 answer to a v2 question. It doesn't make retrieval faster by default, and it's only as secure as the code that builds the filter.
What are the three main types of metadata?
Descriptive, structural, and administrative, following NISO's primer. In a RAG system, that's what a chunk is about (product, topic), where it sits in its source (section, page), and how it's managed (version, date, owner).
What are the four types of metadata?
NISO adds markup languages as a fourth category next to descriptive, structural, and administrative metadata. Markup mixes metadata into the content itself, and for RAG it's what a parser turns into section and heading fields at ingestion.
What are the five types of metadata?
No standard defines exactly five. Count NISO's three administrative subtypes (technical, preservation, and rights) separately alongside descriptive and structural, and you get five. For retrieval, the administrative side, where versions, dates, and owners live, supplies the most useful filters.