rapidfire.

RAG Metadata Filtering: How It Works and How to Set It Up

Written By:

Matt Tanner

Published on

Consider five short chunks of API documentation and the question, "How do I upload a file with the v2 API?" An unfiltered similarity search can rank v1 instructions or another customer's note above the matching v2 chunk.

That need not mean the embedding is broken: v1 and v2 upload docs can be semantically similar. Vector similarity cannot reliably enforce version, tenant, date, and permission constraints, so if your RAG pipeline needs to respect them, they have to travel with each chunk as metadata and be enforced at query time.

This guide covers the whole loop: what to store, how to attach it at ingestion, how to turn a question into a filter, how pre- and post-filtering change the results, and how to test whether any of it helped.

What Is Metadata Filtering in RAG?

Metadata filtering in RAG limits retrieval to chunks whose stored attributes match a condition, such as version == 2 or tenant == "acme", so vector similarity only ranks chunks that pass. Similarity decides which chunks are closest to the question, and metadata filters decide which ones are allowed in the results at all. Ideally, the search space shrinks to the matching subset, though, as the pre-filtering section shows, not every store applies the filter before the search.

In practice, a filter is a small structured object sent alongside the query vector. Pinecone, Chroma, and LangChain's FAISS wrapper all accept the MongoDB-style form {"$and": [{"tenant": {"$eq": "acme"}}, {"version": {"$eq": 2}}]}, while Milvus and pgvector express the same idea as a boolean expression or a SQL WHERE clause.

Why Vector Search Alone Returns the Wrong Documents

Embeddings compress meaning into a vector, and most of the facts that decide whether a chunk is usable (version, date, department, language, access rights) aren't meaning. "Upload a file with POST /v1/files" and "upload a file with POST /v2/uploads" land close together because they describe the same task. The model has no reason to push them apart.

That gap shows up in a handful of predictable ways in RAG applications:

  • Version: user queries about v2 retrieve v1 instructions, as in the test above.

  • Date: a question about the current refund policy pulls the 2023 policy, which is worded almost the same.

  • Tenant or department: a shared index returns another customer's notes, which is a correctness bug and a data exposure bug at once.

  • Access rights: a chunk from a document the user can't open in the source system still ranks, because the index never learned who could open it.

Reranking and better chunking can improve ordering inside the candidate set, but they can't know that a chunk belongs to the wrong customer.

Types of Metadata That Help RAG

When people ask about types of metadata, the standard answer comes from library science. NISO's Understanding Metadata primer splits metadata into descriptive, structural and administrative types, with administrative covering technical, preservation and rights information, and it adds markup languages as a final category. For retrieval, those categories map onto fields you can actually filter on:

Type

What it describes

RAG fields that use it

Typical filter

Descriptive

What the content is about

title, author, product, topic tags

product == "upload-api"
product == "upload-api"
product == "upload-api"

Structural

Where a chunk sits in its source

section, heading path, page number

section == "Authentication"
section == "Authentication"
section == "Authentication"

Administrative

How the content is managed

created/updated date, version, owner, permissions

version == 2, updated >= 2026-01-01

Source

Where the chunk came from

system, URL, file name, document ID

source == "confluence"
source == "confluence"
source == "confluence"

Source isn't a NISO category, but in RAG it earns its own row because it drives citations and lets you drop a whole system from retrieval when its content goes stale. Custom metadata, like customer tier or jurisdiction, usually fits under descriptive or administrative.

How to Design a Metadata Schema

Start from the questions people ask, not from the fields your documents happen to have. Pull 50 to 100 real user queries and mark every phrase that implies a constraint ("in v2," "for EU customers," "since the March release"). Each recurring constraint becomes a candidate metadata field, and fields nobody filters on are just one more thing that can be wrong at ingestion.

For each of the metadata fields that survive, decide four things:

  1. Type. Store versions as integers and dates as dates or epoch numbers, so range operators work. Pinecone, for example, only accepts numbers for $gt and $lt.

  2. Allowed values. Use a controlled vocabulary (upload-api, not "Upload API" in one source and "uploads" in another).

  3. Who sets it. Some fields come from the source system, some from parsing, some from an LLM. Write that down per field, because it decides how much you trust the value.

  4. Whether a model may filter on it. Relevance fields (product, version, date) can come from the question. Access fields (tenant, group, clearance) must come from the authenticated session.

NVIDIA's RAG Blueprint enforces this with a typed schema per collection: a file containing an undefined metadata field fails ingestion. Rejecting that file early is easier to debug than a filter that silently matches nothing.

Adding Metadata at Ingestion

Metadata has to be on the chunk before it's embedded and stored, and it usually comes from three places:

  • From the source system. Version, owner, permissions, and timestamps already exist in your CMS, ticketing tool, or file store. Copy them across with the document ID so you can resync later. Amazon Bedrock Knowledge Bases takes this route for S3 data sources, reading metadata from companion fileName.extension.metadata.json files.

  • From parsing. Section headings, page numbers, and table captions come out of the document structure. If you split with semantic chunking or any other splitter, check that the parent document's fields land on every child chunk.

  • From an LLM. When the source has no structure, LLM metadata extraction fills the gap: a model reads the text and returns field values. Haystack's LLMMetadataExtractor does this and parks documents it fails on in a failed_documents list, and LlamaIndex ships extractors such as TitleExtractor and EntityExtractor.

Treat extracted metadata as a guess until it passes the same checks you'd run on a filter: field allowlist, type, and allowed values. A wrong version tag is worse than a missing one, because the filter will confidently hide the right chunk.

Filtering at Query Time

Filters reach a query in three ways, often combined:

  • Explicit filters from the UI. The user picks a product and version from dropdowns, and the app passes them straight through. This is the most reliable option.

  • Rule-based filters. The app adds filters it already knows, like the tenant from the login session or status == "published", plus simple patterns such as a regex for "v2" in the question.

  • LLM-generated filters from natural language. A model reads the natural language question plus a description of the metadata fields and proposes a filter. LangChain's self-querying retriever (SelfQueryRetriever) and LlamaIndex's VectorIndexAutoRetriever work this way, and Amazon Bedrock Knowledge Bases offers it as implicit filtering, which "generates and applies a retrieval filter based on the user query and a metadata schema."

NVIDIA's RAG docs file this third option under advanced metadata filtering, and it's where most of the risk sits. Treat the model's output as a proposal. Parse it, check every field and value against the schema, and add the access fields yourself from the session. Here's a short, store-agnostic version:

import json

# Fields the model may propose, with the type and values each one accepts.
SCHEMA = {
    "product": (str, {"upload-api", "billing-api"}),
    "version": (int, {1, 2}),
}

def build_filter(tenant_id: str, model_output: str) -> dict:
    """Turn the model's JSON proposal into a store filter.
    The tenant always comes from the session, never from the model."""
    if not isinstance(tenant_id, str) or not tenant_id.strip():
        raise ValueError("authenticated tenant is required")
    proposal = json.loads(model_output)
    if not isinstance(proposal, dict):
        raise ValueError("filter proposal must be a JSON object")
    clauses = [{"tenant": {"$eq": tenant_id}}]
    for field, value in proposal.items():
        if field not in SCHEMA:
            raise ValueError(f"field not allowed: {field}")
        expected_type, allowed = SCHEMA[field]
        if type(value) is not expected_type or value not in allowed:
            raise ValueError(f"bad value for {field}: {value!r}")
        clauses.append({field: {"$eq": value}})
    return {"$and": clauses} if len(clauses) > 1 else clauses[0]
import json

# Fields the model may propose, with the type and values each one accepts.
SCHEMA = {
    "product": (str, {"upload-api", "billing-api"}),
    "version": (int, {1, 2}),
}

def build_filter(tenant_id: str, model_output: str) -> dict:
    """Turn the model's JSON proposal into a store filter.
    The tenant always comes from the session, never from the model."""
    if not isinstance(tenant_id, str) or not tenant_id.strip():
        raise ValueError("authenticated tenant is required")
    proposal = json.loads(model_output)
    if not isinstance(proposal, dict):
        raise ValueError("filter proposal must be a JSON object")
    clauses = [{"tenant": {"$eq": tenant_id}}]
    for field, value in proposal.items():
        if field not in SCHEMA:
            raise ValueError(f"field not allowed: {field}")
        expected_type, allowed = SCHEMA[field]
        if type(value) is not expected_type or value not in allowed:
            raise ValueError(f"bad value for {field}: {value!r}")
        clauses.append({field: {"$eq": value}})
    return {"$and": clauses} if len(clauses) > 1 else clauses[0]
import json

# Fields the model may propose, with the type and values each one accepts.
SCHEMA = {
    "product": (str, {"upload-api", "billing-api"}),
    "version": (int, {1, 2}),
}

def build_filter(tenant_id: str, model_output: str) -> dict:
    """Turn the model's JSON proposal into a store filter.
    The tenant always comes from the session, never from the model."""
    if not isinstance(tenant_id, str) or not tenant_id.strip():
        raise ValueError("authenticated tenant is required")
    proposal = json.loads(model_output)
    if not isinstance(proposal, dict):
        raise ValueError("filter proposal must be a JSON object")
    clauses = [{"tenant": {"$eq": tenant_id}}]
    for field, value in proposal.items():
        if field not in SCHEMA:
            raise ValueError(f"field not allowed: {field}")
        expected_type, allowed = SCHEMA[field]
        if type(value) is not expected_type or value not in allowed:
            raise ValueError(f"bad value for {field}: {value!r}")
        clauses.append({field: {"$eq": value}})
    return {"$and": clauses} if len(clauses) > 1 else clauses[0]

A proposal of {"tenant": "globex"} raises field not allowed: tenant, and {"version": "2"} fails because the string "2" isn't the integer 2. When validation fails, send the error back to the model once or ask the user a clarifying question. Don't quietly drop the bad field and search anyway.

Save the first snippet as filters.py beside the script below. This example uses chromadb 1.5.9 and fastembed 0.9.1; install them with pip install chromadb==1.5.9 fastembed==0.9.1. The BAAI/bge-small-en-v1.5 embedding model downloads on first use. The model's filter proposal is a fixed string standing in for an LLM call:

import chromadb
from fastembed import TextEmbedding
from filters import build_filter

chunks = [
    ("acme-v1-upload", "To upload a file, POST it to /v1/files as multipart form data.",
     {"tenant": "acme", "product": "upload-api", "version": 1}),
    ("acme-v2-upload", "To upload a file, create an upload session with POST /v2/uploads, then PUT the bytes.",
     {"tenant": "acme", "product": "upload-api", "version": 2}),
    ("acme-v2-limits", "Uploads in v2 accept files up to 5 GB per session.",
     {"tenant": "acme", "product": "upload-api", "version": 2}),
    ("globex-v2-upload", "Globex note: upload files through the v2 sessions endpoint, then PUT the file bytes.",
     {"tenant": "globex", "product": "upload-api", "version": 2}),
    ("acme-v2-invoices", "Invoices are created with POST /v2/invoices.",
     {"tenant": "acme", "product": "billing-api", "version": 2}),
]

embedder = TextEmbedding("BAAI/bge-small-en-v1.5")
embed = lambda texts: [v.tolist() for v in embedder.embed(texts)]

col = chromadb.EphemeralClient().create_collection("docs", metadata={"hnsw:space": "cosine"})
col.add(ids=[c[0] for c in chunks], documents=[c[1] for c in chunks],
        metadatas=[c[2] for c in chunks], embeddings=embed([c[1] for c in chunks]))

question = "How do I upload a file with the v2 API?"
model_output = '{"product": "upload-api", "version": 2}'  # stand-in for the LLM's reply
where = build_filter("acme", model_output)

q = embed([question])
print("before:", col.query(query_embeddings=q, n_results=3)["ids"][0])
print("after: ", col.query(query_embeddings=q, n_results=3, where=where)["ids"][0])

before: ['acme-v1-upload', 'globex-v2-upload', 'acme-v2-upload']
after:  ['acme-v2-upload', 'acme-v2-limits']
import chromadb
from fastembed import TextEmbedding
from filters import build_filter

chunks = [
    ("acme-v1-upload", "To upload a file, POST it to /v1/files as multipart form data.",
     {"tenant": "acme", "product": "upload-api", "version": 1}),
    ("acme-v2-upload", "To upload a file, create an upload session with POST /v2/uploads, then PUT the bytes.",
     {"tenant": "acme", "product": "upload-api", "version": 2}),
    ("acme-v2-limits", "Uploads in v2 accept files up to 5 GB per session.",
     {"tenant": "acme", "product": "upload-api", "version": 2}),
    ("globex-v2-upload", "Globex note: upload files through the v2 sessions endpoint, then PUT the file bytes.",
     {"tenant": "globex", "product": "upload-api", "version": 2}),
    ("acme-v2-invoices", "Invoices are created with POST /v2/invoices.",
     {"tenant": "acme", "product": "billing-api", "version": 2}),
]

embedder = TextEmbedding("BAAI/bge-small-en-v1.5")
embed = lambda texts: [v.tolist() for v in embedder.embed(texts)]

col = chromadb.EphemeralClient().create_collection("docs", metadata={"hnsw:space": "cosine"})
col.add(ids=[c[0] for c in chunks], documents=[c[1] for c in chunks],
        metadatas=[c[2] for c in chunks], embeddings=embed([c[1] for c in chunks]))

question = "How do I upload a file with the v2 API?"
model_output = '{"product": "upload-api", "version": 2}'  # stand-in for the LLM's reply
where = build_filter("acme", model_output)

q = embed([question])
print("before:", col.query(query_embeddings=q, n_results=3)["ids"][0])
print("after: ", col.query(query_embeddings=q, n_results=3, where=where)["ids"][0])

before: ['acme-v1-upload', 'globex-v2-upload', 'acme-v2-upload']
after:  ['acme-v2-upload', 'acme-v2-limits']
import chromadb
from fastembed import TextEmbedding
from filters import build_filter

chunks = [
    ("acme-v1-upload", "To upload a file, POST it to /v1/files as multipart form data.",
     {"tenant": "acme", "product": "upload-api", "version": 1}),
    ("acme-v2-upload", "To upload a file, create an upload session with POST /v2/uploads, then PUT the bytes.",
     {"tenant": "acme", "product": "upload-api", "version": 2}),
    ("acme-v2-limits", "Uploads in v2 accept files up to 5 GB per session.",
     {"tenant": "acme", "product": "upload-api", "version": 2}),
    ("globex-v2-upload", "Globex note: upload files through the v2 sessions endpoint, then PUT the file bytes.",
     {"tenant": "globex", "product": "upload-api", "version": 2}),
    ("acme-v2-invoices", "Invoices are created with POST /v2/invoices.",
     {"tenant": "acme", "product": "billing-api", "version": 2}),
]

embedder = TextEmbedding("BAAI/bge-small-en-v1.5")
embed = lambda texts: [v.tolist() for v in embedder.embed(texts)]

col = chromadb.EphemeralClient().create_collection("docs", metadata={"hnsw:space": "cosine"})
col.add(ids=[c[0] for c in chunks], documents=[c[1] for c in chunks],
        metadatas=[c[2] for c in chunks], embeddings=embed([c[1] for c in chunks]))

question = "How do I upload a file with the v2 API?"
model_output = '{"product": "upload-api", "version": 2}'  # stand-in for the LLM's reply
where = build_filter("acme", model_output)

q = embed([question])
print("before:", col.query(query_embeddings=q, n_results=3)["ids"][0])
print("after: ", col.query(query_embeddings=q, n_results=3, where=where)["ids"][0])

before: ['acme-v1-upload', 'globex-v2-upload', 'acme-v2-upload']
after:  ['acme-v2-upload', 'acme-v2-limits']

The example output above illustrates the filter: although the query asks for three results, only two chunks match. Exact similarity ordering can vary. A filter changes how many results you get, not just which ones.

[Image placeholder: Left-to-right flow diagram. A separate left column labeled "Ingestion" shows: Source documents, then Chunk + attach metadata (tenant, product, version, date), then Vector store. Main flow on the right: User question, then Filter extractor (LLM) producing a proposed filter, then Validate against schema (with a side arrow labeled "reject or clarify" pointing back to the user), then two parallel boxes entering the vector store together: "Metadata filter (tenant from session + validated fields)" and "Vector query", then Vector store (the same box fed by the ingestion column), then Filtered chunks, then Generator, then Answer. Alt text: RAG metadata filtering flow from question through LLM filter extraction and validation to a filtered vector search, with metadata tagged at ingestion.]

Pre-Filtering vs. Post-Filtering

Where the filter runs relative to the nearest-neighbor search decides whether you get k results, fewer, or none. Pre-filtering restricts candidates before or during the vector search. Post-filtering runs the vector search first and then drops chunks that fail the filter, so a selective filter can leave you with almost nothing.

Pre-filtering

Post-filtering

When the filter runs

Before or during the ANN search

After the ANN search returns candidates

Result count

Can return k matching results; approximate-search limits still apply

Can return fewer than k, or zero

Recall on selective filters

Holds up

Drops as the filter gets narrower

Latency

On HNSW, a narrow filter can make the search slower

Skips the cost of filtered graph traversal

Best fit

Narrow filters such as tenant, version, permissions

Broad filters that match most chunks

How common stores handle it:

  • Elasticsearch: a filter inside the knn clause "is applied during approximate kNN search to ensure that k matching documents are returned," while a filter placed outside the kNN clause post-filters. The filtered kNN docs also note that on HNSW, "filters can make a search slower rather than faster," so don't promise faster responses as a side effect.

  • Milvus: standard filtering applies the filter before the ANN search, with an iterative filtering mode for very complex expressions.

  • pgvector: with approximate indexes, the WHERE clause runs after the index scan. The pgvector README gives the numbers: a condition matching 10% of rows at the default hnsw.ef_search of 40 returns about 4 rows. Version 0.8.0 added iterative index scans (SET hnsw.iterative_scan = strict_order or relaxed_order), which keep scanning up to hnsw.max_scan_tuples, 20,000 by default.

  • LangChain's FAISS wrapper: similarity search fetches fetch_k candidates, 20 by default, and filters those.

That last one is easy to hit in a prototype. For a synthetic example, index 1,000 chunks with deterministic two-dimensional vectors and place one tenant's 20 chunks beyond the first 20 candidates. A k=5 query with that tenant filter has no eligible results among the default 20 candidates; raising fetch_k to 1,000 makes enough matching candidates available.

This demonstrates candidate starvation, not its frequency on real embeddings. There's no error or warning, just an empty context, so log the returned-chunk count for every filtered query.

Metadata Filtering With Hybrid Search and Agents

Hybrid search runs a keyword retriever such as BM25 next to vector search and fuses the two ranked lists. If only the vector side is filtered, the keyword side can pull a v1 chunk or another tenant's chunk back into the fused list, and fusion will happily rank it. Apply the same filter to both retrievers before fusion, then test with a question whose exact keywords also appear in an out-of-scope version.

Agentic RAG moves the filter decision into a tool call. The agent calls a search tool whose arguments are the filter fields (product, version, date range), and the tool builds the store query from them in code. The argument schema doubles as your allowlist, and when an entity is ambiguous, the agent can ask a follow-up question instead of guessing. Keep tenant and permission fields out of the tool arguments, though, and have the tool read them from the session.

Common Pitfalls

Over-filtering that returns nothing. Stack enough extracted conditions on a small corpus and nothing matches, and post-filtering stores get there sooner. Decide the fallback per field ahead of time: relax date first, keep version if the user said it, never relax tenant.

Wrongly extracted filters. Models pick the wrong field, invent values, or turn "last year" into a hard-coded date, and a wrong filter looks just as valid as a right one. Some tools fail open by design: when generation is invalid, NVIDIA's RAG Blueprint "returns None, continues without filtering." That's fine for a relevance filter and a serious bug if access control ever rode on the same generated filter.

Inconsistent values. "2" vs. 2, upload-api vs. Upload API, dates as strings in one source and epochs in another. Filters are exact, so each mismatch silently hides chunks, which is why the validator above checks types.

Stale metadata. Permissions and versions change in the source system after you index. Keep the source document ID on every chunk and resync metadata whenever the source changes, including permission changes that never touch the text.

Access-control mistakes. A metadata filter only enforces what your code puts in it. Build tenant and permission filters from the authenticated session on every request and deny by default when they're missing, as OWASP's Authorization Cheat Sheet advises for any authorization check. For hard tenant isolation, Pinecone's guidance recommends a namespace per tenant and calls one shared namespace scoped by per-user metadata filters an anti-pattern. Part of the reason is that it reduces the risk of a bug that passes the wrong filter value.

How to Test Whether Filtering Helps

Every choice in this guide (fields, filter source, pre- or post-filtering, fallback strictness) changes what reaches the generator, and each one interacts with chunk size, k and reranking. A filter that looks right in a demo can still return empty context in production.

Test it like any other retrieval change:

  1. Build a question set that depends on filters. Include version-specific, date-bounded, and tenant-scoped questions, with the expected chunk or document IDs for each. Add control questions that need no filter, and a few whose correct answer is "nothing in scope."

  2. Score retrieval and answers separately. Precision@k and recall@k against the labeled IDs show whether the filter kept the right chunks (see our guide to RAG evaluation metrics). Add two counts of your own: queries that returned zero chunks, and chunks from outside the allowed tenant or version, which should be zero everywhere.

  3. Compare configurations on the same questions. No filter vs. rule-based vs. LLM-generated, strict vs. relaxed fallback, and at least one k above your default.

This is the workflow we built RapidFire for. The open-source engine runs multiple RAG configurations side by side on the same eval set, and in its LangChain RAG spec, the filter lives in search_cfg next to k and the search type.

RFGridSearch accepts a list of config dictionaries, so each filter variant can be its own config, and List-valued knobs such as k inside search_cfg expand into more runs. metadata_func attaches your fields at load time, so your eval functions can read tenant and version off every retrieved chunk.

When your metrics declare their type, online aggregation shows running estimates with confidence intervals as shards complete. From the dashboard or notebook panel, you can then stop a weak configuration, or Clone-Modify a promising one with a different search_cfg. Converge automates those stop and clone decisions and keeps the experiment inside the time and spend limits you set.

Two details matter here. The default vector store is FAISS, so the 20-candidate behavior above applies: a selective tenant filter can exhaust the retrieved candidates, so log returned-chunk counts per question rather than assuming k matching chunks will arrive. And get_context takes only the batch of queries, so the filter in search_cfg is fixed per configuration. Group tenant-scoped questions into per-tenant configs, or, with the Pinecone store, point each config at that tenant's namespace through index_namespace.

To try it on real data, the RAG walkthrough runs a FiQA pipeline end to end (pip install rapidfireai, Python 3.12+). If you're planning metadata filtering for a production retrieval stack, talk to our team.

FAQ

What is metadata filtering in RAG?

It's a retrieval step that only lets chunks with matching stored attributes, such as a version, date range, or tenant ID, into the similarity ranking. It fixes problems vector similarity can't see in a RAG system, like a v1 answer to a v2 question. It doesn't make retrieval faster by default, and it's only as secure as the code that builds the filter.

What are the three main types of metadata?

Descriptive, structural, and administrative, following NISO's primer. In a RAG system, that's what a chunk is about (product, topic), where it sits in its source (section, page), and how it's managed (version, date, owner).

What are the four types of metadata?

NISO adds markup languages as a fourth category next to descriptive, structural, and administrative metadata. Markup mixes metadata into the content itself, and for RAG it's what a parser turns into section and heading fields at ingestion.

What are the five types of metadata?

No standard defines exactly five. Count NISO's three administrative subtypes (technical, preservation, and rights) separately alongside descriptive and structural, and you get five. For retrieval, the administrative side, where versions, dates, and owners live, supplies the most useful filters.