Semantic Chunking for RAG: Methods, Trade-Offs, and Testing
Written By:
Matt Tanner
Published on
A chunk boundary can separate a refund rule from an exception mentioned later on. Your retriever may then find only half of the answer. Semantic chunking tries to avoid this by looking for changes in meaning rather than cutting after a fixed number of tokens.
That is a useful idea, but it needs testing. Qu, Tu, and Bao's NAACL 2025 comparison found that two semantic chunkers did not deliver consistent gains that justified their extra computation over fixed-size chunking. In a small test for this guide, LangChain's default settings also missed both topic shifts in a short three-topic document.
This guide covers how semantic chunking works and how it stacks up against other RAG chunking strategies. Then it walks through a LangChain implementation and shows how to test whether it beats a simpler splitter on your own data.
What Is Semantic Chunking?
Semantic chunking splits text by comparing the meaning of neighboring sentences. It turns sentences into embeddings, numerical representations used to measure similarity, and starts a new chunk where that similarity drops enough. The aim is to keep related information together, rather than cut at a fixed token or character count.
Picture an employee handbook where one page covers the on-call rotation, then travel expenses, then laptop security, with no headings in between. A 200-token splitter cuts wherever the count lands. A semantic chunker tries to cut at the two places where the subject actually changes.
Greg Kamradt's 5 Levels of Text Splitting notebook groups similar neighboring sentences using embeddings, then starts a chunk when the sentence-to-sentence distance exceeds the 95th percentile. That historical implementation uses the percentile as its only parameter. LangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser credit the notebook for the approach. Amazon Bedrock knowledge bases also offer managed semantic chunking.
Why Chunking Matters in RAG
In a chunk-based RAG pipeline, the retriever searches the chunks you indexed. Their boundaries determine which facts stay together and which get separated. If a rule and its exception land in different chunks, retrieving one may not bring back the other.
Changing those boundaries means rebuilding the chunk embeddings and index. That makes chunking a decision to test during ingestion, before you spend time tuning prompts around poorly packaged context.
Chunk size cuts both ways. A tight chunk or no overlap can separate a detail from the passage needed to interpret it. Larger chunks and heavier overlap keep more text together, but also send more tokens—and potentially irrelevant material—to the generator.
Here's how that plays out in practice:
Smaller chunks lose context. A sentence like "This limit doesn't apply to contractors" is useless once it's separated from the limit it refers to.
Larger chunks blur their embeddings. A chunk covering three topics sits somewhere between all three in vector space, so it tends to rank below a focused chunk for each one, and when it does get retrieved, it spends context window tokens on text the model doesn't need.
Boundaries in the wrong place do both at once. The answer gets split across two chunks, and each half is padded with unrelated text.
Chroma's technical report on chunking strategies found that the choice of chunking strategy moved retrieval recall by up to 9% across the strategies they tested. Chunking also sets what the language model sees, which is why it sits alongside prompts and examples as part of context engineering.
How Semantic Chunking Works
LangChain's SemanticChunker, LlamaIndex's SemanticSplitterNodeParser, and Bedrock's managed version all follow the same basic semantic chunking algorithm:
Split the text into sentences. LangChain uses a regex that breaks after a period, question mark, or exclamation mark followed by whitespace.
Add neighboring sentences. With
buffer_size=1, each sentence is joined with the one before and the one after, which Kamradt added to reduce noise.Embed each sentence window with an embedding model.
Measure the distance between consecutive windows. LangChain uses cosine distance, which is 1 minus cosine similarity.
Mark split points wherever the distance exceeds a threshold.
Merge the sentences between breakpoints into chunks, then apply any size limits (a minimum in LangChain, a maximum token count in Bedrock).
The threshold controls which gaps become boundaries. LangChain's default uses the 95th percentile: it looks for unusually large distances relative to the other gaps in that document. Lowering the percentile generally creates more opportunities to split.
Other options use the spread or change in those distances: the mean plus a multiple of the standard deviation (default 3), the mean plus a multiple of the interquartile range (default 1.5), or a percentile of the distances' gradient (default 95). You can also request a target chunk count and let the implementation derive a threshold. These are different rules for selecting boundaries, so compare their output rather than assuming their defaults identify topic changes.
A percentile is relative to the document. It identifies the largest gaps even when none mark a meaningful topic change. In Amazon Bedrock's example, 101 sentences create 100 adjacent pairs. A 90th-percentile threshold splits the 10 least similar pairs, producing 11 chunks before the maximum-size check.
That example explains the limitation: the cutoff follows the distribution of similarity scores, not a count of actual topics. It is not a universal chunk-count rule for every implementation; ties and minimum-size merging can change the result. Chroma's evaluation also notes that a relative threshold can produce larger chunks in bigger corpora.
The test used LangChain's chunker on an 18-sentence handbook excerpt covering three topics (on-call rotation, travel expenses, and laptop security) with the all-MiniLM-L6-v2 embedding model. With buffer_size=0 and the 80th percentile, four gaps cleared the threshold. Two were the real topic boundaries and two fell inside the on-call section.
Schematic illustration: sentence positions, distances, and chunk boundaries are illustrative, not measured experiment output.
Semantic Chunking vs. Other Chunking Strategies
Compare the boundary rule and ingestion work for each chunking strategy:
Strategy | How it splits | Cost to run | Best for | Watch out for |
|---|---|---|---|---|
Fixed-size | Every N tokens or characters, optionally with overlap | Lowest | Baselines, uniform text, fast iteration | Cuts mid-sentence and mid-topic |
Recursive character | Tries paragraph breaks, then line breaks, then spaces, until pieces fit the size limit | Low | General prose; LangChain's recommended starting point | Defaults count characters, not tokens |
Sentence-based | Packs whole sentences up to a size limit | Low | Q&A over short factual text | Sentence detection breaks on abbreviations and code |
Document structure-based (structure-aware) | Markdown headers, HTML tags, JSON objects, code functions | Low | Docs, wikis, code, anything with real headings | Inconsistent formatting gives uneven chunks |
Semantic | Breaks where embedding similarity between neighboring sentences drops | Medium: embeds every sentence window at ingestion | Long, mixed-topic text without headings | No maximum size in LangChain; threshold is relative |
Hierarchical | Several chunk sizes with parent links; retrieve small, return the parent | Medium: more chunks to embed and store | Questions that need both a precise hit and surrounding context | More index plumbing and retriever logic |
LLM-based (agentic) | Prompts a model to choose split points | Highest: tens of minutes in Chroma's test | Small, high-value corpora | Cost, latency, and hallucinated text if the model rewrites the input |
Late chunking | Embeds the whole document with a long-context model, then pools token vectors per chunk | Medium: one long-context embedding pass per document | Chunks that depend on earlier context ("it," "the policy") | Needs token-level outputs from a long-context embedding model |
Start with the structure the document already has. A recursive splitter uses paragraph breaks without embedding every sentence to choose a boundary. Semantic chunking is worth testing when those cues are missing, as in transcripts, scraped pages, or long emails that drift between subjects.
Chroma's chunking report, linked under "Why Chunking Matters in RAG," shows why the simpler baseline matters. In its test, a recursive splitter using 200-token chunks, no overlap, and sentence punctuation among its separators reached 88.1% recall. The default Kamradt-style semantic chunker averaged about 660 tokens per chunk and scored 83.6%, the lowest recall in that table. Those are results for the tested configurations, not a guarantee that one strategy will always win.
Hierarchical chunking vs. semantic chunking answers a different question. Hierarchical chunking doesn't try to find better boundaries. It indexes small chunks for precise matching and swaps in the larger parent when enough of its children are retrieved. You can combine the two, but they fix different failures.
Contextual chunking isn't really a splitter at all. Anthropic's contextual retrieval approach has an LLM write a short, chunk-specific description and prepends it to each chunk before embedding and BM25 indexing. Late chunking goes after the same lost-context problem from the embedding side. Contextual chunking and late chunking both work on top of whatever boundaries you already have.
Running both splitters on the same handbook excerpt shows the difference. Fixed 60-token windows cut through sentences ("Hotel stays are" ends one chunk, "capped at the city rate" starts the next). The tuned semantic chunker split at both topic shifts and added one extra split inside the on-call section, after min_chunk_size merged a one-sentence fragment into its neighbor.
Schematic illustration: sentence positions, distances, and chunk boundaries are illustrative, not measured experiment output.
How to Implement Semantic Chunking
LangChain sunset the langchain-experimental package, home of SemanticChunker, on May 22, 2026, effective immediately. The repository was archived a few days later. Version 0.4.2 still installs and works, but importing it now raises a DeprecationWarning, and LangChain recommends focused packages or example implementations for new experimental work. If you build on LangChain semantic chunking today, pin the version or copy the roughly 300-line module into your own code.
The companion example repository includes the exact handbook.txt, runnable scripts, tested dependencies, and run instructions. This example reads the file, splits it at semantic boundaries, then caps the resulting chunks to fit the embedding model. Before running it:
Use Python 3.12+ in a virtual environment.
Run the
pipcommand below.Use the repository's bundled UTF-8
handbook.txtbeside the script, or replace it with your own non-empty text. Start with several complete sentences covering at least two topics.
The model and tokenizer download on first use, but inference runs locally without an API key.
The script prints each chunk's index, character count, and first 50 characters; it does not build a retrieval index or evaluate answer quality.
The exact eighteen-sentence, three-topic handbook.txt in the repository reproduces the four chunks (index, character count, and opening text). Your own input may produce different results:
Both topic shifts land on chunk boundaries, with one extra split inside the on-call section. The settings that got there are worth knowing:
Parameter | What it controls | Starting value |
|---|---|---|
How the cutoff is computed from the distance list |
| |
How strict the cutoff is; a lower percentile means more chunks | 95 by default; try 80 to 95 | |
Neighboring sentences joined to each sentence before embedding | 1 by default; try 0 for short, distinct sentences | |
Groups shorter than this many characters merge into the next chunk |
| |
Targets a chunk count instead of a threshold | ||
How sentences are detected | Default regex; replace it for abbreviation-heavy or non-English text | |
The model that scores similarity | The model you retrieve with |
Two gotchas showed up even in this tiny test. The default regex split "9 a.m. local time" into two sentences, and the default settings (buffer_size=1, 95th percentile) produced one break, in the middle of the expenses topic.
For LlamaIndex semantic chunking, use the semantic splitter SemanticSplitterNodeParser. It takes buffer_size (default 1), breakpoint_percentile_threshold (default 95), and an embed_model. Its sentence regex mainly targets English; for your own text, check sentence boundaries and tune the percentile threshold rather than relying on the defaults.
Check the embedding model's input limit at both stages: when finding sentence boundaries and when embedding the finished chunks for retrieval. By default, all-MiniLM-L6-v2 truncates anything longer than 256 word pieces, so text beyond that limit does not contribute to the embedding.
The second-pass cap in this example limits the finished chunks using that model's tokenizer. It cannot protect an oversized sentence window that was embedded earlier. Use short sentences for this demonstration, and check window lengths when adapting it to your own documents. Our guide to the best embedding models for RAG covers context length alongside quality and cost.
Budget for the extra embedding work, too. SemanticChunker embeds every sentence window while it splits, and with buffer_size=1 most sentences appear in three windows. Then your vector store embeds the finished chunks again.
When Semantic Chunking Helps, and When It Doesn't
In Qu, Tu, and Bao's NAACL 2025 comparison cited in the introduction, which covered document retrieval, evidence retrieval, and answer generation, two semantic chunkers failed to deliver consistent gains that justified their added computation over fixed-size chunking. That makes the study a reason to test semantic chunking rather than adopt it as the default.
In Qu, Tu, and Bao's experiments, semantic chunkers improved results on long documents made by stitching randomly chosen short documents together, especially where those documents had sharp topic changes. Fixed-size chunking often did better on non-synthetic datasets, and embedding quality could have more influence than the chunking strategy. In Chroma's evaluation, tuned recursive splitting also performed well, while some chunkers' default settings produced relatively poor retrieval results.
The handbook test measures boundary placement, not retrieval quality. Here's what LangChain's chunker produced across settings:
Settings | Chunks | Real topic shifts found |
|---|---|---|
Defaults: | 2 | 0 of 2 |
| 1 | 0 of 2 |
| 3 | 1 of 2 |
| 5 | 2 of 2, plus 2 extra splits |
Single-topic version of the text, defaults | 2 | No real shift to find, but it split anyway |
An eighteen-sentence test isn't a benchmark, but the pattern reflects the mechanism described earlier: at percentiles 80, 90, and 95, both the three-topic and single-topic documents always produced 5, 3, and 2 chunks, for every buffer size tested.
Semantic chunking is worth testing when:
Documents are long and change topics without headings, such as meeting transcripts, support threads, or scraped pages.
One file mixes unrelated content, the closest real-world match to the stitched documents where it won.
You can afford the extra embedding calls at ingestion and will re-run them whenever the corpus changes.
Skip it, or put something else first, when:
Structured documents already supply candidate boundaries through Markdown or HTML headers.
The document types are mostly tables, code, or lists. A sentence regex has nothing sensible to split on.
Documents are short. A two-paragraph FAQ entry is already one topic.
Text isn't English, unless you replace the sentence splitter.
Treat semantic chunking as a candidate to test against simpler splitters, not as a default.
How to Choose a Chunking Strategy: Test It on Your Own Data
Test chunking strategies and chunk sizes with your corpus, embedding model, and retrieval setup. Keep the documents and questions fixed so the results compare the configurations.
Build a Small Eval Set With Known Source Passages
Collect a few dozen to a hundred real questions, or generate candidate questions from your documents and review them. For each one, record the passage or page that supplies the answer and write a reference answer. Include questions whose answers cross a topic boundary, because those expose the differences between chunkers.
The source labels support retrieval checks such as recall, precision, NDCG, and MRR. Reference answers support correctness checks; faithfulness instead checks whether the response is supported by the context actually retrieved. Our guide to RAG evaluation metrics explains these distinctions.
Sweep Chunking Together With Retrieval and Reranking
Chunking doesn't act alone. Chunk size changes what the embedder sees, which changes what the top-k search returns, which changes what the reranker can work with. The chunk size that wins with k=5 isn't necessarily the one that wins with k=15 and a cross-encoder reranker on top, so test combinations rather than chunkers in isolation.
With the generator and embedder pinned, we swept 40 retrieval configurations for a travel-insurance chatbot. The winner split the policy into 512-token chunks, retrieved the top 10, and let a cross-encoder rerank them down to 2, lifting the retrieval composite score from 0.439 to 0.627, about 43 percent on that benchmark.
The 76 question-and-answer pairs were synthesized from the policy, so this is a controlled demonstration. The winning chunk size and k apply to that policy; use a fresh sweep to choose values for another corpus.
RapidFire, our open-source experimentation framework for retrieval and context engineering, runs these comparisons on a shared eval set. In our LangChain RAG spec, you configure chunk size, overlap, and splitting strategy through compatible text splitters. A List lets you supply several options for a splitter, embedder, search setting, or reranker. Grid, random, or Optuna search produces the configurations to compare.
You write the metric functions, then review the results side by side. During a run, you can stop weak configurations and clone promising ones to explore further. The framework also filters specific invalid combinations, such as retrieving fewer chunks than the reranker is configured to keep. SemanticChunker compatibility is not established here; use the documented splitter types for this workflow.
One constraint to plan around: chunking is a document-preprocessing knob, so each chunking variant builds its own index and can't be changed while a run is in flight. Search settings, rerankers, generators, and prompts can be changed with clone-modify. Plan your chunk sizes and splitter types up front.
Compare Retrieval and Answer Metrics, Then Pick
Look at retrieval quality first, because recall and precision tell you whether the right text came back at all. Then check answer quality, since a chunker that returns longer chunks can raise recall while diluting what the generator reads. Compare against your current setup as the baseline, and keep latency and token spend in the same table.
If semantic chunking wins on your data by a margin bigger than run-to-run noise, keep it. If a tuned recursive splitter ties it, take the cheaper one.
To see this end to end, our FiQA tutorial compares 256-token and 128-token chunks (both recursive, with 32-token overlap) across two generators and two reranker depths: 8 combinations from one grid. Swap in compatible splitters and chunk sizes, and you have a chunking experiment.
Chunking Best Practices
Start with a baseline you understand, usually a recursive splitter sized in tokens, and make every other strategy beat it on the same eval set.
Always state your units. LangChain's
chunk_sizecounts characters unless you build the splitter with a token encoder, andSemanticChunker'smin_chunk_sizeis in characters.Match the chunking approach to the document type. Test whether existing headers and sections provide adequate boundaries before adding sentence-embedding work.
Keep tables and code blocks intact. Split around them, not through them, and store them as their own chunks with a caption or heading attached.
Put a hard cap on chunk size so chunks fit your embedding model's input limit, especially with semantic chunking.
Attach metadata to every chunk (source, section, page, and timestamp) so you can filter, cite, and debug retrieval misses.
Use overlap deliberately. A small overlap protects against bad cuts in fixed-size chunking, while
SemanticChunkerhas no overlap setting at all, so test whether its boundaries make overlap unnecessary on your data.Pair chunking with retrieval that suits your queries. Exact IDs and error strings often need keyword search alongside vectors; see our guide to hybrid search for RAG.
Re-test when the corpus or the embedding model changes, since the winning chunk size is specific to the data. If the sweep lives in a RapidFire config, re-testing means re-running it instead of rebuilding the comparison.
Conclusion
Test semantic chunking on long, unstructured text with topic changes. Its default percentile threshold can split a document even when the topic does not change. Before you adopt it, give it a hard size cap, tune the threshold, and make it beat a recursive baseline on questions from your own corpus.
Our RAG quickstart covers both routes: a Colab notebook with zero local setup, or pip install rapidfireai on your own machine. Then open the FiQA tutorial, which already compares two chunk sizes, and add compatible splitters, chunk sizes, and overlaps to its grid alongside the retrieval and reranking settings.
If you're planning chunking and retrieval tests for a production RAG system, you can also talk to our team.
FAQ
What Are the Different Types of Chunking?
The main types of chunking for RAG are fixed-size (every N tokens), recursive character splitting (paragraphs, then lines, and then words), sentence-based, document structure-based (headers, HTML tags, and code functions), semantic (breaks at embedding-similarity drops), hierarchical (small chunks linked to larger parents), LLM-based or agentic, and late chunking. Contextual chunking adds generated context to chunks from any of these.
What Is Semantic Chunking in LangChain?
LangChain's SemanticChunker splits text into sentences, embeds each one with its neighbors, and breaks where cosine distance between consecutive windows passes a threshold (percentile, standard deviation, interquartile, or gradient). It lives in langchain-experimental, which LangChain sunset in May 2026, so pin version 0.4.2 or copy the class into your code.
What Chunk Size Should I Use for RAG?
There's no universal answer. The examples in this guide span 128 to 512 tokens: the FiQA tutorial above compares 128 and 256 with 32-token overlap, and the travel-insurance case study's winner used 512. Pick two or three sizes, run them with your retriever and reranker on a labeled eval set, and keep the winner.
Is Semantic Chunking Better Than Recursive Chunking?
Not consistently. In the NAACL 2025 study discussed above, semantic chunking helped on documents with sharp topic changes, but fixed-size chunking often did better on real-world datasets, and Chroma found that tuned recursive splitting performed well. Semantic chunking is worth testing for long, unstructured, mixed-topic text. For structured documents, start with recursive or structure-based splitting.