To chunk text for RAG, convert each document to structured text, split on its headings, then cap every section at 256 to 512 tokens counted with the embedding model’s own tokenizer. Prepend the heading path to each chunk, keep overlap small, and judge the result by evidence recall at a fixed context-token budget, not by reading answers.
The failure this prevents is quiet. Splitters often count characters; embedding models count word pieces. The all-MiniLM-L6-v2 model card states that input longer than 256 word pieces is truncated by default, so the tail of any longer chunk never reaches the vector. Indexing succeeds, nothing alerts, recall sags, and the generator improvises over the gap.
Picking the size and overlap numbers is covered in the chunking strategy guide. This post is the implementation side: which splitter, what code, and what to measure so a re-chunk is a controlled experiment rather than a guess.
Which chunking method should you pick?
Pick structure-aware recursive splitting as the default: split on Markdown or HTML headings, then recursively on paragraphs and sentences under a token cap. It costs one tokenizer pass. Add contextual prefixes when recall plateaus. Semantic and LLM-driven chunkers cost far more at index time, and the evidence that they beat fixed-size splitting on real documents is thin.
| Method | Where it cuts | Index-time cost | Use when |
|---|---|---|---|
| Fixed token window | Every N tokens | Tokenizer only | Baselines, homogeneous text |
| Recursive | Paragraph, then line, then word | Tokenizer only | General prose |
| Structure-aware | Headings first, then recursive | Parser plus tokenizer | Docs, wikis, knowledge bases |
| Semantic | Where adjacent-sentence embeddings diverge | One embedding per sentence | Unstructured, topic-mixed text |
| LLM-driven | Model picks boundaries | LLM call per window | Small, high-value corpora |
| Contextual prefix | Any method, plus an LLM-written context line | LLM call per chunk | Recall has plateaued |
| Late chunking | Embed whole document, pool per chunk span | Long-context embedding model | Text heavy with cross-references |
One detail matters in the recursive row. LangChain’s recursive splitter defaults to ["\n\n", "\n", " ", ""]. There is no sentence level, so an oversized paragraph gets cut at a word boundary mid-sentence. Add ". " to the list.
What do chunking benchmarks actually show?
They show modest gaps, no universal winner, and nearly all of them are vendor benchmarks. Use them as a shortlist of configurations to run on your own golden set, not as a setting to copy. The consistent signal is that small, structure-respecting chunks hold up well and heavy overlap buys little.
- Chroma (vendor benchmark). Chroma’s evaluation ran 472 queries over five corpora with
text-embedding-3-large, retrieving five chunks.RecursiveCharacterTextSplitterat 200 tokens, no overlap, reached 88.1 recall. Chroma’s own ClusterSemanticChunker reached 91.3 recall at 400 tokens and its GPT-4o-driven LLM chunker 91.9, at similar precision (4.5 and 3.9); the cluster chunker’s best precision, 8.0, came at 200 tokens, where its recall was 87.3. OpenAI’s Assistants default, run as a token splitter at 800 tokens with 400 overlap, scored 87.9 recall and 1.4 precision. Chroma built both top chunkers; weigh accordingly. - NVIDIA (vendor benchmark). NVIDIA’s study tested 128 to 2,048 token chunks at 15 percent overlap against page-level chunking. Page-level had the best average end-to-end accuracy (0.648) and the lowest standard deviation (0.107), yet the best fixed size varied by dataset, from 512 to 1,024 tokens. Factoid queries favoured 256 to 512 tokens; analytical ones, 1,024 or full pages.
- Qu, Tu and Bao (preprint). Is Semantic Chunking Worth the Computational Cost? found semantic chunking won on artificially stitched, topic-mixed documents, while fixed-size chunking often did better on realistic ones. Two of the three authors are at Vectara, a RAG vendor, so call it semi-independent. It is not a product benchmark.
- Anthropic (vendor benchmark). Contextual Retrieval prepends a 50 to 100 token, LLM-written note situating each chunk in its document. Top-20 retrieval failure fell from 5.7 to 3.7 percent with contextual embeddings, 2.9 percent adding contextual BM25, and 1.9 percent with reranking on top.
Late chunking attacks the same context-loss problem from the embedding side: run the whole document through a long-context model, then mean-pool token embeddings per chunk span, with no extra training.
The metric that matters: recall at a fixed token budget
Each golden question carries a short evidence quote copied verbatim from the source document. Score a hit when that quote sits inside a retrieved chunk, counting only the top-ranked results that fit in budget B:
recall@B = (1/|Q|) * sum over q of
1[ evidence_q is contained in a chunk within the highest-ranked
results whose cumulative size is <= B tokens ]Plain recall@k rewards big chunks for being big. Five 800-token chunks carry four times the text of five 200-token chunks, so they contain the evidence more often by volume, then spend the generator’s context on padding. Chroma’s 800/400 result shows the cost side: recall within four points of the leaders, precision about a third of theirs. Set B to the retrieval context you actually send the generator, and every configuration competes for the same budget.
Evidence quotes beat chunk-ID labels for a second reason: chunk IDs change on every re-chunk, so an ID-keyed golden set needs relabelling per experiment, which nobody does. A quote survives any re-chunk and scores boundary failures correctly. If the split lands mid-quote, no chunk contains it, and that counts as a miss.
Wiring it up
The chunker below splits Markdown on headings, caps each section using the embedding model’s tokenizer, re-attaches the heading path, and refuses to emit anything the model would truncate.
from langchain_text_splitters import (
MarkdownHeaderTextSplitter,
RecursiveCharacterTextSplitter,
)
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
tok = model.tokenizer
MAX_TOKENS = model.max_seq_length # 256 for this model
TARGET = min(400, MAX_TOKENS) - 32 # headroom: heading prefix + special tokens
header_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")],
strip_headers=True, # headings move to metadata; re-added below
)
body_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
tok,
chunk_size=TARGET,
chunk_overlap=TARGET // 10,
separators=["\n\n", "\n", ". ", " ", ""],
)
def chunk_markdown(doc_id: str, markdown: str) -> list[dict]:
sections = header_splitter.split_text(markdown)
pieces = body_splitter.split_documents(sections)
chunks = []
for i, piece in enumerate(pieces):
path = " > ".join(
piece.metadata[h] for h in ("h1", "h2", "h3") if h in piece.metadata
)
text = f"{path}\n{piece.page_content}" if path else piece.page_content
n_tokens = len(tok(text)["input_ids"])
if n_tokens > MAX_TOKENS:
raise ValueError(f"{doc_id}:{i} is {n_tokens} tokens, cap {MAX_TOKENS}")
chunks.append({
"id": f"{doc_id}:{i:05d}",
"doc_id": doc_id,
"heading_path": path,
"text": text,
"n_tokens": n_tokens,
})
return chunksThree choices are deliberate. from_huggingface_tokenizer makes chunk_size count the embedder’s tokens; LangChain’s token-splitting docs warn that CharacterTextSplitter.from_tiktoken_encoder can emit chunks larger than chunk_size, and a tiktoken count is not a WordPiece count anyway. The heading path goes into the embedded text, because the header splitter strips headings into metadata by default. And over-length chunks raise instead of truncating.
Evaluation then logs one MLflow run per configuration. retrieve is your vector store query, returning chunks ranked by score with n_tokens in the payload.
import mlflow
import numpy as np
def norm(s: str) -> str:
return " ".join(s.split()).lower()
def recall_at_budget(golden, retrieve, budget: int) -> float:
hits = 0
for q in golden: # {"question": str, "evidence": str}
needle, used = norm(q["evidence"]), 0
for c in retrieve(q["question"], k=50):
used += c["n_tokens"]
if used > budget:
break
if needle in norm(c["text"]):
hits += 1
break
return hits / len(golden)
sizes = np.array([c["n_tokens"] for c in chunks])
with mlflow.start_run(run_name=f"md-headers-{TARGET}"):
mlflow.log_params({
"splitter": "md_header+recursive",
"chunk_size": TARGET,
"chunk_overlap": TARGET // 10,
"embed_model": "all-MiniLM-L6-v2",
})
mlflow.log_metrics({
"recall_at_2k_tokens": recall_at_budget(golden, retrieve, 2000),
"recall_at_4k_tokens": recall_at_budget(golden, retrieve, 4000),
"chunk_tokens_p50": float(np.percentile(sizes, 50)),
"chunk_tokens_p99": float(np.percentile(sizes, 99)),
"runt_ratio": float((sizes < 32).mean()),
"n_chunks": len(chunks),
})Change one variable per run. Splitter, size, overlap and embedding model interact, so a combined change tells you nothing.
What you’ll see
Chunk-size histogram. Healthy looks like a cluster just under the cap, from long sections the recursive splitter packed, plus a spread of short whole sections. Unhealthy looks like a tall bar of runts, usually heading-plus-one-line sections produced by header splitting, which belong merged into a neighbour. A p99 creeping toward MAX_TOKENS means the prefix headroom is too thin and the raise will start firing on long heading paths.
Run comparison in MLflow. A good change lifts recall_at_2k_tokens and narrows its gap to recall_at_4k_tokens. If you also log plain recall@10, the bad pattern is recall@10 climbing with chunk size while recall at 2k falls: big chunks winning on volume. A wide, stable gap between 2k and 4k means evidence is retrieved but ranked low, a job for hybrid search or a reranker rather than the splitter; the retrieval debugging checklist separates the two. Log per-document-type recall as well, since tables and code tend to regress first under splitters tuned on prose.
Caveats
- Golden set leakage. Questions written by reading the current chunks share their vocabulary and boundaries, inflating recall and favouring the configuration you already have. Source questions from support tickets and search logs, and pick evidence quotes from the source document, not from chunks.
- Quote length bias. Long evidence quotes straddle boundaries more often, which penalises small chunks. Keep each quote to one sentence.
- Re-chunking means re-embedding. Every run re-embeds the whole corpus. Contextual prefixes add an LLM call per chunk; Anthropic quotes $1.02 per million document tokens with prompt caching, a vendor figure tied to its pricing at publication.
- Late chunking changes the embedder. It needs a long-context embedding model, so it is a model decision as much as a chunking one; see the embedding model guide before committing.
- Extraction comes first. A splitter cannot recover headings that PDF extraction destroyed; fix parsing before tuning anything here, in the order the RAG pipeline build guide sets out.
FAQ
What chunk size should a RAG pipeline start with?
There is no universal best, but 256 to 512 tokens is a sound starting range for factual lookups. NVIDIA’s vendor benchmark found factoid queries favoured that range while analytical queries did better at 1,024 tokens or full pages. Your embedding model’s input limit is the hard ceiling, so check it before picking any number. LlamaIndex users can map the number onto its settings with the LlamaIndex chunk size guide, and sizes by content type are in the chunking strategy guide.
Should RAG chunks overlap?
A small overlap, one or two sentences, is enough when you split on headings and sentences. Chroma’s vendor benchmark found reducing overlap improved IoU, a score that penalises redundant text, and OpenAI’s 800-token, 400-overlap default scored poorly on it. If heavy overlap measurably helps, the splitter is probably cutting mid-sentence and needs better separators. The overlap range and its trade-offs are covered in the chunking strategy guide.
Is semantic chunking better than fixed-size chunking?
Usually not on real documents. A 2024 preprint by Qu, Tu and Bao found semantic chunking won mainly on artificially stitched, topic-mixed text, while fixed-size chunking often matched or beat it on realistic documents. Semantic chunking also costs one embedding per sentence at index time, so measure structure-aware splitting first.
What happens if a chunk exceeds the embedding model’s input limit?
Depending on the stack, it is either rejected or silently truncated, and truncation is the dangerous case: only the opening tokens are embedded and the rest of the chunk becomes unsearchable. The all-MiniLM-L6-v2 model card says input past 256 word pieces is truncated by default. Count tokens with the model’s own tokenizer and fail loudly.
Related across the network
- LangChain RAG Pipeline: Setup to First Answer — langchainguide.com
- LlamaIndex Ingestion, Indexing and Retrieval Explained — llamaindexhub.com
