RAG Chunking Strategy: Picking Chunk Size and Overlap

Chunking is the highest-leverage decision in a RAG pipeline, and most teams make it by accident by accepting the framework’s default splitter. The short version: start at 300 to 500 tokens per chunk with 10 to 15 percent overlap, split on document structure rather than raw character counts, attach enough metadata that a chunk read in isolation still makes sense, then measure retrieval recall before tuning anything else.

Chunking is stage two of seven; if you are still assembling the pipeline, the RAG pipeline build order sets out what has to be right before this decision is even measurable.

Why chunking sets the ceiling

Everything downstream inherits the chunk boundary. A better embedding model cannot recover an answer that was cut in half at index time, and a larger context window only means you send more of the wrong text. Retrieval returns whole chunks, and that single fact drives every trade-off.

If chunks are too small, the embedding loses the context that made the passage meaningful. A chunk reading “This limit does not apply to enterprise tenants” is a fine sentence and a useless retrieval unit: the vector carries no signal about which limit. Small chunks also fragment one answer across several units, so a top-5 retrieval spends every slot rebuilding a single paragraph.

If chunks are too large, the embedding becomes an average of unrelated topics. Dense vectors are fixed-width whatever the input length, so a 2,000-token chunk covering installation, licensing, and troubleshooting lands near none of them. Large chunks also burn context budget and push relevant text into the middle of the prompt, where models attend to it less reliably.

Chunk size by content type

Content shape matters more than any universal number:

Content type	Suggested chunk size	Notes
Prose docs, KB articles, policies	300 to 500 tokens	Split on headings first
Dense reference material, specs	200 to 350 tokens	Facts are locally dense
Conversational transcripts, tickets	Whole turn or thread	Never split mid-turn
Source code	Whole function or class	Use a syntax-aware splitter
Tables and spreadsheets	Whole table, or row plus header	Rows without headers are unretrievable

The pattern: whenever a document has a natural unit a human would quote, that unit is the chunk.

Overlap: how much, and what it is for

Overlap stops an answer that straddles a boundary from being lost. It is insurance, not a quality knob.

10 to 15 percent of chunk size is the practical range. On a 400-token chunk that is 40 to 60 tokens, one or two sentences. Go much below and boundary-straddling answers slip through. Push past 20 percent and index size and cost climb while quality flattens, because near-duplicate chunks compete for the same top-k slots.

Two things cut how much overlap you need: structure-aware splitting, so chunks already end at heading or clause boundaries, and a sentence-aware splitter that backtracks rather than cutting mid-sentence. If heavy overlap measurably helps, the splitter is cutting in the wrong places.

Split on structure, not character counts

The default recursive splitter tries separators in order until the chunk fits. LangChain’s RecursiveCharacterTextSplitter, the usual reference implementation, defaults to paragraph break, line break, space, then bare character. Note the gap: no sentence level, so once a paragraph exceeds the cap the splitter drops straight to word and character boundaries. Sensible general-purpose behaviour, poor fit for structured documents. A better order of operations:

1. Parse to a structured intermediate. Markdown or HTML with headings intact beats raw text scraped from a PDF. A splitter cannot recover headings the extractor destroyed, so fix extraction before touching chunk size.
2. Split on the heading hierarchy. Every H2 or H3 section becomes a candidate chunk.
3. Only then apply a size cap. Oversized sections get subdivided at paragraph boundaries, with the heading path carried onto every piece.
4. Merge runt chunks. A 40-token section that is a heading plus one sentence belongs with its neighbour, not indexed alone.

Step 3 is where most quality is won or lost. A chunk beginning Billing > Invoices > Refund windows embeds far closer to a question about refund windows than the bare paragraph does.

Metadata does more work than chunk size

Every chunk should carry, at minimum:

• Source document ID and title
• Heading path, prepended into the embedded text, not merely stored alongside it
• A stable chunk ID and position, so you can fetch neighbours at query time
• Filterable attributes: product, version, language, tenant, publish date, access level

Filterable attributes matter for correctness, not convenience. A system serving several product versions that cannot filter by version will answer v2 questions with v1 documentation, and no chunk tuning fixes that. Access level is the same problem with worse consequences: filter inside the vector query rather than discarding results afterwards, because post-filtering lets restricted chunks consume top-k slots and squeeze out the answer the user is allowed to see.

Two techniques trade indexing cost for retrieval quality. Contextual prefixing prepends a generated line saying what the chunk is and where it sits in the parent document, written once at index time, which directly addresses the “which limit?” failure above. Late chunking embeds the whole document with a long-context model and pools token embeddings per chunk, so every vector carries document-level context.

Embedding model constraints

Chunk size lives inside the embedding model’s limits. OpenAI’s text-embedding-3-small and text-embedding-3-large both cap input at 8,192 tokens, returning 1,536 and 3,072 dimensions by default. Over-length input is rejected outright by the API, which is the safe failure. The quiet one sits upstream: client libraries and ingestion frameworks often truncate to fit, so indexing looks successful while the tail of every long chunk was never embedded. Check whether anything in your pipeline truncates for you.

Two consequences follow:

• Dimensionality is a storage and index decision. The text-embedding-3 family shortens output through a dimensions parameter, trading a little retrieval quality for a large cut in index size. Confirm your vector store indexes the width you pick: pgvector accepts vectors up to 16,000 dimensions but will not build an HNSW index above 2,000. To see what a given chunk size, overlap ratio and embedding width cost in vectors, index RAM and one-time embedding spend, run the numbers through the chunk and index sizing calculator.
• Model choice needs domain evaluation. The MTEB leaderboard is the right place to build a shortlist, but its rankings come from public retrieval sets. Domain jargon, non-English content, and code all reshuffle them. Evaluate two or three candidates on your own questions.

Hybrid search and reranking absorb chunking mistakes

Dense retrieval alone is weak on exact tokens such as product codes, error numbers, and function names, which are exactly the queries where users expect a precise hit.

Hybrid search runs a lexical index (BM25) alongside the dense index and fuses the result lists. Reciprocal rank fusion is the usual merge because it needs no score normalisation between the two systems; it scores each document by summing 1 / (k + rank) across lists, with k = 60 as the conventional constant.

Reranking puts a cross-encoder over the fused candidates. Retrieve wide, 50 to 100, rerank, then pass the top 5 to 10 to the generator. Cross-encoders score query and document together instead of comparing pre-computed vectors, which makes them much more accurate and much slower per pair, so they run over a shortlist and never the corpus.

Add both after you have a recall measurement, never before.

Measure recall before tuning anything

You cannot tune chunking by reading generated answers. The generator masks retrieval failures by improvising, and masks successes by ignoring context it was handed.

Build a golden set of 50 to 200 real questions, each labelled with the chunk that answers it. Support logs are the best source; questions invented by the team skew toward the documentation’s own vocabulary. Then track retrieval in isolation:

• Recall@k (k = 10, 20, 50): does the correct chunk appear at all? Chunking moves this most directly, and it caps everything downstream.
• nDCG@10: is it ranked near the top? This is the metric reranking moves.
• MRR: the mean of 1 divided by the rank of the first correct result, so it rewards one right answer placed high.

Change one variable at a time, re-index, re-measure. Chunk size, overlap, splitter strategy, and embedding model each need their own run, because their effects interact and a combined change tells you nothing.

Resist adopting a published pass mark: any specific recall target is borrowed from someone else’s corpus. Read the shape of your own numbers. While many golden questions never surface the right chunk at all, retrieval is the bottleneck and prompt engineering is wasted effort. Once the right chunk nearly always appears but ranks low, chunking has hit diminishing returns and the ranking layer holds the rest.

What to leave alone at first

• HNSW index parameters. Common defaults (m = 16 with a moderate ef_construction) are fine until recall says otherwise. When you do tune, raise the query-time ef first: it trades latency for recall without a rebuild. How much of this the store exposes at all varies: see Qdrant vs Milvus vs Pinecone.
• Exotic splitters. Semantic splitting cuts where consecutive sentence embeddings diverge, at the cost of an embedding pass over every sentence at index time. Try it only after measuring structure-aware splitting.
• Multiple embedding models in one index. Vectors from different models are not comparable, so switching models means a full re-index.
• Chunk size as a response to bad answers. If the right chunk is retrieved and the answer is still wrong, the fault is the prompt or the model, and shrinking chunks makes it worse. Locate the real stage first with the retrieval debugging checklist.

See also

• RAG pipeline architecture: components and build order
• Qdrant vs Milvus vs Pinecone: vector database comparison
• RAG retrieval debugging: why results come back wrong
• Chunk and index sizing calculator
• Production LLM telemetry and ops
• LLM observability and tracing
• Local LLM inference and VRAM sizing