Chunking is the #1 factor in a RAG's quality
Before changing your embedding model or LLM, check your chunking strategy. The way you split your documents into fragments determines 60% of the quality of the results of your RAG chain. A chunk that is too small loses context. A chunk that is too large drowns the relevant information in noise. A split that cuts a sentence in two makes the passage unusable. The best practices: adapt the size to the document type, use an overlap of 10 to 20%, and enrich each chunk with metadata (section title, page number, date).
The problem
Your AI assistant gives partial answers. It cites a paragraph that starts in the middle of a sentence. It mixes information from two different sections. Or it simply can't find the information even though it does exist in the document base. In 60% of cases, the problem isn't the LLM — it's the chunking.
Chunking (splitting documents into indexable fragments) is the first step of the RAG pipeline, and it's the one most often neglected. Most teams use a RecursiveCharacterTextSplitter with the default parameters (1000 characters, 200 overlap) without ever measuring the impact on retrieval quality.
The most common problems of poor chunking:
- Orphan chunks — A fragment starts with "Moreover, this rate…" without the preceding context being available. The retriever surfaces an incomprehensible passage, the LLM hallucinates to fill the gaps.
- Oversized chunks — A 2000-token chunk contains 3 different topics. The retriever selects it for one topic, but the LLM gets distracted by the other two. Result: a diluted answer that mixes the themes.
- Loss of structure — Titles, subtitles, tables and bullet lists are flattened into plain text. The LLM loses the hierarchy of information and treats a section title as a content sentence.
- Absence of metadata — Impossible to filter by date, author or document type. The retriever searches the whole base when a simple filtered query would have found the result first.
The AI solution
An effective chunking strategy combines three complementary approaches, adapted to the nature of your documents and your use cases.
Structural chunking (by section)
Split following the document's native structure: titles, subtitles, paragraphs. Use Markdown or HTML markers as natural split points. Each chunk inherits its parent section's title as metadata. Ideal for technical documentation, wikis and manuals. Implementation: MarkdownHeaderTextSplitter (LangChain) or SentenceSplitter (LlamaIndex).
Semantic chunking (by theme)
Use an embedding model to detect thematic breaks in a long text. When the cosine similarity between two consecutive sentences drops below a threshold (e.g. 0.75), it's a natural split point. Better for long narrative documents (reports, studies, articles). Implementation: SemanticChunker (LangChain) or with sentence-transformers.
Metadata enrichment
Each chunk is associated with a set of metadata: parent section title, page number, document date, author, document type. This metadata enables pre-retrieval filtering that reduces noise by 40%. Essential when your base contains thousands of documents of different kinds.
Implementation
Chunking optimization is done in three iterative phases. Each phase produces measurable results thanks to RAG evaluation metrics.
Document base audit (week 1)
Classify your documents by type: technical (manuals, API docs), legal (contracts, terms of service), commercial (product sheets, presentations) and conversational (emails, tickets). For each type, analyze the structure (titles, tables, lists) and the average length. Define a chunking strategy per document type: target size, splitting method, metadata to extract. Document everything in a versioned configuration file.
Implementation and benchmark (weeks 2-3)
Implement 3 chunking variants for your main corpus: fixed size (512 tokens), structural (by section) and semantic. For each variant, re-index a sample of 500 documents and run your evaluation Golden Set. Compare the context recall and context precision scores. In our experience, structural chunking outperforms fixed size by 15 to 25% on well-formatted documents.
Fine optimization and monitoring (week 4)
Adjust the parameters of the winning strategy: chunk size (test 256, 512, 768, 1024), overlap (10%, 15%, 20%), included metadata. Set up an automatic re-indexing pipeline that triggers on each document addition. Monitor context recall in production to detect degradations. Plan a quarterly review of the chunking strategy.
Results
Results measured at our clients after optimizing the chunking strategy on enterprise RAG projects.
FAQ
What is the ideal chunk size for a RAG?
There is no universal size. For technical or legal documents, 512 to 1024 tokens offer the best compromise between sufficient context and retrieval precision. For FAQs or product sheets, 256 to 512 tokens are preferable. The rule of thumb: a chunk should contain a complete and self-sufficient idea. Test 3 different sizes on your Golden Set and measure the context recall.
What is overlap and why is it important?
Overlap consists of repeating the last sentences of a chunk at the beginning of the next chunk. An overlap of 10 to 20% of the chunk size prevents cutting a piece of information into two incomplete parts. For example, with 512-token chunks and a 64-token overlap, the last 64 tokens of chunk N are also the first 64 of chunk N+1. This improves recall by 5 to 15% according to our benchmarks.
Should you use semantic chunking or fixed-size chunking?
Fixed-size chunking (RecursiveCharacterTextSplitter in LangChain) is sufficient for 80% of use cases. Semantic chunking (which detects thematic breaks via embeddings) improves results by 10 to 20% on long, heterogeneous documents (annual reports, multi-topic technical documentation). Start with fixed chunking, measure your metrics, then test semantic chunking only if the context precision is insufficient.
For technical profiles
Technical implementation of chunking
The choice of chunking strategy depends on your technical stack and the nature of your documents. Here are the reference implementations with LangChain and LlamaIndex.
Recommended configuration per document type:
- Technical documentation (Markdown/HTML) — MarkdownHeaderTextSplitter + chunk_size=768, overlap=128. Preserve titles as metadata.
- Complex PDFs (tables, columns) — Extraction via Unstructured.io or PyMuPDF, then structural chunking. Treat tables as standalone chunks.
- Legal contracts — Chunking by article/clause via regex, chunk_size=1024, overlap=200. Metadata: article number, date, parties.
- FAQ / product sheets — One chunk per question-answer or per sheet. No overlap needed. Metadata: category, product, update date.
Comparison of chunking strategies
| Criterion | Structural (sections) | Fixed size | Semantic |
|---|---|---|---|
| Context Recall | 0.82 | 0.65 | 0.85 |
| Context Precision | 0.78 | 0.60 | 0.80 |
| Implementation complexity | Medium | Low | High |
| Compute cost | Low | Low | Medium (embeddings) |
| Suited to structured docs | Excellent | Fair | Good |