Garbage in, garbage out — even with the best LLM
You've deployed an internal chatbot powered by RAG. Users ask questions and get answers… often wide of the mark. The problem isn't the model — it's the quality of your document base. Poorly formatted PDFs, duplicates, outdated documents and rough splitting produce mediocre retrieval. The result: the LLM hallucinates or gives partial answers. A "RAG-ready" document base — cleaned, structured and intelligently split — improves answer relevance by 40 to 60%.
The problem
A 150-employee services SMB launches an AI assistant to answer staff questions about internal procedures. The document base comprises 1,200 documents: procedure PDFs, Word files of internal memos, Confluence wiki pages, and a few Excel dashboard files. The chatbot is deployed in 2 weeks. And the feedback is disastrous.
The most frequent problems:
- Outdated documents that pollute the answers — The expense reimbursement procedure exists in 4 versions: 2019, 2021, 2023 and 2024. The RAG sometimes retrieves the old version and the LLM generates an answer based on outdated rules. An employee gets reimbursed an incorrect amount.
- Unreadable scanned PDFs — 30% of the documents are poor-quality scans. The OCR produces text with 15% errors. The RAG retrieves these documents but the LLM can't extract meaning from them. The answers contain visible inconsistencies.
- Chunks that cut in the wrong place — A key 800-word paragraph explaining a complex procedure is split into 3 chunks without overlap. The central chunk, out of context, is incomprehensible. The LLM produces a partial or wrong answer.
- Duplicates and multiple versions — The same document exists in 3 different folders with varied names. The RAG retrieves the 3 versions and the LLM doesn't know which to prioritize. The user loses confidence in the tool.
The solution isn't to change LLM — it's to prepare the documents upstream. Discover our complete approach on the enterprise RAG page.
The AI solution
Building a RAG-ready base is a three-step process: sort, clean and split. AI assists each step to speed up processing and ensure quality.
AI-assisted audit and deduplication
An analysis script goes through your document base, identifies duplicates by semantic similarity (not just by file name), detects outdated documents by date and content analysis, and classifies documents by topic. The result: you go from 1,200 documents to 600 unique, up-to-date documents in 3 days instead of 3 weeks.
Automated conversion and cleaning
Each document is converted into structured text: text extraction from PDFs (with AI-enhanced OCR for scans), conversion of Excel tables into textual descriptions, normalization of formats (dates, amounts, names). Metadata is enriched automatically: title, author, date, category, keywords.
Intelligent splitting (semantic chunking)
Instead of splitting mechanically every 500 tokens, AI identifies the semantic boundaries: section endings, topic changes, conclusions. Each chunk is self-contained and understandable on its own. A 15% overlap ensures continuity. Measured gain: +35% relevance on retrieval compared to naive splitting.
Implementation
The process unfolds in three phases over 3 to 6 weeks depending on the document volume.
Audit and sorting (week 1)
Inventory all documents: location, format, size, last modified date. Run a deduplication script by hash and by semantic similarity. Classify documents into 3 categories: to keep (up to date and relevant), to archive (outdated but worth keeping), to delete (duplicate or out of scope). Validate the sorting with the relevant business teams.
Cleaning and conversion (weeks 2-4)
Convert each document into Markdown or structured text. For scanned PDFs, use a quality OCR (Azure Document Intelligence, Google Document AI or Tesseract 5 with AI post-processing). Normalize the metadata into a common schema. Manually check a 10% sample to validate the quality of the extraction.
Chunking and indexing (weeks 4-6)
Split the documents into semantic chunks of 500 to 1,000 tokens with 15% overlap. Add metadata to each chunk (source document title, section, date). Vectorize with an embedding model suited to French (CamemBERT, multilingual-e5, or OpenAI ada-002). Index into a vector database (Qdrant, Weaviate, Pinecone). Test with 50 real questions and measure recall.
Results
Here are the results measured at our clients after putting a RAG-ready base into production.
Frequently asked questions
What is a RAG-ready document base?
It's a set of documents that are structured, cleaned and split into segments (chunks) optimized to be retrieved by a RAG (Retrieval-Augmented Generation) system. Concretely, your PDFs, Word files, web pages and internal wikis are converted into clean text, enriched with metadata (date, author, category) and indexed in a vector database. When a user asks a question, the system retrieves the most relevant segments and sends them to the LLM to generate a contextualized answer.
Which document formats work best with RAG?
Structured text formats give the best results: Markdown, HTML, plain text with headings. Text-based PDFs work well after extraction. Scanned PDFs (images) require prior OCR, with a loss of quality. Excel files and relational databases require conversion into descriptive text. Avoid image-heavy PowerPoint presentations — the text of the slides is often too fragmented for effective RAG.
What chunk size should you choose for RAG?
The optimal size depends on the type of content and the model used. General rule: 500 to 1,000 tokens per chunk for technical documents, 200 to 500 tokens for FAQs or product sheets. An overlap of 10 to 20% between chunks improves continuity. Test with your real data: if the LLM lacks context, increase the size; if the answers are too vague, reduce it.
How long does it take to prepare a RAG-ready document base?
For a base of 500 to 2,000 documents, count on 3 to 6 weeks: 1 week of audit and sorting, 2 weeks of cleaning and conversion, 1 to 2 weeks of indexing and testing. The time depends mainly on the initial quality of the documents. Clean French PDFs with a clear structure are processed 5 times faster than scans of handwritten documents.
For tech profiles
Document preparation pipeline
Azure Document Intelligence / Docling
Azure Document Intelligence (formerly Form Recognizer) extracts text, tables and structure from PDFs with 95%+ accuracy. Docling (IBM, open source) offers a free alternative with layout extraction. For scans, Tesseract 5 + GPT-4 post-processing corrects common OCR errors.
LangChain / LlamaIndex
LangChain and LlamaIndex offer advanced chunking strategies: RecursiveCharacterTextSplitter (by size), SemanticChunker (by similarity), MarkdownHeaderTextSplitter (by structure). French embedding: CamemBERT-large, multilingual-e5-large or OpenAI text-embedding-3-small.
Recommended stack
Comparison of chunking strategies
| Criterion | Semantic chunking | Fixed-size chunking | By section/heading |
|---|---|---|---|
| Retrieval relevance | Excellent | Average | Good |
| Complexity | Medium (LLM required) | Simple | Medium |
| Cost | Moderate (LLM calls) | Near zero | Near zero |
| Suited to FR docs | Yes (multilingual model) | Risk of cutting | Yes (if headings present) |