A clean index beats a better model
The leading cause of bad answers in an enterprise RAG is not the LLM — it's the quality of the index. Repeated headers, legal footers, duplicates, poorly parsed PDFs, obsolete versions: these artifacts pollute your vector base and degrade every query. Our audits show that 15 to 35% of the chunks of an uncleaned index are pure noise. Cleaning, deduplication and metadata enrichment improve context precision by 20 to 35% — a gain often greater than an embedding model change.
The problem
You've indexed 10,000 documents in your vector base. But how many chunks are actually useful? Your index probably contains thousands of parasitic fragments that blur the search results and degrade answer quality.
The most common sources of noise in an enterprise RAG index:
- PDF conversion artifacts — Headers, footers, page numbers, "Confidential" notices repeated on every page. A 200-page report generates 200 chunks containing "© 2024 MyCompany — All rights reserved" that pollute the retrieval.
- Duplicates and near-duplicates — The same document exists in Word, PDF and HTML versions. A procedure has been updated 5 times, and all 5 versions are indexed. The user receives an answer based on an obsolete version without knowing it.
- Poorly extracted tables and images — Tables of figures become sequences of numbers without context. Image captions are separated from their reference. The LLM receives unreadable tabular data and improvises.
- Absence of metadata — All chunks are treated the same way, whether they come from a validated 2025 procedure or a 2019 draft. The retriever cannot prioritize reliable and recent sources.
According to our RAG index audits at our clients, a significant share of chunks (often around a fifth) are pure noise, and an additional fraction are duplicates or obsolete versions. Cleaning this noise mechanically improves quality without touching the model.
The AI solution
A clean indexing pipeline is structured in three steps: cleaning at extraction, systematic deduplication, and metadata enrichment.
Cleaning at extraction
Use Unstructured.io or LlamaParse to extract structured content from your documents (PDF, Word, HTML). Configure removal rules for headers, footers, page numbers and recurring notices. Preserve the structure (titles, subtitles, tables) in Markdown. Treat tables as standalone chunks with an automatically added descriptive context.
Multi-level deduplication
First pass: exact hashing (SHA256) to eliminate identical files. Second pass: MinHash + LSH to detect near-duplicates at document scale (Jaccard similarity threshold > 0.85). Third pass: cosine similarity on chunk embeddings (threshold > 0.95) for redundant fragments. Keep only the most recent version of each document.
Metadata enrichment
Associate each chunk with a set of structured metadata: source, date, author, type, status, parent section. Use an LLM to automatically extract keywords and a summary of each chunk (cost: ~$0.005/chunk). Store the metadata in your vector store (Pinecone, Weaviate, Qdrant) to enable pre-retrieval filtering.
Implementation
Cleaning and re-indexing an existing document base takes 2 to 4 weeks. The investment is amortized within the first month through improved answer quality.
Index audit and diagnosis (week 1)
Sample 500 random chunks from your index and classify them manually: useful content, noise (header/footer), duplicate, obsolete, poorly formatted. Compute the signal/noise ratio. Identify the 3 to 5 main noise sources. This audit takes a day and provides the cleaning roadmap. Use a Python script that extracts the most similar chunks (cosine > 0.98) to quantify the duplicates.
Cleaning pipeline (weeks 2-3)
Build an ingestion pipeline that chains: structured extraction (Unstructured.io), regex cleaning (headers, footers, recurring notices), deduplication (hash + MinHash), metadata enrichment (LLM extraction), optimized chunking, and embedding. Version the pipeline configuration. Test on 10% of the corpus, compare the RAG metrics before/after, then progressively re-index the entire base.
Continuous monitoring (week 4+)
Set up a continuous ingestion pipeline that automatically applies cleaning to each new document. Monitor the signal/noise ratio with weekly sampling. Configure alerts when the duplicate rate exceeds 5% or when unknown noise patterns appear. Plan a complete re-audit every 6 months.
Results
Results measured at our clients after cleaning and re-indexing their RAG document base.
FAQ
Why is document cleaning so important for a RAG?
Because the embedding model does not distinguish useful content from noise. A header repeated over 200 pages generates 200 nearly identical chunks that pollute your index. A recurring legal footer takes the place of real results in the retriever's top-K. By cleaning these artifacts before indexing, you improve context precision by 20 to 35% on average.
How do you detect duplicates in a vector index?
Three complementary approaches: exact hashing (MD5/SHA256) for perfect duplicates, cosine similarity on embeddings (threshold > 0.95) for near-duplicates, and MinHash/LSH for large-scale deduplication. In practice, combine an exact hash in a first pass then similarity-based deduplication in a second pass. Expect to find 10 to 30% duplicates in an uncleaned index.
Which metadata should be added to chunks to improve retrieval?
The essential metadata are: document title, section title, creation/modification date, author, document type (contract, procedure, FAQ), and a unique identifier. Advanced metadata include: department, related product, status (draft/validated/archived) and automatically extracted keywords. This metadata enables pre-retrieval filtering that reduces noise by 40% and speeds up search.
For technical profiles
Clean indexing pipeline: reference architecture
The indexing pipeline is organized into 5 sequential steps, each producing a versioned intermediate artifact. The whole is orchestrated by an Airflow or Prefect DAG, with automatic retry and structured logging.
Pipeline steps:
- 1. Extraction — Unstructured.io (PDF, DOCX, PPTX) or LlamaParse (complex PDFs with tables). Output: structured Markdown with YAML front-matter metadata.
- 2. Cleaning — Configurable regex for headers/footers + boilerplate detection via TF-IDF (passages that appear in >30% of documents are noise). Unicode normalization.
- 3. Deduplication — SHA256 hash (files) + MinHash with 128 permutations (documents, Jaccard threshold 0.85) + cosine on embeddings (chunks, threshold 0.95).
- 4. Enrichment — Metadata extraction by LLM (GPT-4o-mini, cost ~$0.005/doc): summary, keywords, category, named entities.
- 5. Chunking + Embedding — Chunking strategy adapted to the type (see chunking article), embedding via Cohere Embed v3 or OpenAI text-embedding-3-large.
Comparison of extraction tools
| Criterion | Unstructured.io | LlamaParse | PyMuPDF + custom |
|---|---|---|---|
| Complex PDFs (tables) | Good | Excellent | Medium |
| Supported formats | 25+ formats | PDF only | PDF only |
| Structure preservation | Excellent | Excellent | Basic |
| Open source | Yes | Freemium | Yes |
| Cost | Free (self-host) | $0.003/page | Free |