A RAG pipeline that survives real documents

RAG demos use clean markdown. Real work arrives as a scanned purchase order at a slight angle, a 400-page contract with a table spanning eleven pages, and a spreadsheet somebody exported to PDF.

The gap between those two worlds is where every retrieval project I have rescued was stuck. And in almost every case the team was tuning the embedding model when the damage had already happened two steps earlier.

Read your chunks. Today.

Before any other work, dump fifty random chunks from your index and read them as text. Not metrics. The actual strings.

What people find: page numbers and running headers in every chunk, sentences cut mid-clause, table rows flattened into unreadable runs of numbers, footnote markers glued to words, and the same legal boilerplate occupying a third of the index.

Extraction is the real project

Documents fall into three classes and each needs a different path. Trying to use one parser for all three is the root cause of most bad indexes.

1. Born-digital text. Extract with a layout-aware parser, keep the reading order, drop the furniture.
2. Scans and images. Needs OCR, and needs deskewing and quality checks first. Track a confidence score per page and route low confidence pages to a human or a vision model rather than indexing garbage silently.
3. Tables and forms. Do not flatten them into prose. Emit structured rows, then generate a short natural-language description per row group for the embedding, while keeping the structured version for answers.

That last technique is the one that most improved my numbers on financial and legal corpora. Embed a sentence a human would say. Answer from the structured data.

Chunk on structure, not on character count

Fixed-window chunking is convenient and wrong for documents with sections. Split on headings and paragraph boundaries, then merge small pieces up to a target window and split oversized ones at sentence boundaries.

• Target roughly 300 to 800 tokens, with modest overlap.
• Prefix every chunk with its document title and section path. This alone fixes a surprising share of "the retrieval found the right page but the answer was wrong" cases.
• Never let a chunk begin or end mid-sentence. It costs nothing to avoid and hurts measurably when you do not.
• Store metadata: source, section, page, effective date, document type. Cheap to add, and it enables filtered retrieval later.

Retrieve hybrid, then rerank

Vector-only retrieval fails on exactly the queries business users care about: an invoice number, a part code, a specific clause name. Keyword-only fails on paraphrase. Run both and fuse.

1. Retrieve 20 to 40 candidates from vector search and from keyword search in parallel.
2. Fuse the ranked lists, then rerank with a cross-encoder or a small model.
3. Pass the top three to five to the model, with citations attached.
4. Apply metadata filters before search, not after. Filtering post-hoc throws away candidates you paid for.

Passing fewer, better passages is also a context and cost decision, which connects directly to context engineering and spend control.

Grounding, or the whole thing is theater

Require every factual claim in the answer to carry a citation to a retrieved chunk, and validate that programmatically. Uncited claims get stripped or the answer gets flagged.

This does two things. It gives users a way to verify, which is what makes the system trustworthy in a legal or financial setting. And it gives you a hallucination metric that needs no judge model, which becomes a first-class signal in your eval suite.

Their embedding model was fine. Every chunk started with the same eleven-word confidentiality header, so everything was similar to everything.

Evaluate retrieval separately from generation

A single end-to-end score cannot tell you which half is broken. Measure them apart.

• Retrieval. For a set of real questions with known source passages, measure whether the right passage appears at all, and where it ranks.
• Generation. Given the correct passages, is the answer right and fully cited.

Ninety percent of the time the failure is retrieval, and teams have spent two weeks rewriting the answer prompt. Splitting the metric prevents that entirely.

Operational details that decide whether it lasts

• Incremental reindexing keyed on content hash, so reprocessing a corpus is not an all-night job.
• Versioned pipelines. Store which extractor and embedding version produced each chunk. Without this, mixed-quality indexes become unfixable.
• Permission filtering at query time, enforced in the database rather than in the prompt. Retrieval is a data access path and must respect the same rules as any other.
• Freshness. Carry effective dates and prefer current documents, or you will confidently answer from a superseded contract.

The order to work in

1. Read fifty chunks. Fix extraction until they read like documents.
2. Rechunk on structure with title and section prefixes.
3. Add keyword search alongside vectors and fuse.
4. Add a reranker and cut what you pass to three to five passages.
5. Enforce citations and measure grounding.
6. Split retrieval and generation evals, then iterate on the weak one.

That sequence has never failed me, and it is almost the exact reverse of where most teams start. If you have a retrieval system that demos well and disappoints in production, the AI production readiness audit will tell you which step is costing you, and agent development covers building it properly.

Frequently asked