RAG from scratch: from document to verified answer
A production RAG pipeline from ingestion, normalization, chunking, and embeddings through retrieval, reranking, grounded generation, citations, and evaluation.
Article contents
What RAG solves
Retrieval-Augmented Generation adds external knowledge fragments to a request. This lets a system work with private documents, fresh data, and sources that were not present during model training.
RAG does not update the LLM parameters. Knowledge remains in an external store, so it can be versioned, deleted, access-controlled, and cited.
Ingestion and normalization
PDFs, HTML, email, and tables contain noise: menus, duplicates, headers and footers, hidden characters, and incorrect reading order. If ingestion loses structure, retrieval cannot reconstruct it later.
Every fragment should retain document_id, version, source, date, language, access rights, and its position in the original. Metadata is required for filtering, citations, and removal of stale records.
Chunking and representation
Chunks that are too small lose context; chunks that are too large mix topics and consume valuable context-window space. Prefer semantic boundaries such as headings, paragraphs, policy clauses, or code functions.
After normalization, chunks are converted into embeddings. Vector search finds semantic similarity, but names, codes, and exact terms are usually better served by combining it with keyword search and metadata filters.
Retrieval, reranking, and grounding
The first-stage retriever quickly returns candidates, while a reranker more precisely decides which fragments actually answer the query. The model should receive a compact set of the strongest evidence.
The model should answer from the supplied sources and explicitly report insufficient evidence. Citations are built from real source_id values, and the system verifies that every identifier was present in the context.
Evaluating RAG
Retrieval and generation should be evaluated separately. If the correct document was not found, the problem is not the model. If the evidence was found but the answer adds unsupported claims, the problem is grounding or generation policy.
An evaluation set should contain real questions, expected sources, correct answers, and negative cases where the system must abstain. Useful metrics include retrieval recall, precision, groundedness, and citation accuracy.
A minimal production RAG architecture
A basic RAG stack consists of ingestion, normalization, chunking, embeddings, an index, retrieval, and answer generation. Production systems also need ACLs, document versioning, metadata filters, reranking, query logging, and a mechanism for deleting stale fragments. Without these layers, a prototype quickly becomes an unmanaged collection of vectors.
Every chunk should preserve provenance: document, version, page or section, indexing time, and access rights. That makes an answer explainable, verifiable, and revocable after a source changes. A vector without provenance is barely usable in a reliable system.
Evaluate retrieval separately from generation
When an answer is wrong, first determine whether the model received the required evidence. Evaluate retrieval with recall, precision, hit rate, MRR, or nDCG on queries with known relevant fragments. Evaluate generation separately: whether the answer used the sources, avoided contradictions, and cited them correctly.
This separation shortens diagnosis. If a relevant chunk was not retrieved, changing the prompt will not help; improve chunking, embeddings, query rewriting, or reranking. If the evidence is present but ignored, investigate context assembly, instructions, or the model.
Common reasons RAG fails
Typical failures include chunks that are too large or too small, missing metadata filters, duplicates, mixed document versions, weak user queries, and returning too many fragments. More context does not guarantee a better answer: irrelevant passages reduce signal and create contradictions.
Start with a small eval dataset, measure retrieval before adding the LLM, and log every stage. After launch, collect failed queries, classify their causes, and update the index or pipeline based on evidence rather than a few impressive demos.
End-to-end acceptance for the first RAG
Treat the first RAG as a measurable pipeline: ingestion, parsing, chunking, embedding, indexing, retrieval, reranking, generation, and citation validation. Each stage needs its own identifiers and traces so you can see where the correct document was lost. The evaluation dataset should contain real queries, expected sources, and completeness criteria. When evidence is insufficient, the answer should abstain or ask for clarification instead of compensating for weak retrieval with confident generation.
The production baseline includes access filters during retrieval, a versioned index, idempotent reindexing, and rollback. Metrics should cover recall@k, citation precision, groundedness, p95 latency, and cost per successful answer. After any change to embeddings, chunking, or prompts, rerun the same regression set.
- Separate retrieval errors from answer errors.
- Keep source IDs in the final response.
- Use a controlled reindex process.
Practical examples
Evidence package
The retriever returns source_id, document_version, fragment, score, and access policy. The answer may cite only fragments contained in that package.
FAQ
Does every RAG system need a vector database?
No. A small corpus may work well with PostgreSQL, a search engine, or even controlled keyword search.
Why does RAG fabricate citations?
This happens when the model generates references as free text. Source identifiers should be supplied structurally and validated after generation.
When should the index be updated?
On source-change events or according to a defined SLA. Every document version should remain traceable.