What is retrieval-augmented generation?

Short answer

Retrieval-augmented generation puts a search step in front of the model: the system retrieves passages relevant to the user's question from a corpus it controls, places them in the prompt, and asks the model to answer from that material. It was introduced in 2020 as a way to combine parametric knowledge with an external, updatable store. In practice most RAG failures are retrieval failures — the model answers faithfully from passages that were the wrong passages.

How it works

Documents are chunked, embedded into vectors and indexed. At query time the system embeds the question, retrieves nearest neighbours, usually reranks them with a cross-encoder, and assembles a prompt containing the surviving passages plus instructions to answer only from them and to cite. Strong systems add lexical search alongside vector search, because exact identifiers, part numbers and error codes are precisely what embeddings blur.

Example

An internal support assistant asked "is the 90-day return window different for enterprise contracts?" needs the one clause that mentions both. Semantic similarity will happily return five paragraphs about consumer returns; a hybrid retriever plus reranking is what surfaces the clause. If it never enters the prompt, no amount of prompt engineering downstream recovers the answer.

Why it matters

RAG is the default architecture for grounding model output in material an organisation can audit, update and permission. It also makes the failure surface legible: you can measure retrieval separately from generation, which is the only reliable way to debug these systems.

Common misunderstandings

That RAG eliminates hallucination. It reduces unsupported claims when retrieval succeeds and can increase them when retrieved passages are wrong but plausible.

That a longer context window makes retrieval unnecessary. Cost, latency and mid-context degradation all argue against dumping whole corpora into prompts.

That a vector database is the hard part. Chunking, hybrid search, reranking and permission filtering usually decide quality.

What is retrieval-augmented generation?

Short answer

How it works

Example

Why it matters

Common misunderstandings

Limitations of this answer

Sources

Related answers