RAG, or retrieval-augmented generation, is an architecture where an AI system retrieves relevant passages from a knowledge source and passes them to a large language model as context, so the model's answer is grounded in that information and can cite sources.

Does RAG eliminate hallucinations?

RAG reduces hallucinations but does not eliminate them. The model can still misread or go beyond the retrieved context, so instruct it to answer only from the context, show citations, and evaluate faithfulness regularly.

Is RAG better than fine-tuning?

For answering questions from private or frequently changing documents, RAG is usually better and cheaper. Fine-tuning is better for teaching consistent style, format, or specialized tasks. Many production systems combine both.

What are the main components of a RAG system?

A RAG system includes document ingestion and chunking, an embedding model, a vector or hybrid search index, an optional reranker, a large language model for generation, and an evaluation and monitoring layer.

What Is RAG? Retrieval-Augmented Generation Explained

Key takeaways

RAG retrieves relevant information from your own sources and gives it to a large language model (LLM) as context, so answers are grounded and can cite sources.
It is usually the fastest, cheapest way to make an LLM useful on private or frequently changing data.
Retrieval quality determines answer quality. Most RAG problems are retrieval problems.
Evaluate both retrieval and generation, and enforce document permissions at retrieval time.

Retrieval-augmented generation (RAG) is an AI architecture in which a system first retrieves relevant passages from a knowledge source, such as company documents in a vector database, and then passes them to a large language model, which writes an answer based on that retrieved context. The approach was formalized in a 2020 paper by Lewis et al. at Facebook AI Research, and it has become the standard way to build AI assistants that answer from private data.

Why use RAG?

Fresh and private knowledge: the model can use information it was never trained on, updated whenever your documents change.
Fewer hallucinations: answers are grounded in retrieved text rather than the model's memory alone.
Citations: users can check the source passage.
Access control: you can retrieve only documents the user is allowed to see.
Lower cost than retraining for most knowledge problems.

The components of a RAG pipeline

Indexing (offline)

Ingest documents from sources such as file stores, wikis, ticketing systems, and databases.
Parse and clean text, tables, and metadata.
Chunk content into passages, typically a few hundred tokens, respecting headings and sections.
Embed each chunk with an embedding model. See our embeddings guide.
Store vectors, text, and metadata such as source, date, and permissions in a vector database.

Answering (online)

Understand the query, optionally rewriting or expanding it.
Retrieve candidate chunks with vector, keyword, or hybrid search, applying permission filters.
Rerank candidates with a more precise model, such as a cross-encoder.
Generate an answer with the LLM using the top passages, with instructions to cite sources and to say when the answer is not in the context.

Advanced RAG techniques

Hybrid search combines keyword and semantic retrieval, which helps with product codes, names, and acronyms.
Reranking improves precision before the LLM sees the context.
Context ordering: research titled "Lost in the Middle" found that models use information at the start and end of long contexts better than information in the middle. Send fewer, better passages.
Metadata filtering by date, product, region, or permission.
Agentic RAG, where an agent decides which sources to query and can search iteratively. See our AI agent guide.

RAG vs. fine-tuning

Need	Better fit
Answer from private or changing documents	RAG
Cite sources and enforce permissions	RAG
Consistent style, format, or tone	Fine-tuning, or prompting with examples
Specialized classification or extraction at high volume	Fine-tuning a smaller model

Many systems combine both. Read more in LLM development: RAG vs. fine-tuning.

Evaluating RAG

Measure retrieval and generation separately:

Retrieval: recall at k and precision. Did the right passages appear?
Generation: faithfulness (is the answer supported by the context?), answer relevance, and correctness against a reference.

Frameworks such as RAGAS provide automated metrics, but always include human review on a sample.

Security considerations

Retrieved documents can contain malicious instructions (indirect prompt injection), and poorly filtered retrieval can leak confidential data. The OWASP Top 10 for LLM Applications lists both risks. Enforce permissions in the retrieval layer, not in the prompt.

A worked example: one question through a RAG pipeline

An employee asks a company assistant: "How many days of paid leave do new employees in the UK office get?" Here is what happens:

Query processing: the system identifies the user, their office, and their permissions, and may rewrite the question to improve retrieval, for example adding "annual leave policy".
Retrieval: the question is embedded and searched against the indexed HR policies, filtered to documents the employee can access and to the UK region. Keyword search runs in parallel to catch exact terms such as "annual leave".
Reranking: a reranking model scores the top candidates and keeps the few passages most relevant to the question.
Prompt assembly: the passages are placed in the prompt with their titles and sources, along with instructions to answer only from them and to cite the source.
Generation: the model writes a short answer, such as the number of days and any conditions, and cites the policy document and section.
Post-processing: the system checks the answer has citations, logs the question, sources, and response, and shows the answer with links to the policy.

If no relevant passage is found, the system says it does not know and suggests contacting HR, rather than guessing.

Chunking strategies compared

Strategy	How it works	Best for
Fixed size	Split every few hundred tokens with some overlap	Simple, uniform text
Structure-aware	Split by headings, sections, or paragraphs	Manuals, policies, documentation
Semantic	Split where the topic changes, using embeddings	Long, loosely structured text
Parent-child	Search small chunks but return their larger parent section	Precise retrieval with enough context
Table and code aware	Keep tables and code blocks intact	Technical and financial documents

Include the document title and section heading with each chunk, because a passage like "This applies after 30 days" is meaningless without context.

Common RAG failure modes and fixes

The right document is not retrieved: improve chunking, add hybrid search, use query rewriting, or try a better embedding model.
The right document is retrieved but ranked too low: add a reranker or retrieve more candidates before reranking.
The answer ignores the context: strengthen instructions, reduce irrelevant passages, and place the most relevant context clearly.
Answers are outdated: re-index changed documents automatically and remove deleted ones from the index.
Answers mix up similar documents: add metadata filters, such as product, version, or region.
Users see documents they should not: enforce permissions with server-side filters at retrieval time.

RAG cost and latency

Each RAG request involves an embedding call, one or more searches, optional reranking, and a model call with the retrieved context. Context length usually dominates cost and latency, so retrieving fewer, better passages saves money and often improves answers. Cache embeddings for repeated queries, use a smaller model for simple questions, and stream responses so users see text quickly. Monitor cost per question as usage grows.

Where RAG is heading

RAG systems increasingly combine retrieval with agents that decide when and where to search, use multiple tools, and check their own answers. Graph-based retrieval, which uses relationships between entities, helps with questions that span many documents. Long-context models reduce the need for aggressive chunking in some cases, but retrieval remains important for cost, freshness, permissions, and traceability. Our AI agent guide covers agentic retrieval, and our production RAG guide walks through implementation step by step.

Writing the RAG prompt

The generation prompt has a large effect on answer quality. A reliable RAG prompt does four things. First, it sets the rules: answer only from the provided context, cite the source for each claim, and say clearly when the context does not contain the answer. Second, it labels the context, giving each passage an identifier, title, and source so citations can be matched back to documents. Third, it separates instructions from content, so text inside a document cannot override the rules. Fourth, it specifies the format, such as a short direct answer followed by supporting detail and a list of sources.

Keep the number of passages modest. Including many loosely related passages can confuse the model and increase cost. Three to eight highly relevant passages are usually more effective than twenty mediocre ones.

Measuring RAG quality in detail

Evaluate retrieval and generation separately, because they fail for different reasons:

Retrieval recall: for each test question, is the passage containing the answer among the retrieved results?
Context precision: how much of the retrieved context is actually relevant?
Faithfulness or groundedness: is every claim in the answer supported by the retrieved passages?
Answer relevance and completeness: does the answer address the question fully?
Citation accuracy: do the cited sources actually support the statements?
Refusal behavior: does the system decline appropriately when the answer is not available?

Use a test set of real questions with reference answers and source documents. Combine automated scoring, often using a model as a judge, with human review of a sample. Track scores over time and rerun them whenever you change the chunking, embedding model, retrieval settings, prompt, or generation model.

Keeping the index fresh

A RAG system is only as current as its index. Build ingestion as an automated pipeline rather than a one-off script. Detect new, changed, and deleted documents in each source, re-chunk and re-embed changed documents, and remove deleted ones so outdated information is never returned. Record when each document was last indexed and show source dates to users where relevant. For sources that change constantly, such as ticket systems or inventories, consider querying live data through tools rather than indexing it.

Access control in RAG systems

Retrieval must respect the same permissions as the source systems. Store access information, such as groups, roles, or tenant IDs, as metadata with each chunk, and filter every query on the server using the authenticated user's permissions. Sync permission changes from source systems regularly. Avoid the shortcut of giving the assistant access to everything and asking the model to withhold restricted information, because models cannot be relied on to enforce access rules.

When RAG is not the right tool

RAG answers questions from documents. It is less suitable when the task requires precise calculations over structured data, which are better handled by querying a database or calling an API; when the model needs a consistent style or specialized behavior across all outputs, which may call for fine-tuning; or when the answer requires real-time actions, which call for tools and agents. Many production systems combine these approaches. Our guide to RAG versus fine-tuning covers the decision in detail, and our guide to AI chatbot development shows how RAG fits into a customer-facing assistant.

A RAG readiness checklist

You have identified the questions users need answered and the documents that contain the answers.
Those documents are reasonably accurate, current, and owned by someone who will keep them up to date.
You know who may see which documents, and that information can be attached as metadata.
You have collected at least 50 real questions with reference answers for evaluation.
You have decided what the assistant should do when it cannot answer, such as routing to a person.
You have a plan for logging, monitoring, and reviewing answers after launch.

If several of these items are missing, spend time on them before building. In most RAG projects, content quality and evaluation matter more than the choice of model or vector database. When you are ready to compare options, see our vector database comparison and our guide to choosing an embedding model.

Example rollout plan

A practical rollout starts with a single department and a single document source, such as the IT help desk knowledge base. The first release is offered to a small pilot group, who rate answers and flag errors. The team reviews low-rated answers each week, improving documents, chunking, and prompts. Once accuracy meets the agreed threshold on the evaluation set and pilot users trust the answers, the assistant is opened to the whole company, and additional sources such as HR and finance policies are added one at a time, each with its own permission rules and evaluation questions. Expanding gradually keeps quality high and makes problems easy to trace to a specific source.

This staged approach also builds the organizational trust that AI assistants need: users see the system improve in response to their feedback.

Build RAG with Vertex Cyber Tech

RAG systems for document processing and customer support are a core part of our AI/ML services. Ready to build? Follow our production RAG guide or book a RAG consultation.

Retrieval-Augmented Generation (RAG) Explained: Architecture, Components, and Best Practices