What Is RAG (Retrieval-Augmented Generation)? How It Works and When to Use It

You ask an AI assistant about your company’s return policy, and it confidently gives you an answer that sounds right but is completely made up. The model was never trained on your policy, so it filled the gap with a plausible guess.

Retrieval-augmented generation, or RAG, is the most common fix for this problem. Instead of relying only on what a large language model (LLM) memorized during training, a RAG system first searches your own content, then hands the most relevant passages to the model and asks it to answer from them. This guide explains how that works, when RAG is the right tool, when fine-tuning makes more sense, and the mistakes that cause RAG projects to disappoint.

What RAG means in plain English

Think of an LLM as a very well-read new employee with no access to your company files. It can write, summarize and reason well, but it does not know your prices, contracts or internal procedures.

RAG gives that employee a search tool and a rule: before answering, look up the relevant documents and base your answer on them.

The term comes from a 2020 research paper by Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” The authors combined a language model with a searchable index of Wikipedia and reported more specific and factual output than the model alone. Today the idea has spread far beyond that original setup, and “RAG” now describes almost any system that retrieves information and feeds it into a model’s prompt.

The key point: RAG does not change the model itself. It changes what the model sees at the moment it answers.

How RAG works, step by step

Most RAG systems follow the same 2 phases: preparing your content ahead of time, and retrieving it at question time.

Phase 1: prepare your knowledge

1. Collect sources. Gather the content you want the system to use, such as help-center articles, PDFs, policy documents, product data or database records.
2. Split into chunks. Long documents are broken into smaller passages, often a few paragraphs each, so the system can retrieve just the relevant part.
3. Create embeddings. An embedding model converts each chunk into a list of numbers that represents its meaning. Passages with similar meanings end up with similar numbers.
4. Store in an index. The embeddings, the original text and useful metadata (title, date, department, access level) go into a vector database or a search index that supports vector search.

Phase 2: answer a question

1. Embed the question. The user’s question is converted into an embedding using the same model.
2. Retrieve. The system finds the chunks whose meaning is closest to the question. Many systems also run a traditional keyword search and combine the results, which helps with exact terms such as product codes or names.
3. Rerank and filter. An optional second step reorders results by relevance and removes anything the user is not allowed to see.
4. Augment the prompt. The top passages are inserted into the prompt along with instructions such as “answer only from the sources below and cite them.”
5. Generate. The LLM writes an answer grounded in those passages, ideally with citations back to the original documents.

When your documents change, you re-run the preparation phase for the changed content. No model retraining is needed. AWS describes this refresh step as keeping documents and their embeddings updated through real-time processes or periodic batches.

Why businesses use RAG

RAG has become a default pattern for business AI projects for 4 practical reasons:

• Current information. The model can answer from documents updated this morning, not from whatever existed when it was trained.
• Company-specific answers. Internal policies, product catalogs and customer histories are not in any public model. RAG makes them available without sharing them for training.
• Citations and trust. Because answers come from specific passages, the system can show sources so people can verify them.
• Control. You decide which sources are included, and you can restrict retrieval based on each user’s permissions.

Common use cases include internal knowledge assistants, customer support bots that answer from help articles, sales tools that pull product specs, and research assistants that search contracts or reports. If you are just getting started with AI tools in a smaller company, our generative AI playbook for small business covers simpler first steps.

RAG vs fine-tuning: which one do you need?

Fine-tuning means further training a model on your own examples so it changes how it behaves. RAG and fine-tuning solve different problems, and many teams confuse them.

A simple rule of thumb:

• Use RAG when the problem is “the model does not know our information.”
• Use fine-tuning when the problem is “the model knows enough, but does not respond the way we need,” for example a strict output format, a domain-specific writing style or a narrow classification task.
• Use both when you need consistent behavior and access to changing knowledge.

Fine-tuning also matters for smaller models. A compact model tuned for a narrow job can be cheaper to run and can even run on a laptop or phone. Our guide to small language models and on-device AI explains that trend, and RAG works well alongside small models because the retrieved context carries most of the facts.

Key takeaway: RAG gives a model access to knowledge; fine-tuning changes its behavior. Start with RAG for most business question-answering projects, and add fine-tuning only when you have a clear behavior problem that better prompts cannot fix.

Common RAG pitfalls and how to avoid them

RAG is simple to demo and harder to get right in production. These are the issues that most often cause disappointing results.

Poor retrieval

If the right passage is not retrieved, the model cannot use it. Causes include chunks that are too large or too small, scanned PDFs with bad text extraction, and pure vector search missing exact terms. Fixes: test several chunk sizes, clean your source documents, combine keyword and vector search, and add a reranking step.

Answers that still go beyond the sources

Even with good context, models can add details that are not in the retrieved text. The US National Institute of Standards and Technology (NIST) calls this risk “confabulation” in its Generative AI Profile (NIST AI 600-1). Instruct the model to answer only from the provided sources, require citations, and tell it to say “I don’t know” when the sources do not cover the question.

Stale or conflicting content

If your index holds 3 versions of the same policy, the model may quote the old one. Assign document owners, remove outdated files, and store dates in metadata so newer content can be preferred.

Security gaps

A RAG system can leak data if it retrieves documents a user should not see. It can also be manipulated: a document containing hidden instructions can try to hijack the model, a form of prompt injection. The OWASP Top 10 for LLM Applications (2025) lists Prompt Injection as LLM01 and Vector and Embedding Weaknesses as LLM08. Enforce permissions at retrieval time, sanitize ingested content and treat retrieved text as data, not instructions. The same zero trust principles you apply to people apply to AI systems.

No way to measure quality

Teams often launch without a test set. Build a list of real questions with known correct answers and source documents, then measure how often the right passage is retrieved and how often the final answer is correct. Re-run the test every time you change chunking, models or prompts.

RAG and AI agents

RAG is also a building block for more capable systems. An AI agent may decide on its own when to search, which source to query, and whether to search again if the first results are weak. This is sometimes called agentic RAG. Our complete guide to AI agents covers how agents plan and use tools, and our comparison of AI agents vs chatbots vs copilots explains where RAG fits in each.

Where to start with RAG

1. Pick 1 narrow use case. For example, “answer employee questions about our HR handbook” or “answer customer questions from our help center.” Narrow scope makes quality easier to measure.
2. Clean the source content. Remove duplicates and outdated versions, and make sure documents are text-searchable.
3. Try a managed option first. Major cloud platforms and many AI products offer built-in RAG features, often called knowledge bases or grounding. They let you test value before building a custom pipeline.
4. Write 30 to 50 test questions. Include easy, hard and out-of-scope questions with expected answers.
5. Set permissions and logging. Decide who can see which documents, and log questions and retrieved sources for review.
6. Launch to a small group. Collect feedback, fix retrieval gaps, then expand.

Unfamiliar with terms such as embeddings, vector database or tokens? Our glossary defines them.

The bottom line

RAG is the practical way to make an AI model useful with your own information. It retrieves relevant passages at question time, keeps knowledge fresh without retraining, and supports citations and access control. Its weak points are retrieval quality, content hygiene and security, all of which you can manage with clean data, testing and permission-aware design. For most businesses, it is the first technique to reach for before considering fine-tuning.

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., arXiv) — origin of the RAG term and approach
• AWS: What is Retrieval-Augmented Generation? — RAG steps, data refresh and benefits
• NIST AI 600-1, Generative AI Profile — definition of confabulation risk
• OWASP Top 10 for LLM Applications 2025 — prompt injection and vector and embedding weaknesses