Retrieval-augmented generation gives an AI agent something its training data cannot: access to information that lives outside the model. Instead of answering only from what it learned during training, the agent first retrieves relevant passages from external documents, then generates an answer grounded in what it retrieved. That two-step pattern — retrieve, then generate — is why retrieval-augmented generation has become a standard way to build agents that answer questions about private, changing, or specialised knowledge.
The idea is not new, but its fit with agents is what makes it matter now. A language model on its own stores knowledge in its parameters: the weights learned during training. That knowledge is fixed in its weights at training time; it may not cover a company’s particular private documents or later updates, and on its own it offers no provenance for its answers. Retrieval-augmented generation adds a second kind of memory — an explicit, searchable store of documents — and lets the agent consult it before it speaks or acts.
What retrieval-augmented generation means for an AI agent
At its simplest, retrieval-augmented generation (RAG) means the model does not answer from memory alone. It looks something up first. The original RAG formulation, introduced by Lewis and colleagues in 2020, combined a pre-trained sequence-to-sequence model — the parametric memory — with a non-parametric memory: a dense vector index of documents accessed with a neural retriever. The model generated language conditioned on the passages it retrieved, rather than only on its weights.
For an agent, that definition gains a practical twist. Retrieval can be a step the agent takes inside its loop, as one supported pattern: in some designs it is a one-off preprocessing step, in others the agent calls a retrieval tool when it needs facts. In an agent-decided design, the agent receives a goal or a question, may decide it lacks the facts it needs, calls a retrieval tool, reads what comes back, and only then generates an answer or takes an action. Other designs use single-pass retrieval, where retrieval happens once before generation. The retrieved passages can also be quoted or cited, which gives the user something a bare model answer cannot: a trail back to the source.
Agents need this for three reasons. First, model knowledge is fixed in its weights: anything published, changed, or decided after training is not in those weights. Second, particular private documents — internal documentation, product manuals, and support policies — may not be part of a given model’s training data, and even when similar material was trained on, it is not the current, citable source. Third, facts change, and updating a document in an index is far cheaper and faster than retraining a model. The core loop is therefore: form a query, retrieve relevant passages, add them to the prompt, generate, and cite or act on the result.
How a RAG pipeline works step by step
A RAG pipeline has three stages, and each one matters. The first stage happens before any question is asked. Documents are collected, cleaned, and split into smaller pieces called chunks. Each chunk is converted into a vector embedding — a numerical representation of its meaning — and those embeddings are stored in a vector database or search index, ready to be searched.
The second stage begins when a question arrives. The agent’s query is itself converted into an embedding, and the retriever compares that query vector with the stored chunk vectors, returning the chunks whose meaning is closest to the question. Vector search matches by semantic similarity rather than by exact keyword match, so a chunk can be retrieved even when it uses different words for the same concept.
The third stage is generation. The retrieved chunks are assembled into a prompt together with the user’s question and instructions — for example, to answer only from the provided sources and to say when the evidence is missing. The language model then writes an answer grounded in those specific documents. In the original RAG research, this grounding produced language that was more specific, diverse, and factual than a parametric-only baseline on generation tasks.
Consider an illustrative documentation assistant. A user asks an agent how to configure a particular feature. The agent does not guess from general knowledge. It searches the product documentation, retrieves the two or three most relevant sections, and answers from those sections, quoting the relevant passage. If the documentation changes next week, the index is updated and the agent’s answers change with it — no model retraining required. This is an illustration of the pattern, not a report of any particular deployment.
The pieces an agent’s RAG system is built from
Chunking comes first, and it is a genuine design decision rather than a detail. A chunk is usually a paragraph or a few paragraphs of text. Practitioner guides commonly report chunk sizes in the range of roughly 200 to 500 words, or 300 to 800 tokens, depending on the guide — there is no single standard size. Chunks that are too large dilute relevance; chunks that are too small lose the context needed to answer. Overlapping chunk boundaries slightly is a common way to avoid splitting an answer across two chunks.
Embeddings are the second piece. An embedding model turns each chunk into a vector of numbers that captures its meaning, and turns each query into a vector in the same space. Similarity between vectors then approximates similarity of meaning. The vector store is where those embeddings live: it must return the nearest neighbours of a query vector at the scale the application requires, and it usually stores metadata alongside each chunk — the source document, section, and date — so retrieval can be filtered as well as ranked.
Retrievers and rerankers form the third piece. A retriever’s job is recall: find a broad set of plausibly relevant chunks. A reranker then re-scores that candidate set with a stronger relevance model, such as a cross-encoder that reads the query and each candidate passage together, and keeps only the most answer-bearing passages for the prompt. Reranking pipelines commonly retrieve a broader set first — for example, the top 20 to 50 candidates — and then narrow to a small top-k for generation.
Finally, there is the agent’s tool layer. In agent frameworks, retrieval is exposed as a tool the agent can call, just like a web search tool or a database query tool. That is one pattern that makes RAG agentic: the agent may choose when to retrieve, what to retrieve, and whether the results are good enough to answer or whether another query is needed — alongside single-pass retrieval, where retrieval happens once before generation.
# Simplified illustration of the agent loop, not a complete program:
# 1. The agent forms a query from the user's question.
# 2. The retriever returns the most relevant document chunks.
# 3. The retrieved chunks are added to the prompt.
# 4. The model generates an answer grounded in those chunks.
query = agent.form_query(user_question)
retrieved_chunks = retriever.search(query)
prompt = build_prompt(user_question, retrieved_chunks)
answer = model.generate(prompt)
RAG compared with fine-tuning, long context and agent memory
Fine-tuning is one alternative. It bakes knowledge into the model’s weights by training on domain data. That can improve style and behaviour, but it is slow and expensive to update, and fine-tuning alone does not provide a reliable source trail: knowledge in weights carries no citation to a document. Changing knowledge encoded in weights requires further training; a fine-tuned model can still receive retrieved or prompt-supplied sources at inference time, which is why fine-tuning and retrieval are often combined.
Long context windows are another alternative: put the whole document set into the prompt and let the model read it. That works for small corpora, but it does not scale to large documentation sets, it costs more per query as context grows, and it still leaves the model to find the relevant passage itself among a great deal of irrelevant text. Retrieval solves the selection problem before generation begins, so the model reads a small, relevant set of passages rather than everything.
Agent memory is a third, complementary piece. Memory stores past interactions, user preferences, and the state of an ongoing task — what happened before. RAG supplies external knowledge — what the documents say. An agent that remembers a user’s project but cannot look up the current documentation is half-built, and an agent that can retrieve documents but remembers nothing about the conversation will repeat itself. Memory and retrieval can serve complementary needs: memory for continuity, retrieval for facts.
Where RAG fits inside the agent loop
For an agent, the interesting question is not only how retrieval works, but when the agent decides to retrieve. A well-built agent retrieves when it faces uncertainty: when the question concerns facts it does not reliably know, when the answer must come from a specific source, or when the user’s intent triggers a tool — for example, a request about a product policy that must be answered from the policy itself. When the model already knows the answer reliably, retrieving may add latency and cost without improving the answer — though retrieval can still be worthwhile when the answer must be verified against a source or cited.
Retrieval is often multi-step. A first retrieval may return partial information that suggests a better query, or reveal that the question needs two different documents. Agentic RAG patterns therefore include single-pass retrieval for straightforward FAQ-style questions, multi-hop retrieval that iterates to answer complex questions, and fully agent-decided retrieval where the model chooses when and how to retrieve dynamically as its plan unfolds. The agent must also know when to stop: each additional retrieval round costs time and tokens, and may return diminishing value, though multi-hop questions can genuinely need further rounds.
Grounding actions are what make RAG answers trustworthy. The agent can quote the retrieved passage, cite its source document, and flag low-confidence answers when the retrieved evidence is thin or contradictory. That honesty is a feature, not a weakness: an agent that says the documentation does not cover a question is more useful than one that invents a confident answer.
Limits, failure modes and common mistakes
Retrieval misses are the first failure mode. The right passage may exist in the index but not be retrieved, because the query used different words, because the chunking split the answer across a boundary, or because a more popular but less relevant passage scored higher. When retrieval misses, the model answers from a partial picture — or from its own parametric knowledge, presented as if it came from the documents.
Garbage in is the second. Stale, duplicated, or poorly chunked sources degrade every answer downstream. If the index contains three conflicting versions of a policy, the agent may retrieve the oldest one. If chunks are cut mid-sentence, the model reads fragments. Poor source quality can degrade answers regardless of model quality.
Hallucination is reduced, not eliminated. Grounding an answer in retrieved text lowers the chance of invention, and the original RAG research found more factual generation than a parametric-only baseline — but the model can still misread a retrieved passage, combine two passages incorrectly, or fill gaps between passages with plausible invention. RAG is a mitigation for hallucination, not a cure, and any article or product that promises otherwise is overselling it.
Over-retrieval is a subtler mistake. Stuffing too many passages into the prompt bloats cost, slows responses, and can actually hurt accuracy: the model must weigh more evidence, some of it weak or contradictory, and may be distracted from the strongest passage. Precision — retrieving fewer, better passages, with reranking where it helps — can improve results over broad recall alone, depending on the task.
When to use RAG for an AI agent, and when not to
Use retrieval-augmented generation when answers must be current, private, or verifiable against sources. Internal documentation, product knowledge, legal or policy questions, and any domain where the facts change faster than models are retrained are natural fits. If users need to check the answer against a source, RAG’s citation trail is itself a reason to choose it.
Skip it when the model already knows the answer reliably and the cost of retrieval outweighs the benefit, or when latency matters more than freshness — for example, in fast, conversational turns where a general answer is enough. Retrieval also adds infrastructure: an index to maintain, embeddings to compute, and a pipeline to monitor. For a prototype answering general questions, that machinery may be unnecessary.
A short checklist closes the decision. Is the source material high quality and well maintained? How often does it change? Do answers need citations? Is there an evaluation plan — a set of test questions with known-good answers — to check that retrieval actually improves results? If those answers point to current, private, verifiable knowledge, retrieval-augmented generation is a strong candidate — provided retrieval demonstrably improves results on test questions. If they do not, a simpler approach may serve better.
