What is RAG? How an AI knowledge base retrieves information before answering
Understand retrieval-augmented generation through an expense-policy example, including chunking and common retrieval methods, the difference between RAG and a vector database, and why a cited answer can still be wrong.
Understand the concept
RAG stands for retrieval-augmented generation. A system retrieves relevant external material for a question, then supplies it with the question to a model that generates an answer. It can let an assistant consult external material such as company documents at answer time, commonly without changing model parameters. RAG does not guarantee correctness: outdated sources, missing conditions, and misreading can still cause errors. Check the retrieved material and final answer separately, not just the presence of a citation.
Separate retrieval and generation with a policy example
Suppose a fictional company’s policy permits taxi reimbursement after 10 p.m. under specified conditions. When an employee asks, the system retrieves the clause and gives it to the model to formulate an answer. This example is not a real company policy. The model does not need to have encountered the company during training to answer using the retrieved text.
Retrieval supplies candidate evidence, while generation turns that evidence into a readable answer. A list of search results alone does not include this generation step. Uploading an entire short file into a prompt also does not establish that retrieval occurred. Check whether the application actually selected relevant material for the question.
Chunking and vector search solve different parts
Long documents are often split into smaller passages while retaining file and section information. Passages that are too short may separate a reimbursement rule from its conditions; overly long passages can add irrelevant text. Check chunking against the document structure and real questions instead of copying a fixed character count.
Embeddings represent text numerically to help find semantically similar material. RAG does not require every document to be stored in a vector database: keyword retrieval or a combination of keyword and vector search can also be used. For exact strings such as a policy identifier, check whether retrieval found the specific identifier.
A source link still needs version and condition checks
If a knowledge base holds both old and new policies, retrieval may select an expired clause. Even with the correct document, a model might omit a condition such as required overtime approval. More citations do not fix that error. Open the cited passage, check its effective date, and match each condition in the answer to the source.
Classify failures as shown below. Inspect the passages actually retrieved before reviewing the final answer; this locates problems more effectively than blaming every error on the model. When evidence is insufficient, return that limitation and identify the missing material rather than letting a fluent answer hide a retrieval failure.
| Observed problem | Where to check first |
|---|---|
| No relevant clause was found | Check document ingestion and retrieval for the relevant passage |
| The citation points to an old policy | Check file versions and date filters used in retrieval |
| The source is right but the answer omits a restriction | Compare the supplied context with the generated answer |
RAG can complement fine-tuning; agents also direct actions
RAG describes using retrieved material for generation; fine-tuning updates model parameters through training. They can be combined—the original RAG research included training—while a knowledge-base application can update external material without retraining the model. For frequently changing policies, checking document updates and retrieval is more direct than treating every change as a training task.
An agent can use retrieval as a tool and decide from the result whether to search again or take another action. A RAG application that only answers policy questions does not also need permission to change systems. Test it with known-answer questions and questions the documents cannot answer. This article provides concepts and a test design, not measured accuracy for a knowledge base.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.