RAG in Plain English: The Library, the Librarian, and the Answer
An AI model can write a convincing answer and still get your company's facts completely wrong. Retrieval-Augmented Generation (RAG) helps solve this problem by giving the model relevant information before it answers.
Think of RAG as a library, a librarian, and someone who explains what the books say. Once you understand those three roles, the technology behind modern AI assistants becomes much easier to follow.
1. Imagine Asking a Librarian a Question
Imagine walking into a library and asking, “What is the company's refund policy?” The librarian does not guess based on general knowledge; they search the library, find the relevant policy document, open the correct section, and use that information to answer you.
RAG works in a similar way. Instead of relying only on what a language model learned during training, an application finds relevant information from an external source and provides it to the model when answering a question.
The library represents your knowledge base, which might contain PDFs, Word documents, website pages, product manuals, or database records. The librarian represents the retrieval system, and the person explaining the policy represents the language model generating the final response.
This approach is useful when information changes frequently or belongs to a specific organization. A customer support assistant, for example, can retrieve the latest troubleshooting guide rather than depend on general knowledge that may be incomplete or outdated.
2. How the Library Becomes a Real RAG System
Before a librarian can find the right book quickly, the library needs to be organized. RAG applications do something similar through several technical components.
Chunking: Dividing books into useful sections. A 100-page employee handbook is too large to send with every question. Chunking divides it into smaller passages, such as individual policy sections or groups of paragraphs. A starting configuration might use chunks of 500 tokens with 75 tokens of overlap, although the best size depends on the documents and questions.
Embeddings: Giving passages searchable representations. An embedding model converts text into a list of numbers that captures aspects of its meaning. This helps a system recognize that “How do I get my money back?” may be related to a passage titled “Refund and reimbursement policy,” even though the wording differs.
Vector store: Organizing the searchable representations. A vector store holds embeddings alongside the original text or references to it and useful metadata. Options include PostgreSQL with pgvector, Pinecone, and Weaviate. Think of this as a library catalog designed to find passages by meaning, not just exact words.
Retriever: Finding the most relevant passages. When a question arrives, the retriever searches for useful chunks and returns a selected number of results. That number is often called top-k; for example, top-k = 4 asks the retriever to return four candidate passages.
Generation: Writing the final answer. The application sends the question, retrieved passages, and instructions to a language model such as GPT or Claude. The model then uses that supplied context to produce a response, ideally with references to the supporting sources.
3. Follow One Question Through the Entire Pipeline
Suppose an employee asks an internal AI assistant: “How many vacation days do new employees receive?” The company's handbook contains the answer, but the language model should not be expected to know that private policy in advance.
First, the application processes the handbook and divides it into chunks. It generates embeddings for those chunks and stores them in PostgreSQL with pgvector, together with document names and section details.
Next, the employee's question is converted into an embedding. The retriever searches for relevant passages and returns four candidates, including the section that explains vacation eligibility. The application checks that the employee is permitted to access the information.
Finally, the application sends the retrieved policy text and question to the language model. If the policy says eligible new employees receive 15 vacation days annually, the model can answer with that figure and identify the handbook section supporting it, provided the policy and employee's eligibility match.
The important point is that the model does not perform every step itself. The application handles document processing, search, access controls, and context assembly; the model generates the answer from the information it receives.
4. Where RAG Usually Breaks in Practice
Bad chunking. If the vacation entitlement appears in one chunk and its eligibility conditions appear in another, the retriever may return only half the information. Overly large chunks can also include irrelevant material. Test different chunk sizes and overlap settings against real questions rather than assuming one configuration fits every document.
The wrong top-k value. Retrieving only one passage may miss a necessary exception, while retrieving 20 passages can overwhelm the context with unrelated details. Start by testing values such as 3, 5, and 8, then measure whether the correct evidence appears and whether answer quality improves.
No citations or source tracking. An answer may sound correct without showing where its facts came from. Preserve document titles, URLs, section identifiers, or page numbers so the application can display supporting sources. Citations make verification easier, although they do not guarantee that the answer accurately represents those sources.
Other problems include outdated documents, weak search results, missing access controls, and questions that have no answer in the knowledge base. A reliable system should recognize missing evidence and ask for clarification or say it cannot verify the answer instead of inventing a fact.
5. Build a Small RAG Project Yourself
If you already know SQL or data engineering, start with a collection of 20–50 documents, such as product guides or public policy documents. Use Python, an embedding model, PostgreSQL with pgvector, and a model API to build a question-answering assistant.
Create 30 realistic test questions and record the expected source for each answer. Measure retrieval quality, answer correctness, response time, and cost; then experiment with chunk size and top-k to see what changes.
Salesforce developers can use approved CRM knowledge, QA professionals can build repeatable evaluation tests, and support professionals can organize troubleshooting content. RAG is not just a model feature; it is a complete application workflow involving data, search, permissions, and generation.
Which part of RAG still feels fuzzy to you — retrieval or generation?
Join the KKRGENAI Innovations community discussion to share your questions, compare RAG projects, and learn from other professionals building practical Generative AI applications.
