Retrieval-augmented generation in plain English: when a chatbot needs your documents

A general chatbot knows a lot about the world and nothing about your returns policy. Ask it how long a customer has to send back a damaged order from your shop and it will either decline to answer or guess. Retrieval-augmented generation, usually shortened to RAG, is the standard way to close that gap. Before the model answers, the system looks up the relevant parts of your own documents and hands them over with the question.

This piece explains how it works in plain terms, when it is the right tool, where it goes wrong, and how to check whether yours is any good.

Where the idea comes from

The term comes from a 2020 paper by Patrick Lewis and eleven co-authors, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Their starting point was a known weakness. Language models store facts in their parameters, but they struggle to access and manipulate that knowledge precisely, it is hard to update, and it is hard to trace an answer back to a source.

The paper paired a pretrained text generator with a searchable index of Wikipedia, which the authors called non-parametric memory, plus a neural retriever that fetched passages from it. On three open-domain question-answering tasks the combination set state-of-the-art results, and its generated text was more specific and factual than the generator produced alone. Business systems today follow the same shape: a search step feeds a writing step.

How it works, step by step

Most RAG systems have two halves. One runs ahead of time to prepare your documents. The other runs every time someone asks a question.

1. Chunking: your documents are split into smaller pieces. Anthropic’s engineering write-up on retrieval describes chunks of “usually no more than a few hundred tokens”.
2. Embeddings: each chunk is turned into an embedding, which OpenAI’s documentation defines as “a vector (list) of floating point numbers”. Texts about similar things end up with vectors that sit close together, and the distance between two vectors measures how related they are.
3. Storage: the vectors go into a vector database or index, next to the original text and its source link.
4. Vector search: when a question arrives, it is embedded the same way, and the system pulls back the chunks whose vectors are nearest. Many systems also run an ordinary keyword search such as BM25 at the same time, because meaning-based search can miss exact product codes and names.
5. Prompt with retrieved text: the top few chunks are pasted into the prompt with an instruction along the lines of “answer using only the material below, and say so if the answer is not there”. The model then writes its reply.

Nothing about the model itself changes. You are changing what it reads before it answers.

An example (illustration only): a help-centre bot for an online shop

Picture a small online shop selling handmade candles to customers in the UK and India. It has a returns policy, a delivery page with different times for each country, care instructions for each product and about forty FAQ answers. The owner wants a chat box that handles routine questions at 2am.

With RAG, those pages are chunked and embedded once. A customer types “my candle arrived cracked, can I get a refund?” The search step returns the damaged-items paragraph of the returns policy and the delivery FAQ about breakages. The model reads both and replies with the shop’s own deadline and the photo it asks for, rather than a generic answer about consumer rights. When the returns policy changes, the owner re-indexes that one page and the bot’s answers change with it.

This bot cannot look up a particular order unless you connect it to your order system, which is separate work with its own security questions.

RAG, fine-tuning or a long context window?

Anthropic is direct on the last point. If your knowledge base is smaller than 200,000 tokens (about 500 pages), it says you can include the entire knowledge base in the prompt instead of building retrieval. RAG earns its extra moving parts when the material is too large to send every time, changes often, or needs different access rules for different users. Fine-tuning is usually a poor way to keep facts current: every change means retraining, and the model still cannot point to where an answer came from.

Where RAG goes wrong

• Bad chunking: if one chunk holds “refunds are available within 30 days” and the next holds “except for personalised items”, the bot can retrieve half a rule. Anthropic notes that “traditional RAG solutions remove context when encoding information”, which often leads to the right material not being found. Split along headings and keep exceptions with their rules.
• Stale data: the index is a copy. If nobody re-indexes after the delivery page changes, the bot quotes last year’s prices with full confidence.
• Retrieval misses: the search can fail to find the chunk that holds the answer, often because the customer’s words differ from yours (“parcel” against “shipment”). The model then falls back on general knowledge.
• Hallucination is still possible: as Wikipedia’s article on RAG puts it, “RAG does not prevent hallucinations in LLMs.” The model can misread a correct passage or fill gaps around it.
• Conflicting sources: two versions of the same policy in the index force the model to pick one. Move old versions out of the index.

How to test it with a small set

A spreadsheet is enough to evaluate a RAG system.

1. Write 30 to 50 real questions taken from your inbox or support tickets, including a few your documents cannot answer.
2. For each one, note the correct answer and the page or section that holds it.
3. Check retrieval first: did the right chunk appear in the top results? Anthropic’s write-up measures exactly this, as the share of relevant chunks that fail to appear in the top 20.
4. Then mark each answer as correct, partly correct, wrong or “I don’t know”. For the unanswerable questions, “I don’t know” is the right result.
5. Re-run the same set after every change to chunking, prompts or documents.

If retrieval is failing, no amount of prompt rewording will fix the answers. Fix the search first.

Privacy and security points

Anything you index can end up in an answer, so choose what goes in with care.

• Index only what the bot needs. A help-centre bot has no use for staff files, supplier contracts or customer emails.
• Apply access rules in the search step. If different users may see different documents, filter by permission before the model sees anything. An instruction in the prompt is not a security control.
• Check where the data goes. Embedding and answering often run on a third-party service, so read its terms on data retention and training, and mention it in your privacy notice where personal data is involved.
• Treat retrieved text as untrusted. Wikipedia’s article reports cases of “RAG poisoning”, where sources were flooded with content designed to steer answers. If your bot indexes reviews or user posts, someone can plant text there.
• Decide how long you keep chat logs and who can read them. Customers will type names, addresses and order details into a chat box.