If you've heard "RAG" in conversations about AI and assumed it was a technical detail you didn't need to understand, this article is for you. RAG is the architecture behind almost every business AI assistant that actually answers questions accurately — and understanding it helps you know when you need it and what to ask for.
The core problem RAG solves
Every large language model is trained on a snapshot of text data up to a certain date. It doesn't know anything about:
- Your company's specific products, policies, or procedures
- Events that happened after its training cutoff
- Confidential data you haven't shared with the model provider
- Internal documents, knowledge bases, or proprietary research
When you ask a generic LLM about your return policy or your SLA terms, it has two options: admit it doesn't know, or hallucinate a plausible-sounding answer. Both are bad for production use.
RAG is the solution. Before the model generates an answer, the system retrieves relevant content from your knowledge base and adds it to the prompt. The model now has the information it needs to answer accurately.
How RAG works in three steps
Step 1 — Index: Your documents (support articles, product docs, policy PDFs, internal wikis) are split into chunks and converted to vector embeddings — numerical representations of their semantic meaning. These embeddings are stored in a vector database.
Step 2 — Retrieve: When a user asks a question, the question is also converted to an embedding. The vector database finds the chunks most semantically similar to the question — not just keyword matching, but meaning matching. The top k most relevant chunks are returned.
Step 3 — Generate: The retrieved chunks are added to the LLM's prompt as context. The model reads the context and generates an answer grounded in your actual documents. The answer cites specific content rather than relying on training data.
When RAG works well
RAG delivers high accuracy when:
- The question can be answered from a specific passage in your knowledge base
- Your knowledge base is well-structured and comprehensively covers the topic area
- Chunks are the right size — small enough to be specific, large enough to contain context
- The retrieval step surfaces the genuinely most relevant content, not just the most keyword-similar
When RAG underperforms
RAG struggles with:
- Synthesis questions: "What is the overall risk profile of our product portfolio?" requires synthesising across many documents, not retrieving one passage. RAG retrieves well but synthesises imperfectly.
- Questions not covered by the knowledge base: If the user asks about something not in your documents, RAG retrieves the least-wrong content, which may mislead the model into generating a wrong answer confidently.
- Poorly structured source material: Scanned PDFs, inconsistently formatted documents, and content without clear headers or structure chunk and retrieve poorly.
RAG vs fine-tuning: which one do you need?
| Dimension | RAG | Fine-tuning |
|---|---|---|
| Knowledge update frequency | Update documents, re-embed; instant | Retrain model; expensive and slow |
| Use case | Knowledge-intensive Q&A, document search | Style, tone, task-specific behaviour |
| Data privacy | Data stays in your infrastructure | Data shared with model provider for training |
| Cost to build | £3,000–£15,000 | £10,000–£80,000+ |
| Hallucination risk | Lower for in-scope questions | Similar to base model |
Most business use cases that need "AI trained on our data" are actually RAG use cases, not fine-tuning use cases. If you want your AI to know things from your documents, use RAG. If you want your AI to behave differently (follow specific instructions, match a tone, do a specific task format), fine-tuning is the tool.
For the broader context on AI agents (which often use RAG as their knowledge layer), see AI Agents for Business Automation in 2026 and AI Agent Guardrails.