- What RAG actually does
- Why documents are split up
- How meaning-based search works
- Why errors drop but remain
- What to tidy before you start
The short answer
RAG is a way of finding the relevant parts of your own documents first, then passing them to the AI together with the question. The AI does not memorise your documents. Each time, it is told "read this passage and answer from it", with the passage chosen on the spot.
RAG stands for Retrieval-Augmented Generation: text generation supported by search. It makes it much easier to get grounded answers about internal rules and procedures the AI would otherwise know nothing about. But the quality of the answer is set by the quality of the documents you feed in. Put in outdated or contradictory material and that is what comes back out.
Why RAG is needed
The large language model at the heart of generative AI answers from general text it saw during training. It has never seen your employee handbook, your expense procedure or the arrangements you have with particular clients. Asked about them, it may still produce a plausible-sounding answer. That is a hallucination.
Pasting every internal document into the chat is not a real solution either. There is a limit to how much the model can read at once (the context window), and sending a several-hundred-page policy manual with every question is impractical. The alternative is to find just the relevant part and send that. That is RAG.
How RAG works
There is a preparation stage and a question stage.
Preparation: split documents up and make them searchable
- Gather the material: policies, manuals, FAQs.
- Split it into small pieces of a few paragraphs each. These pieces are called chunks.
- For each chunk, create an embedding — its meaning converted into a list of numbers — and store it.
Chunking keeps the unit of retrieval small. If a hundred-page manual is one block, a question about the deadline for requesting leave means sending all hundred pages. Split by heading or paragraph, and only the few relevant paragraphs need to go.
When a question arrives: find, pass, answer
- Convert the question into an embedding the same way.
- Search the stored chunks for the ones closest in meaning. This is the retrieval step.
- Send the chunks and the question to the LLM with an instruction to answer from that material.
- The LLM writes the answer. It can also be set up to show which documents it used.
Embeddings and vectors: searching by meaning
Ordinary search looks for documents containing the same characters you typed. Search for "PTO" and a policy that only says "annual leave" may not turn up.
An embedding turns the meaning of a passage into a list of numbers — a vector. Passages with similar meaning end up with similar vectors. So "When do I need to request time off?" and "Annual leave should normally be notified a set number of days in advance" can be matched even though they share almost no words.
A system built to store large numbers of vectors and find the closest ones quickly is called a vector database. In most RAG setups, that is what handles the search.
Meaning-based search has limits. For things that must match exactly, such as employee numbers or product codes, combining it with ordinary text search often works better.
Hallucinations drop, but do not disappear
Because the AI answers from the material it has been given, answers invented from nothing become less common. If the system shows which document and which passage it used, checking the answer also becomes easier.
Errors still happen. The usual causes are these.
- Wrong retrieval: an unrelated chunk is picked and the relevant one is missed.
- Stale documents: both the old and revised versions of a policy are in the collection, and the old one gets used.
- Misreading: the right chunk is passed, but the AI gets a condition or exception wrong.
- Nothing to find: there is no relevant document, and the AI fills the gap with generalities.
Do not assume that "we gave it our documents, so the answers are right". For anything touching pay, contracts or legal matters, run the system on the basis that a person opens the cited source and checks it.
What this looks like in practice
Picture a company where someone in the admin team answers staff questions all day: "Does this count as leave?", "How do I claim travel expenses?" Make the policies and manuals the RAG source, and staff ask the AI first; it answers while pointing to the relevant paragraph. What reaches the admin team is only what the AI could not answer, or what needs a judgement call.
What decides success is less the AI and more how the documents are organised.
- Make it obvious which version is current. Remove older versions or label them clearly as outdated.
- Use headings, and keep one topic per paragraph so that chunks do not cut meaning in half.
- Instruct the AI to say it does not know when the documents do not cover the question.
- Log the questions it could not answer, and add to the documents over time.
If files are scattered and inconsistently named, adding RAG mostly adds retrieval mistakes. Getting document storage in order is worth doing before RAG, not after.
Summary
- RAG finds the relevant parts of your documents and passes them to the AI to answer from.
- Documents are split into chunks and turned into embeddings (vectors) so they can be found by meaning.
- Grounded answers increase and hallucinations drop, but they do not disappear.
- Answer quality follows document quality; version control and clear headings matter most.
- For important answers, a person checks the cited source.