What Is RAG? Getting AI to Answer from Your Own Documents

Understand RAG without being a programmer: how it works, why companies use it, what to watch out for, and how to start step by step.

Ask a general-purpose AI "How many days of annual leave does our company give?" and you may get a very confident answer that cannot possibly be right, because the AI has never seen your company's employee handbook. That gap is why so many organisations have turned to a technique called RAG, the foundation of almost every company chatbot you see today.

What is RAG?

RAG stands for Retrieval-Augmented Generation: generating answers with the help of information retrieval. The idea comes from the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by the Facebook AI team (now Meta AI), published at NeurIPS 2020.

The principle: instead of letting the AI answer from the memory baked into the model (the paper calls this parametric memory), you have it look up the relevant documents first and then write an answer based on them. Think of a new employee who never answers from gut feeling but opens the handbook every time.

How RAG works: 3 main steps

1. Retrieve: find the information

When a user types a question, the system searches a prepared document library, such as the employee handbook, policies or an FAQ, and pulls out a few of the most relevant passages. Most systems use semantic search, which matches on meaning rather than exact words.

2. Augment: add context to the prompt

The retrieved text is attached to the question, together with an instruction along the lines of "Answer using only the information below. If it is not there, say you don't know."

3. Generate: write the answer

The LLM reads both the question and the supporting passages and writes a natural-language answer. A good system also cites its source, for example "Based on the employee handbook, section 4".

Why not just paste in every document?

A common question: newer models accept very long inputs, so why not drop in the whole handbook? The answer is cost and accuracy. The Lost in the Middle study (Liu et al., 2023) found that language models perform best when the key information sits at the beginning or end of a long input, and miss it more often when it is in the middle. Selecting only the relevant passages usually works better.

Why RAG matters to organisations

• It uses internal knowledge: the AI answers from your documents, not from general information on the internet.
• It is easy to update: when a policy changes, you change the document in the library. No model retraining needed.
• It is verifiable: you know which document an answer came from, so readers can click through and check.
• It reduces guessing: with real information in front of it, the AI is less likely to make things up, although the risk does not disappear.

The last two points match the problems the original paper named: providing provenance for answers and updating a model's world knowledge were still open problems for large language models.

An example from a Thai workplace

A retail company has branches across the country, and store staff keep calling HR with the same questions about medical reimbursement, shift swaps and maternity leave. The HR team gathers the employee handbook, company announcements and FAQ into one document library and builds a RAG chatbot in the company's internal chat channel.

When an employee asks "Can I claim for glasses?", the system finds the relevant benefits section and answers, citing which announcement it used. If the handbook does not cover the topic, a well-designed system replies "No information found in the handbook, please contact HR" instead of guessing.

Watch out for

• Garbage in, garbage out: if documents in the library are outdated or contradict each other, the AI will be confidently wrong.
• Not found means no answer: RAG quality depends mostly on the retrieval step. If it fetches the wrong passage, the answer will be wrong too.
• Hallucination still exists: the LLM can misread a document or add details that are not there, so important matters still need human review.
• Access rights: some documents, like salary data, should not be searchable by everyone. Design permissions from day one and keep Thailand's PDPA in mind.

RAG doesn't make AI smarter. It makes AI "open the book before answering".

How to get started

1. Pick a small problem with frequently repeated questions, such as an HR FAQ or IT helpdesk.
2. Collect the latest versions of your documents and remove older, conflicting ones.
3. Try an AI tool that lets you upload files and ask questions, to understand how RAG behaves before investing in your own system.
4. Prepare 20 to 30 test questions with correct answers so you can measure results before going live.
5. Open it to a small group first, collect the questions it gets wrong, and improve both the documents and the search.

To see where RAG fits in the bigger picture of an AI system, read Understanding AI Agent architecture next. And if you want to build your AI skills for work, browse curated courses at Plants.work Learn.