Evaluating RAG and Reducing Hallucination

A chatbot that "seems fine" can be quietly wrong. Learn how to evaluate RAG piece by piece, the key metrics, and practical ways to reduce hallucination.

Two weeks after launching an internal chatbot, a team lead overhears an employee say, "The bot said I could claim it, but accounting rejected it." Nobody knows how many times the bot has been wrong, because at launch the team only asked it a few questions and felt "it seems fine". This is where many RAG projects go wrong: there is no systematic way to measure them.

What is hallucination in RAG?

Hallucination is when AI produces an answer that sounds convincing but is untrue or unsupported. RAG reduces the problem because the AI reads documents before answering, but it does not eliminate it. Wrong answers in a RAG system usually come from one of two places:

• Wrong retrieval: the system fetches irrelevant documents, or misses the one with the answer.
• Wrong generation: the right documents were retrieved, but the AI misreads them, over-generalises, or adds details the documents do not say.

Separating these two is the key, because the fixes are completely different.

The key metrics to know

An open-source tool called Ragas, widely used to evaluate RAG systems, groups the core metrics like this.

Retrieval side

• Context Precision: how useful the retrieved documents really are for answering, or whether a lot of noise came along.
• Context Recall: whether the retrieved documents contain enough information to answer correctly.

Generation side

• Faithfulness: whether the statements in the answer are supported by the retrieved documents. This directly catches hallucination.
• Answer Relevancy: whether the answer addresses what the user asked, rather than being correct but off-topic.

A point from the Ragas documentation: Faithfulness and Answer Relevancy can be measured without a reference answer, but Context Precision and Context Recall need a correct answer prepared in advance. That is why a test question set with answers matters so much.

How to build a test set (golden set)

A good test set should reflect real questions, not questions the development team invents.

• Pull real questions from emails, chats or tickets previously sent to HR or IT.
• Write the correct answer and note which document and section it comes from.
• Include questions that have no answer in the documents, to see whether the system is willing to say "I don't know".
• Include ambiguous questions and questions that need information combined from several documents.

Example: a logistics company

A transport company built RAG so drivers could ask about standard operating procedures (SOPs). The team prepared 40 real questions from the drivers' group chat. The first round showed that most wrong answers came from retrieval, because drivers use casual language like "truck broke down on the road, what do I do?" while the document says "Procedure in case of vehicle malfunction during transport". The team improved search instead of changing the model, then re-ran the same test set to compare.

Ways to reduce hallucination that work

1. Tell it to admit when it doesn't know: state clearly in the prompt that if the documents lack the information, it should say so.
2. Require citations: have the AI name the document and section it used every time. Users can verify, and the team can see where things went wrong.
3. Fix retrieval first: if the right document is not found, no model, however capable, can answer correctly.
4. Clean out conflicting documents: old and new policies sitting side by side in the library are a common cause of wrong answers.
5. Have humans check important matters: questions about money, law or health should have a step where an expert confirms.

Watch out for

• Many metrics use an LLM as the judge (LLM-as-a-judge), which can also be wrong. Have people spot-check regularly.
• High scores on a test set do not guarantee good real-world use. Keep adding real post-launch questions to the test set.
• Re-evaluate every time you change documents, models or settings, because fixing one thing can make another worse.

You can't improve what you don't measure, and you shouldn't trust a chatbot you haven't measured.

What to do next

1. Collect 30 to 50 real questions with answers and source documents.
2. Test retrieval and generation separately to find where the problem is.
3. Fix one thing at a time, then re-measure with the same set.
4. Give users a button to flag wrong answers, and add those to the test set.

If your problem is on the retrieval side, Chunking and Embedding will help a lot. And to plan your own learning in AI skills, try the Plants.work AI Career Assistant.