GraphRAG: Using Knowledge Graphs with AI
When the question is about the big picture across all your documents, standard RAG may fall short. How knowledge graphs and GraphRAG work, when to use them, and the costs.
A customer service manager has thousands of call notes and wants to ask AI, "What did customers complain about most this year, and which products is each issue linked to?" Standard RAG handles this poorly, because it is good at finding a few passages that match a question, not at summarising the big picture across an entire library. This is the problem GraphRAG was designed to solve.
What are knowledge graphs and GraphRAG?
A knowledge graph stores knowledge as a network of "things" (entities) and "relationships". For example, "Product A" is made by "the Lamphun factory", and "Customer segment B" complains about "late delivery" of "Product A". Each thing is a node and each relationship is an edge.
GraphRAG uses a knowledge graph as part of searching and summarising information before the AI answers. A widely discussed approach comes from Microsoft, published in the paper From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024).
How Microsoft's GraphRAG works
1. Chunk documents and extract knowledge
Documents are split into chunks, and an LLM reads each one to extract entities, relationships and claims, along with short descriptions.
2. Build the graph and group it
Entities become nodes and relationships become edges. The Leiden algorithm then splits the graph into communities of related topics, nested at several levels.
3. Summarise each group
The LLM writes a summary report for each community in advance, like having an executive summary prepared for each topic.
4. Answer big-picture questions
When a question arrives, the LLM reads the community summaries to write partial answers, then combines the most helpful ones into a final answer.
What the research found
The study tested two datasets, podcast transcripts and news articles, of about 1 million tokens each, using an LLM as judge. GraphRAG gave more comprehensive and diverse answers than vector RAG for big-picture questions, while vector RAG gave more direct answers. The authors themselves note the limits: only two datasets were tested, and fabrication rates were not yet measured.
What it suits
- Big-picture questions: main themes in customer feedback, employee surveys or large sets of research reports.
- Data where relationships matter: who is connected to whom, which supplier provides materials for which product.
- Multi-hop questions: questions that require linking information across several documents.
For specific factual questions such as "How many days of annual leave do I get?", standard RAG is still the better fit: faster and cheaper.
An example from a Thai organisation
The HR team at a company with several thousand employees receives an engagement survey with a large number of open-ended comments. Instead of reading them one by one, the team tries GraphRAG to extract themes such as workload, line managers and commuting, and to see which departments or branches each theme links to. The output is a starting point for choosing which issues to follow up, not a final conclusion.
Watch out for
- High indexing cost: every chunk must go through an LLM. The paper reports that indexing the podcast dataset took 281 minutes on gpt-4-turbo.
- The graph can be wrong: the LLM may extract the wrong entities, or merge similarly named things into one. Spot-check it.
- Harder to update: frequently changing documents mean rebuilding the graph and summaries often.
- Sensitive data: a graph linking people, projects and events can reveal information that individual documents do not reveal on their own. Set permissions carefully.
Vector RAG finds the needle in the haystack. GraphRAG tells you what shape the haystack is.
How to get started
- Check whether most user questions are specific or big-picture.
- If they are specific, get standard RAG working well first.
- If they are big-picture, trial GraphRAG on a small document set and estimate the indexing cost.
- Compare both approaches' answers on real questions, with subject-matter experts as judges.
- Decide based on value versus cost, not on how new the technique is.
If you are unsure about the basics, start with What is RAG. And if you are looking for data or AI roles where you'd work on projects like this, browse jobs on Plants.work.