Chunking and Embedding: The Heart of an Accurate RAG System

Why the same RAG setup is accurate in one company and messy in another: understand document chunking and embeddings, with practical settings.

Two companies use the same AI model to build the same kind of document Q&A chatbot. One is so accurate that staff love it; the other answers so badly that nobody uses it. Most of the difference is not in the model. It is in a preparation step people tend to overlook: chunking and embedding.

What are chunking and embedding?

In a RAG system you do not hand the AI an entire manual. You first cut documents into small pieces. That cutting is called chunking, and each piece is a chunk.

Each chunk is then converted into a long list of numbers called an embedding, or vector. Think of it as a "coordinate of meaning": texts with similar meanings get coordinates close to each other. When a user asks a question, the system converts the question into a coordinate the same way and finds the nearest chunks. A popular way to measure closeness is cosine similarity.

Why chunking matters

Chunks that are too large

One chunk mixes several topics, so its meaning coordinate gets blurry. It becomes harder to find, and the AI gets more irrelevant text than it needs.

Chunks that are too small

The text loses context. A chunk that just says "limit of 3,000 baht per year" with no heading leaves the AI with no idea which benefit that limit belongs to.

The lost-context problem

Anthropic described this problem in an approach called Contextual Retrieval (September 2024): before embedding a chunk, you prepend a short note explaining where in the document it comes from. According to Maginative's report on Anthropic's tests, this reduced failed retrievals by 35%, and by 49% when combined with BM25 keyword search.

Common ways to chunk documents

• Fixed-size: cut by an equal number of characters or tokens. Simplest, but often cuts mid-sentence.
• Overlap: let neighbouring chunks share a little text so key points are not severed at the boundary.
• By document structure: cut by heading, section or paragraph. Good for manuals, regulations and contracts with clear structure.
• Add metadata: attach the document name, heading, date and department to every chunk for filtering and citation.

How to choose an embedding model

The embedding model is what turns text into coordinates. If your documents mix Thai and English, choose a multilingual model. One example is BAAI's BGE-M3, which its official documentation says supports more than 100 languages and inputs of up to 8,192 tokens. Things to weigh together:

• Does it support the languages you actually use?
• The maximum input length per call
• Whether data leaves your organisation, or the model can run on your own systems
• Results on your real questions, which matter more than general benchmark scores

A real example from a Thai office

The accounting team at a parts manufacturer built RAG on its "expense reimbursement rules". The first version used fixed-size chunks. When asked "How much can I claim for upcountry accommodation?", the system answered with the per diem figure, because the two topics sat next to each other and ended up in the same chunk.

In the second round the team cut by sub-heading and added a leading line to every chunk such as "Expense reimbursement rules, accommodation section". Answers became noticeably more accurate without changing the AI model at all.

Watch out for

• Whenever you change embedding models, you must re-embed every document, because coordinates from different models cannot be compared.
• There is no chunk size that is best for every task. Test with your own documents and questions.
• Tables, images and scanned files often lose structure when converted to text. Check them before chunking.
• Sending confidential documents to an external embedding service must pass your organisation's data security policy first.

How well AI answers depends on how well we slice the knowledge we give it.

How to get started

1. Pick the 3 to 5 most frequently asked-about documents as a pilot set.
2. Clean them first: remove repeated headers, page numbers and unnecessary text.
3. Start by chunking by heading with a small overlap, and put the heading at the front of every chunk.
4. Write test questions with correct answers and check whether the system retrieves the right chunk.
5. Change chunk size and method one variable at a time, measuring and comparing each time.

If you are not sure how RAG works overall, start with What is RAG. To keep growing your data and AI skills, browse related courses at Plants.work Learn.