Using AI with Long Documents: Long Context Techniques That Work
Why AI slips on long documents, the Lost in the Middle effect, and how to arrange documents, questions and quotes, based on Anthropic guidance.
You upload a 120-page tender TOR to an AI and ask, "Is there a penalty clause for late delivery?" The AI confidently says no, even though the clause is on page 67. Newer models accept longer and longer inputs, but "can accept" does not mean "uses every part well". This article covers techniques that help AI read long documents more accurately.
What is long context?
Long context means sending a large amount of data, such as contracts, annual reports or several documents, into a model's context window in one go instead of cutting it into small pieces. Anthropic's prompting documentation gives specific advice for prompts with roughly 20,000 tokens or more of data.
Why AI slips on long documents
Lost in the Middle
The research paper "Lost in the Middle: How Language Models Use Long Contexts" by Nelson F. Liu and colleagues (2023) found that models perform best when the key information is at the start or end of the input, and that performance degrades significantly when it sits in the middle, even for models designed for long contexts.
Context rot
Anthropic's context engineering article calls the decline in a model's ability to recall information as context grows "context rot", which happens across all models to different degrees (more in Context Engineering).
Four techniques from Anthropic's guide
1. Documents at the top, question at the end
Anthropic recommends placing long documents near the top of the prompt, above instructions and examples, and states that putting the query at the end "can improve response quality by up to 30 percent in tests", especially with complex, multi-document inputs.
2. Wrap each document in tags
Put each document in <document>, with <source> for where it came from and <document_content> for the content, all inside <documents>. Add metadata such as a date or version number, so the AI refers to the right document.
3. Extract quotes before answering
Anthropic recommends asking the AI to quote the relevant parts of the documents first, then work from those quotes. This keeps the model focused on what matters and helps it ignore the rest.
4. Send only what is needed
If you know which chapter the question concerns, send only that chapter. The "fewer tokens, higher signal" principle still applies even when the model can take more.
Example: comparing annual reports from 3 companies
<documents> <document><source>Company A annual report 2025</source><document_content>...</document_content></document> (Companies B and C in the same format) </documents> First, quote the passages about supply chain risk from each report into <quotes>, noting the source. Then compare how each company handles it differently. If a report does not cover this, say so plainly.
Notice the question comes last, tags separate each report, quotes are required first, and there is a way out when information is missing.
When not to stuff everything in
- When documents exceed the context window or grow daily, use a retrieval system to pull only the relevant parts.
- For summarising very long documents, summarise chapter by chapter and combine the results with Prompt Chaining.
- If you ask the same questions of the same documents often, consider cost, because every token you send is billed.
How long a document you can fit matters less than how you arrange it.
Things to watch out for
- Check every quote against the original, because AI can misquote or invent text.
- For confidential company or customer documents, check the AI service's data policy before uploading.
- A "not found" answer on a long document may mean the model missed it, not that it is absent. Double-check important points.
How to start
- Pick a long document you read often, such as a contract, policy or report.
- Restructure the prompt: documents on top, wrapped in tags, question at the end.
- Add a step to extract supporting quotes before answering.
- Test with questions you already know the answers to, especially about information in the middle of the document.
Reading and summarising documents with AI is useful in almost every field, from accounting and law to procurement. Explore courses to build on it at Plants.work Learn, or find roles that use this skill on the jobs page.