AI Agent Memory: Short-Term, Long-Term and Company Knowledge

How AI Agents remember: context rot, compaction, three types of long-term memory, and pulling company knowledge just in time.

Yesterday you told your HR assistant AI that you work at the Chiang Mai branch and prefer answers in Thai. Today you open a new chat to ask about annual leave, and it asks again which branch you are at. That feeling of talking to a new employee every day has a clear technical cause: the agent does not have a well-designed memory system.

What is AI Agent memory

A language model on its own remembers nothing between calls. Each time it runs, it sees only what is in its context window at that moment. Agent memory therefore means the system that decides what to keep, where to keep it, and what to load back into context when it is needed. To review the basic parts of an agent first, read Understanding AI Agent architecture.

Short-term memory: within one conversation

LangGraph's documentation describes short-term memory as "thread-scoped": it keeps the message history within a single session and saves state so the work can resume later. The problem is that the longer you talk, the longer the context gets.

Anthropic calls this context rot in Effective context engineering for AI agents (September 2025): as tokens accumulate, the model's ability to accurately recall information from its context declines. The article suggests:

• Compaction: summarise a conversation that is nearing the context limit and restart from that summary
• Clearing old tool results: long search results that have already been used do not need to stay in context

Long-term memory: across conversations

Long-term memory lives outside the context and persists across sessions. LangGraph groups it into three types modelled on human memory.

Semantic: facts

For example, "This employee is at the Chiang Mai branch and prefers Thai answers." It can be stored as a profile (one document that keeps being updated) or a collection (many small documents, less likely to lose information but harder to search and update).

Episodic: experiences

For example, "Last time, a mismatched payslip was resolved by referring the person to accounting." These are often used as examples to guide similar tasks.

Procedural: how to work

The agent's own rules and steps, such as a system prompt that says "Always cite the section number of the employee handbook."

Agents that take their own notes

Another technique Anthropic describes is structured note-taking: the agent writes notes outside its context, such as a NOTES.md file or a to-do list, and reads them back when needed. The article's example is Claude playing Pokémon, which tracked its progress, maps of areas explored and which attacks worked, so it could carry on after its context was reset.

A similar idea appeared in the research paper MemGPT: Towards LLMs as Operating Systems (Packer et al., 2023), inspired by the hierarchical memory of operating systems: moving data between fast and slow tiers to give the impression of more memory than is really available.

Company knowledge: memory that is not the agent's

The employee handbook, leave policy or product prices should not be "remembered" in the agent's memory. They should be fetched from the source every time. Anthropic calls this just-in-time retrieval: keep lightweight references such as file paths or links, and load the content with tools when needed. This keeps information current, because when a policy changes the agent sees the new version immediately.

Watch out for

• Wrong memories stick: if the agent stores a fact incorrectly, the mistake follows every conversation. Give users a way to view and delete memories
• Personal data: storing employee or customer data across sessions falls under Thailand's Personal Data Protection Act (PDPA). Keep only what is necessary and consult your legal team or a professional before going live
• Cost and latency: LangGraph notes that writing memories during the conversation (the hot path) adds latency, while writing in the background is faster but the data may not be up to date immediately

A good agent does not remember everything. It remembers the right things and knows where to look up the rest.

How to get started

1. Separate clearly which data belongs to the conversation, to the user, or to the organisation
2. Start long-term memory with a short profile of only a few fields you actually use
3. Fetch company knowledge from the source instead of copying it into memory
4. Set a compaction rule for when conversations exceed a set length
5. Let users see and edit what the agent remembers

Want to try using AI to plan a career path in AI? Talk to the Plants.work AI Career Assistant, or build an agent in a real workflow through the n8n AI course.