Large Language Models have a limited “context window” (short-term memory) for any given prompt. By default, they forget everything outside this window once they complete a response.
To put things in perspective, the context window of OpenAI GPT-4 (Standard) is 8,192 tokens, which is roughly 32,000 characters, or about 6,000 words, or comparable to two or three chapters of a standard novel. That is not sufficient context to solve many real-world challenges.
If we look at history, we find the computers we use today had exactly the same problem: their CPU memory was too small to perform meaningful tasks. It gave birth to many further innovations.
Back to LLMs. To overcome the limited context challenge, developers use memory expansion strategies that let models retain and retrieve information over longer sessions or across large documents. This explainer covers how retrieval-augmented generation (RAG), agentic memory systems like MemGPT, and long-context extensions work to expand an LLM’s effective memory.
1. Retrieval-augmented generation — external knowledge memory
Basic idea: give the LLM access to an external knowledge base, often a vector database of embeddings. Rather than relying only on what is in the model’s trained weights, a pre-step retrieves relevant facts on the fly and adds them to the prompt.
How it works: when a query comes in, the system generates an embedding of the query and finds similar vectors — passages, documents — in an external store. The retrieved snippets are then prepended or appended to the prompt as context. The model reads this augmented prompt and incorporates the facts into its answer.
- Embed query. Convert the user’s query or conversation context into a vector.
- Retrieve documents. Search the vector store for relevant information.
- Augment prompt. Insert the retrieved text into the prompt, often with citations.
- Generate response. The model produces an answer grounded in both its own knowledge and the provided reference text.
Where it applies:
- Model-agnostic. It works with open-source models and closed APIs alike, since it just feeds additional text into the prompt.
- Works well when the knowledge base is much larger than the context window.
- Ideal for Q&A systems, knowledge retrieval tools, and chatbots that need factual accuracy.
The CPU analogy: like a computer’s disk cache, which receives data from external storage for specific tasks. The model can be given a knowledge base much larger than its normal memory.
Where it is going: for more intelligent retrieval, many enterprises are building hybrid approaches that integrate RAG with agentic memory architectures. There are also more advanced RAG implementations, predominantly focused on improving the accuracy of the context provided to the model.
Further reading: Pinecone, NVIDIA, LangChain.
2. Agentic memory systems, for example MemGPT
Basic idea: let the model itself manage memory — deciding what to store, where, and when to retrieve it. MemGPT treats the LLM as an operating system process with limited RAM, and equips it with functions to swap data between that “RAM” (the context window) and tiers of external storage. The model manages its own context, much like virtual memory in an OS.
How it works:
- The model is given multiple memory tiers:
- Context memory — limited working memory, like RAM.
- Recall storage — fast external memory for recent interactions, like a disk cache.
- Archival storage — long-term memory for older or less-used data, like a database.
- The model also acts as controller: it can call functions such as
STOREorRECALLto move data between tiers.
Where it applies:
- Better suited to open-source models, or systems where external orchestration is possible.
- Useful for long-running agents, personalised assistants, and autonomous systems.
- Acts like a database plus a memory manager, where the model decides what to remember or forget.
The analogy: an OS memory manager swapping data between RAM and disk.
Where it is going: recent systems focus on multi-tier memory architectures to improve persistence and contextual understanding over time.
Further reading: the research paper and an example system prompt.
3. Extended context — stretching the window
Increasing the context window outright allows a model to process much longer texts at once. This approach is straightforward but resource-intensive.
Where it applies:
- Best suited to commercial models with large context capabilities.
- Useful for analysing long documents or maintaining detailed conversations.
The analogy: adding more RAM to increase working memory.
A myth worth busting: extended context does not eliminate the need for RAG or MemGPT-style approaches.
RAG remains crucial for scalable knowledge access. Loading vast databases or external knowledge through retrieval is more practical and cost-effective than loading everything into a context window.
Similarly, agentic memory is essential for persistent, structured memory across long sessions — adapting over time and managing knowledge actively, as a personal assistant or autonomous agent must. A large context window alone lacks dynamic memory management.
Where this is heading
Combining several strategies produces systems that maintain both knowledge and context over time. An advanced assistant might use RAG to pull in fresh information, a long-context model to hold a detailed conversation, and an agentic memory module to store important user details for future sessions.
The most recent work focuses on making models more autonomous in managing memory. Techniques like MemGPT and other agentic memory frameworks are at the frontier, letting a model function more like an operating system with RAM and disk than a stateless predictor.
Commercially, APIs are offering larger context windows and plugins for retrieval, which suggests long-term memory will soon be a standard feature rather than an engineering project.
Thanks to MIT professors Abel Sanchez and John R. Williams for thoughtful discussions on these topics. The views expressed are personal.