Managing unbounded context with MemGPT memory tiers

MemGPT uses an OS-inspired architecture to manage LLM context through three memory tiers. Implementing this system increases GPT-4 retrieval accuracy from 32.1% to 92.5% by using paging, summarization, and semantic indexing to handle context overflow.

Managing unbounded context with MemGPT memory tiers

Virtual context management and memory tiers

I find the OS-inspired architecture of MemGPT is the most effective way to handle context overflow. It treats the LLM context window as physical memory and uses external storage as a disk. The system manages information through paging, summarization, and semantic indexing. The architecture organizes memory into three distinct tiers to maintain efficiency. The fast-tier sliding window handles recent tokens for immediate tasks. A mid-tier compressed segment handles summarized history. The slow-tier archival or semantic index stores long-term data in a vector database. These levels allow the agent to manage its own information without human intervention. You already know that fixed context windows fail during long conversations, so I focus on how MemGPT manages the overflow.

The main context includes system instructions, a read/write working context, and a FIFO history queue. The external context contains content that the LLM evicts from the main context. The system uses explicit primitives to move data between these tiers. The LLM uses the store primitive to transfer data from the main to the external context. It uses the retrieve or search primitive to pull archival content into the main context. The summarize primitive recursively condenses evicted messages. The update primitive modifies working context or persona data. Paging occurs when memory pressure hits specific thresholds, such as 70% or 100% prompt occupancy. The system manages the movement of data between the main context and external storage through function calls that allow the LLM to store, retrieve, summarize, and update its own information based on memory pressure or user requests.

Implementation and deployment steps

I set up MemGPT by running pip install mymemgpt and then executing memgpt configure. I find the configuration for Azure OpenAI requires the 2023-07-01-preview API version to support function calling. For local setups, I load models like Zephyr-7b or Mistral-7b through the text-generation-webui interface to provide an OpenAI-compliant endpoint. I also tested the system with the llama.cpp backend. To use external data, I use the /attach command to link a vectorized data source to the agent. MemGPT uses LanceDB as the default archival storage, which provides persistent HDD storage. This storage allows the user to scale from gigabytes to petabytes.

I can create a persona by adding a .txt file to the /memgpt/personas/examples directory and using the --persona flag. For those using the Discord bot, I found the process involves using the /profile command to create a profile, the /key command to enter an OpenAI key, and the /create command to build a chatbot. I also found that configuring Azure OpenAI involves selecting specific models like gpt-4-0613 or gpt-3.5-turbo-1106. If I encounter an "invalidrequesterror" in Azure, I remove the .memgpt directory in my home folder and restart the configuration.

Performance results in retrieval tasks

The performance gains in retrieval tasks are clear. In the Deep Memory Retrieval task, the accuracy for GPT-4 jumped from 32.1% to 92.5% when I applied MemGPT. For GPT-4 Turbo, accuracy rose from 35.3% to 93.4%. I observed that the ROUGE-L recall for GPT-4 increased from 0.296 to 0.814. These results hold even for multi-hop retrieval tasks where standard models struggle to connect information across documents.

Model Base Accuracy MemGPT Accuracy Base ROUGE-L MemGPT ROUGE-L
GPT-3.5 Turbo 38.7% 66.9% 0.394 0.629
GPT-4 32.1% 92.5% 0.296 0.814
GPT-4 Turbo 35.3% 93.4% 0.359 0.827

The accuracy for GPT-3.5 Turbo moved from 38.7% to 66.9%. I noticed that the system relies on the LLM to manage its own memory, which can lead to performance issues if the model has limited function calling capabilities. GPT-3.5 showed significantly degraded performance because of these limitations. I also find that retrieval precision depends on the quality of the embeddings. Can the system maintain this accuracy when the number of external documents grows into the millions? The accuracy for GPT-4 Turbo rose from 35.3% to 93.4%, and the ROUGE-L score increased from 0.359 to 0.827.

airtrain.ai
airtrain.ai

The airtrain.ai newsroom covers AI research, models and the tools built on them.

More on this topic

Stay ahead of AI

Get the week's most important AI stories delivered to your inbox every Monday.

No spam. Unsubscribe anytime.

More Stories