Skip to content

Working Memory Management

Overview

Working memory (also called short-term memory) is the active state the agent is reasoning over right now — the running conversation and the scratchpad the model sees at inference time. It lives in the LLM's context window — the most expensive and capacity-constrained part of the memory system. Effective working memory management is critical for maintaining coherent conversations, avoiding context overflow errors, and controlling inference costs.

In the CoALA taxonomy, working memory is the only in-context memory type. The three long-term types (semantic, episodic, procedural) all live outside the context window and must be retrieved into working memory when needed.

The Core Challenge

LLMs have a fixed context window (ranging from 4K to 1M+ tokens depending on the model). As conversations grow longer, you must decide: - What to keep in the context window (working memory) - What to summarize or compress - What to offload to semantic, episodic, or procedural memory - What to discard entirely

Context Window Token Budget Layout

Production agents on large-context models (e.g., Claude 200K tokens) must manage a layered token budget across several competing consumers:

Context Region Typical Size Notes
System prompt Stable, design-time Best candidate for KV cache — stable across turns saves ~85% of input token cost for that portion
Windowed conversation history ~1,200–3,500 tokens per turn (windowed) Variable; managed by the strategies below
Retrieved RAG chunks top-k × chunk_size tokens Injected per turn; size depends on retrieval strategy
Tool call results Per-tool, often large Accumulated during reasoning; may need summarization
Agent scratchpad / CoT Intermediate reasoning LLM-internal; may be exposed or hidden depending on framework
Reserved for response max_tokens budget Must be reserved; cannot be consumed by history

Key metric: KV cache hit rate. On Amazon Bedrock, Claude caches the system prompt portion of every context — cache hits save approximately 85% of input token cost for that portion. Design system prompts to be stable across turns (no per-turn dynamic content injected into the system prompt block).

Working Memory Techniques

Sliding Window

Logic: Retains only a fixed number of the most recent messages, discarding the oldest once the limit is reached.

Primary Benefit: Prevents context overflow errors and maintains low latency.

Tradeoff: Loses older context entirely — the agent "forgets" early parts of the conversation.

Best For: Simple chatbots and task agents where recent context is most relevant.

# LangChain ConversationBufferWindowMemory
from langchain.memory import ConversationBufferWindowMemory

memory = ConversationBufferWindowMemory(k=5)  # Keep last 5 exchanges

Token Trimming

Logic: Dynamically calculates and removes the minimum number of tokens from the start of the history to fit the model's context limit.

Primary Benefit: Maximizes use of the available context window without crashing.

Tradeoff: May cut off important early context mid-sentence.

Best For: Applications where you want to use as much context as possible without manual tuning.

Recursive Summarization

Logic: Periodically condenses older parts of the conversation into a short paragraph while keeping recent messages in full.

Primary Benefit: Preserves long-term narrative context while saving significant token space.

Tradeoff: Summarization introduces information loss and adds latency.

Best For: Long-running conversations where early context matters but verbatim recall is not required.

# LangChain ConversationSummaryMemory
from langchain.memory import ConversationSummaryMemory
from langchain_openai import ChatOpenAI

memory = ConversationSummaryMemory(llm=ChatOpenAI())

Message Selection

Logic: Uses a "supervisor" model to retrieve only the past messages that are relevant to the current user query.

Primary Benefit: Highly efficient for complex, non-linear tasks where only specific historical facts matter.

Tradeoff: Adds latency and cost for the retrieval step; may miss relevant context.

Best For: Long-running agents with diverse conversation history where most past messages are irrelevant to the current query.

Scratchpad / Working Memory

Logic: Provides a dedicated space (like a file or hidden block) for the agent to store intermediate thoughts and calculations.

Primary Benefit: Offloads technical complexity from the main chat history to keep the conversation clean.

Tradeoff: Requires explicit agent behavior to use the scratchpad effectively.

Best For: Multi-step reasoning tasks, code generation, and complex problem-solving where intermediate steps should not pollute the conversation.

Context Pinning

Logic: Locks vital information (like system instructions or core user preferences) so they are never discarded by trimming or sliding window operations.

Primary Benefit: Ensures the agent never forgets its primary persona, mission, or critical user preferences.

Tradeoff: Reduces available space for dynamic context.

Best For: All production agents — always pin system instructions and critical user context.

Semantic Window (Semantic Compression)

Logic: Embeds all history chunks into a vector store, then retrieves only the turns semantically relevant to the current query instead of the most recent k turns.

Primary Benefit: Reduces a 50+ turn history to 3–5 highly relevant turns. Improves response quality for long-running sessions with branching topics.

Tradeoff: Requires a vector store for history (+15ms retrieval latency). More complex to implement than sliding window.

Best For: Long-running sessions where topic branching makes recency a poor proxy for relevance.

Progressive Summarization

Logic: When the accumulated token count exceeds a threshold, call a model to summarize older turns, replacing the raw turns with a structured summary object.

Primary Benefit: Preserves semantic content while reducing token count by 80–90%.

Tradeoff: One extra LLM call per compression event introduces latency.

Best For: Sessions where raw history exceeds context budget but semantic fidelity must be preserved.

Working Memory Solutions

Category Solution Memory Philosophy Best For
Frameworks LangGraph Checkpoints Stateful Threads Saving the exact "snapshot" of a multi-step workflow
Frameworks LangChain Window Sliding Buffer Keeping only the last N interactions to save tokens
Frameworks Letta (MemGPT) Virtual Context Dynamically swapping info in/out of the context window
In-Memory Redis / Upstash Ephemeral Cache Low-latency session storage for high-speed chat apps
Managed OpenAI Threads Stateful API Hands-off management of current conversation history
Technique Summarization Recursive Compression Condensing old messages into a brief "recap" to save space
Technique Scratchpad Working Draft Agent-written notes used only for the current task logic

Windowing Strategy Comparison

Strategy Eviction Logic Complexity Latency Best For
Last-k turns Keep most recent k=10–20 turns O(1) Zero Most conversational agents
Token-budget window Slide window until token count ≤ budget; add newest first, evict oldest Low Zero Precise token control
Semantic window Embed all turns; retrieve top-k most relevant to current query Moderate (+vector store) +15ms Long sessions with branching topics
Progressive summarization Summarize oldest segment when threshold exceeded High (+LLM call) +model latency High-fidelity preservation at scale

LangGraph Checkpointer Pattern

The LangGraph checkpointer implements the checkpoint storage and retrieval pattern for session memory. On each invoke(), the agent runs a think-act-observe loop, serializes state, and writes an atomic checkpoint keyed by thread_id + checkpoint_id. On the next turn, it loads the checkpoint, deserializes state, and continues.

Storage backends: Amazon DynamoDB (consistent read, GET by thread_id) or Amazon ElastiCache (sub-millisecond, PUT/GET).

LangGraph's checkpointing system saves the complete state of a graph execution at each step, enabling:

  • Pause and Resume: Stop a long-running agent and resume later
  • Human-in-the-Loop: Pause for human approval before continuing
  • Time Travel: Replay from any previous checkpoint for debugging
  • Fault Tolerance: Recover from failures without restarting from scratch
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import StateGraph

checkpointer = MemorySaver()
graph = StateGraph(AgentState)
# ... add nodes and edges ...
app = graph.compile(checkpointer=checkpointer)

# Run with thread ID for persistence
config = {"configurable": {"thread_id": "user-123"}}
result = app.invoke({"messages": [...]}, config=config)

Letta (MemGPT) — OS-Inspired Memory

Letta (formerly MemGPT) implements an OS-inspired memory architecture where the agent manages its own context window like an operating system manages RAM:

  • Main Context (RAM): The active context window — fast but limited
  • Archival Storage (Disk): External storage for overflow — unlimited but requires retrieval
  • Recall Storage: Searchable history of past interactions

The agent uses special memory management functions (core_memory_append, archival_memory_search) to explicitly manage what stays in context and what gets offloaded.

Best Practices

  1. Always pin system instructions: Use context pinning to ensure the agent's persona and core instructions are never trimmed
  2. Stabilize system prompts: Keep system prompt content stable across turns to maximize KV cache hit rates (~85% cost savings on cached tokens on Amazon Bedrock)
  3. Choose the right windowing strategy: Last-k for simple conversational agents, token-budget for precise control, semantic window for branching topics, progressive summarization for high-fidelity long sessions
  4. Monitor context utilization: Track what percentage of the context window is being used to identify optimization opportunities
  5. Test edge cases: Verify behavior when context is nearly full — does the agent degrade gracefully?
  6. Consider cost: Longer contexts cost more per inference — balance quality with cost

See Also

References