Multi-Layer LLM Memory Architecture for Conversational AI

Giving an LLM memory isn't hard. Giving it the right memory at the right moment — without blowing your context window — is the actual engineering problem. Here's the architecture I use.

Layer 1 — History window

A rolling buffer of recent turns. Size depends on your use case: customer support needs ~10 turns, coding assistants need ~30. Always sent in full for cache continuity.

Layer 2 — Personality and user profile

Static facts about the user that almost never change: role, preferences, communication style, timezone, team context. Injected once at session start, cached aggressively.

Layer 3 — Decisions window

A compressed log of decisions made in previous sessions. Not the full transcript — just the outcomes. Rebuilt each session from a structured store.

Layer 4 — Vector retrieval

Semantic search over documents, past conversations, or knowledge bases. Only retrieved when relevant — keeps context window lean. The retrieval trigger is the most important design decision.

How the layers interact

Each layer has different staleness tolerance and retrieval cost. The key insight: layers 1-3 should be static within a session, layer 4 is dynamic per turn.