Agent memory patterns: vector stores, knowledge graphs, and context engineering

An agent’s context window is working memory: fast, expensive, and wiped between sessions. Everything else — what it learned yesterday, what the user prefers, what happened in the last nine steps of a ten-step task — has to be engineered. That engineering has exactly four recurring patterns.

Step 1 — The four patterns

Pattern	Answers	Typical store
Scratchpad	”Where was I in this task?”	Plain file / task state object
Retrieval memory	”What do I know that’s relevant right now?”	Vector store
Knowledge graph	”How do these entities relate?”	Graph / triples
Profile memory	”What’s durably true about this user/project?”	Small curated documents

Most teams jump straight to a vector database. Wrong order: scratchpad and profile memory are cheaper, more debuggable, and cover most real failures. A NOTES.md the agent reads at session start is profile memory. A todo list it updates mid-task is a scratchpad. Ship those first.

Step 2 — Build retrieval memory you can actually inspect

The retrieval loop — store, score, recall, inject — is identical whether the scorer is bag-of-words or a 3,072-dimension embedding. So build it with a scorer you can debug by eye. Create memory.py:

import math
import re
from collections import Counter

def _tokens(text: str) -> Counter:
    return Counter(re.findall(r"[a-z0-9]+", text.lower()))

def _cosine(a: Counter, b: Counter) -> float:
    dot = sum(a[t] * b[t] for t in a.keys() & b.keys())
    norm = math.sqrt(sum(v * v for v in a.values())) * \
           math.sqrt(sum(v * v for v in b.values()))
    return dot / norm if norm else 0.0

class MemoryStore:
    """Retrieval memory with a transparent scorer.

    Swap _cosine-over-token-counts for embedding similarity in production;
    the interface — and every bug you'll meet — stays the same.
    """

    def __init__(self):
        self._items: list[tuple[str, Counter]] = []

    def add(self, text: str) -> None:
        self._items.append((text, _tokens(text)))

    def search(self, query: str, k: int = 3) -> list[tuple[float, str]]:
        q = _tokens(query)
        scored = [( _cosine(q, vec), text) for text, vec in self._items]
        return sorted(scored, reverse=True)[:k]


if __name__ == "__main__":
    memory = MemoryStore()
    memory.add("User prefers TypeScript over Python for new services.")
    memory.add("The staging database refresh runs Sundays at 02:00 UTC.")
    memory.add("Deploys require two approvals since the March incident.")

    for score, text in memory.search("should we pick Python or TypeScript for new services?"):
        print(f"{score:.2f}  {text}")

Run python memory.py — the preference memory scores on top by a wide margin (0.59 vs 0.00 for the unrelated entries on this toy corpus). Before the model sees anything, you can print exactly why a memory was recalled. That transparency is worth keeping until scale forces you off it.

Step 3 — Inject memories as context, not commands

How recalled text enters the prompt matters as much as retrieval quality. Injected memories should be labeled as background, with provenance:

Relevant notes from memory (may be stale, verify before acting):
- [2026-05-12] User prefers TypeScript over Python for new services.

Unlabeled injection is how agents end up obeying a note from three months ago over the user in front of them. Memories inform; the current conversation commands.

Step 4 — When to graduate each component

• To embeddings: when synonyms start missing (“DB” never matches “database”). Use any embedding API or a local model; keep the MemoryStore interface.
• To a real vector DB (pgvector first if you already run Postgres): when memories outgrow RAM or need sharing across processes.
• To a knowledge graph: only when your questions become relational — “which services depend on the thing that changed?” Retrieval memory can’t answer joins; graphs exist for that. Start small: store (subject, relation, object) triples such as ("billing-api", "depends_on", "postgres-main") in a plain table, and give the agent a tool that walks them. Retrieval finds the right entry point; the graph answers what is connected to it. Reach for a graph database only when those walks get deep or slow.

Step 5 — Forgetting is a feature

Memory that only grows becomes noise that costs tokens. Production memory needs an eviction story: expire by age, deduplicate near-identical entries, and — most effective — have the agent periodically rewrite its memory files, compressing ten observations into one conclusion. Curation beats accumulation.

Step 6 — Context engineering: the budget all of this competes for

Context engineering is the discipline of deciding what goes into the model’s context window on each call, and what stays out. Memory is one input. The others are the system prompt, tool definitions, retrieved documents, and the conversation so far. They all compete for the same window, and a stuffed window makes the model slower, more expensive, and less accurate.

The patterns that keep the window lean are the ones in this tutorial, applied to everything:

• Load on demand, not up front. Recall memories when the query needs them (Step 2). Coding agents now do the same with tools: Claude Code’s tool search loads only tool names at session start and fetches full definitions when needed, and skills load their instructions only when relevant. See skills vs. tools vs. hooks vs. subagents.
• Summarize, then drop. Compaction (rewriting a long history into a short summary) is Step 5’s “curation beats accumulation” applied to the conversation itself.
• Isolate noisy work. A sub-agent with its own window can read 50 files and return three sentences. That’s the orchestrator pattern.
• Label what you inject (Step 3), so the model can weigh it.

If the question is whether the agent needs memory, retrieval, or a fine-tuned model in the first place, start with RAG vs. memory vs. fine-tuning.

Troubleshooting

The agent keeps acting on stale memories

Add timestamps to every entry, label injected memories as potentially stale (Step 3), and prefer recency-weighted scoring: score * decay(age). Most “the agent is confused” reports are actually “the memory is old.”

Retrieval returns plausible-looking but irrelevant memories

Top-k always returns something — even when nothing is relevant. Add a minimum-score threshold and inject nothing below it. An empty memory section beats a misleading one.

Where to go next

Memory changes behavior over time, which is exactly what silent regressions look like. Pin it down with golden-task evals, and include tasks where a stale memory should be ignored.