The Four Types of AI Agent Memory — and Why Each One Is a Healthcare Security Concern

AI Security Series #43

If you are building AI agents — or evaluating vendors who are — one of the most important questions you can ask is: what does this agent remember, and how? IBM Technology's Martin Keen released a video this week breaking down the four types of memory every AI agent needs, grounded in the CoALA (Cognitive Architectures for Language Agents) framework from Princeton researchers. It is one of the cleaner conceptual explainers available on this topic, and the framework it draws on has direct implications for how healthcare organizations should design, evaluate, and govern agentic AI systems.

Memory is also, not coincidentally, one of the least-governed aspects of most enterprise AI deployments right now. Most AI governance frameworks focus on model inputs and outputs. What gets stored between sessions, how it is structured, who can modify it, and what happens when it is wrong — these questions are largely unaddressed. That gap is closing, and this is a good moment to understand the terrain before your organization's agentic surface area gets ahead of your governance posture.

The CoALA Framework

CoALA, proposed by Princeton researchers Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths, describes a language agent with modular memory components, a structured action space to interact with internal memory and external environments, and a generalized decision-making process to choose actions. The framework draws on decades of cognitive science and symbolic AI research, mapping well-understood concepts from human memory architecture onto the design of language agents.

The cognitive typology CoALA uses is grounded in Tulving's foundational work on memory — the same taxonomy cognitive scientists use to describe how humans store and retrieve different types of information. The four memory types are working memory, episodic memory, semantic memory, and procedural memory. Each maps to a distinct technical implementation in a production agent system.

The Four Memory Types

Working Memory: The Context Window

Working memory is the agent's active scratchpad — what it currently knows and is reasoning about right now. In LLM-based agents, this is the context window: the system prompt, the current conversation, any documents or tool outputs loaded into the session. It is fast, immediately accessible, and volatile. When the session ends, working memory is gone unless explicitly written elsewhere.

The security implications are immediate: everything in working memory is within the agent's reasoning scope. Prompt injection attacks, jailbreak attempts, and context poisoning all target working memory. The context window is the attack surface for a wide class of agentic threats because it is where instructions, data, and tool outputs all coexist and get processed together.

Semantic Memory: The Persistent Knowledge Base

Semantic memory is where the agent stores facts, rules, documentation, and organizational context that need to persist across sessions. This is not the model's training data — it is external, writable, and specific to the deployment. In production systems, Keen notes this is frequently implemented as structured Markdown files (for example, a claude.md or AGENTS.md file) that load into the agent's context at the start of each session, giving it consistent organizational knowledge without retraining.

This is also where the security and governance surface area begins to expand meaningfully. Semantic memory files are writable artifacts. They can be modified by the agent itself in some architectures, by administrators, or potentially by attackers with access to the underlying file system or repository. Poisoned semantic memory — incorrect facts, tampered rules, injected instructions disguised as documentation — persists across every session that loads it. Unlike a prompt injection that affects a single interaction, a compromised semantic memory file affects every subsequent agent invocation until detected and corrected.

Procedural Memory: The Skills Layer

Procedural memory represents the agent's learned capability to perform specific tasks — not facts about the world, but instructions for how to act in it. In the agent skills architecture Keen describes, this is implemented as skill.md files: modular instruction sets that define how the agent should handle specific task categories. Critically, production systems use progressive disclosure — the agent only loads the detailed skill instructions when a task matching that skill is requested, rather than loading all skills into every context window.

Progressive disclosure is both a performance optimization and a security design. Keeping skill instructions out of the default context window reduces the attack surface for prompt injection attempts that try to manipulate the agent's behavior by referencing capabilities it shouldn't be using for the current task. It also limits the blast radius of a compromised skill file — only sessions invoking that specific skill are affected.

The governance question for procedural memory is access control: who can write or modify skill files, is there version control and audit logging on changes, and is there a review process before a new or modified skill goes into production? In healthcare environments running agent orchestration programs, skill files that can instruct agents to query clinical systems, send communications, or trigger workflows need to be treated with the same change management rigor as application code.

Episodic Memory: The Record of Past Interactions

Episodic memory is the agent's record of what happened in past sessions — decisions made, outcomes observed, mistakes encountered. Keen's key point here is that high-quality episodic memory implementations do not save full transcripts. They use distillation: a process that extracts the most useful information from a session and stores a condensed, structured summary rather than a verbatim record. This keeps episodic memory manageable in size and focused on actionable learning rather than raw history.

The healthcare relevance here is significant on two fronts. First, if an agent's episodic memory contains summaries of past clinical interactions, patient-related queries, or PHI-adjacent activity, that memory store is itself a potentially HIPAA-relevant data repository — regardless of whether any individual entry contains explicit PHI. Second, episodic memory is a persistence mechanism: decisions and learned behaviors from past sessions carry forward into future ones. An agent that learned a bad pattern from a corrupted interaction will repeat it until that episodic record is corrected.

Memory Complexity Scales With Agent Complexity

One of the more practically useful points in the IBM video is that not every agent needs all four memory types. Keen frames it as a complexity ladder: a simple reflex agent — one that responds to inputs with predefined actions without maintaining state — may only need working memory. A more capable task-completion agent adds semantic memory to carry organizational context between sessions. A fully autonomous agent handling complex, multi-session projects typically needs all four.

This matters for governance because it gives organizations a framework for scoping their agent memory policies to actual deployment complexity. A basic customer-facing chatbot and a multi-step clinical workflow orchestration agent are not the same risk profile. The memory architecture they use — and the governance controls required for each layer — should be proportional to what they are actually doing and what they have access to.

Agent Type	Memory Needed	Governance Priority
Simple reflex / FAQ bot	Working only	Context window injection controls
Session-aware assistant	Working + Semantic	Semantic memory integrity and access control
Task automation agent	Working + Semantic + Procedural	Skill file change management, audit logging
Autonomous orchestration agent	All four	Full memory governance framework across all layers

What This Means for Healthcare

Memory Stores Are a New PHI Risk Surface

Current HIPAA guidance focuses on where PHI is stored, transmitted, and processed. Agentic AI introduces a new category: memory stores that accumulate organizational context over time and may come to contain PHI-adjacent information without any explicit decision to store it there. Episodic memory summaries of clinical workflow sessions, semantic memory files that document patient intake procedures, procedural memory files that define how to query an EHR — all of these can constitute PHI exposure points depending on their content and how they are managed. Your AI governance program needs to include memory store inventory, classification, and access control alongside the model governance work.

Skill Files Need Change Management

If your agent orchestration program uses skill files to define how agents interact with clinical systems, billing platforms, or internal data sources — and it likely does or will — those files need version control, access control, and a review and approval workflow before changes go live. A skill file that instructs an agent to query a patient database or initiate a clinical communication is an executable artifact with real-world effects. Treating it as a configuration file rather than code is a governance gap. Treat it as code.

Episodic Memory Distillation Quality Matters

What gets distilled into episodic memory, and what gets discarded, is a design decision with both functional and security implications. If an agent distills the wrong lessons — for example, learning to skip a validation step because it succeeded once without it — that learned behavior persists. In healthcare workflows where agents are assisting with clinical decision support, medication reconciliation, or coding, bad episodic learning compounds over time. Episodic memory content should be periodically reviewed as part of agent performance monitoring, not just at initial deployment.

Semantic Memory Integrity Is an Attack Target

In the same way that RAG poisoning attacks target the knowledge bases used for retrieval-augmented generation, semantic memory poisoning targets the persistent context files that agents load at session start. An attacker with write access to a claude.md or equivalent file can inject persistent instructions that influence every subsequent agent session without touching the model or the application code. This is a supply chain risk for agent deployments — the integrity of the files that define what the agent knows needs to be protected with the same controls applied to application configuration: access control, integrity monitoring, and change audit logging.

The Bigger Picture

The CoALA framework's roots in cognitive science give it durability that tool-specific frameworks lack. The underlying memory taxonomy — working, semantic, procedural, episodic — reflects how researchers understand human cognition, not how a particular vendor has chosen to architect a product. That means it provides a stable conceptual lens for evaluating any agentic system, regardless of which platform or model it runs on.

For healthcare security teams, the practical value is in having a vocabulary and a structure for asking the right questions about any agentic deployment. What memory types does this agent use? How are each of those stores protected? Who can write to them? What happens when they contain errors? Is there audit logging? Is there a process for reviewing and correcting episodic learning over time?

Most enterprise AI governance frameworks don't ask these questions yet. The organizations that build that governance posture now — before their agentic deployments scale — will be significantly better positioned than those who retrofit it after the fact. Memory is what separates a chatbot from an agent. It is also where the most persistent and hard-to-detect security risks in agentic systems live.

One thing worth noting: the four-memory architecture Keen describes is not purely theoretical. It is implementable today using nothing more than a structured file system and a disciplined workflow — no custom platform required. We are running a version of exactly this pattern in our own AI security research workflow here at bregg.com, using structured Markdown files for semantic and procedural memory and a distillation process for episodic memory.

In a future post we will walk through that implementation in detail — what the files look like, how the memory layers interact during a session, and what the security controls around each layer are — as a practical blueprint for security practitioners who want to stand up a governed agentic workspace without waiting for enterprise tooling to catch up.

This is entry #43 in the AI Security Series. For related coverage, see AI Security Series #42: Kali365 PhaaS Kit Hijacks Microsoft 365 OAuth Tokens.