This post is part 6 of the Agentic AI Series — a multi-part exploration of how autonomous systems are reshaping enterprise architecture, governance, and security.
The Short Version
Context is what your agent can see. The system prompt, conversation history, retrieved documents, tool outputs—everything in the context window during inference. What’s in context shapes what the agent can reason about.
Context is a finite resource. Models have token limits, but the real constraint is attention—too much context and the agent loses focus. Curating what's in the window is the core engineering problem.
Three patterns address single-agent context. Compress what doesn't fit (Context Windowing). Select what's needed from external sources (RAG). Isolate what shouldn't cross boundaries (Memory Isolation).
Context engineering is a governance surface. Session isolation prevents data leaks. Pinned constraints prevent safety drift. Retention policies determine what's auditable.
Patterns compose. Most production systems combine windowing for long conversations, RAG for external knowledge, and isolation for multi-user safety. Start simple; add complexity when failures demand it.
Terminology: Context vs. Memory
Context is what’s in the window during inference. Memory is what’s stored outside and retrieved when needed. Most memory discussions focus on cognitive categories—short-term, long-term, episodic, semantic. The engineering challenge is deciding what moves from memory into context, when, and for how long.
This post focuses on infrastructure: what architectural decisions do you face regardless of memory type?
Context Windowing manages what fits. RAG retrieves what’s needed. Memory Isolation enforces who sees what.
Memory types describe content. Context strategies describe control.
The Single-Agent Context Problem
A single agent serving users must manage three types of friction:
The "Vanishing North Star": A project management agent drafts a report from a 4-hour meeting transcript. By the end, it has forgotten the original goals and hallucinates goals that sound plausible but conflict with the original mission. The context window quietly dropped what mattered most. (Context Windowing)
The “Library without an Index”: A support agent has access to 50,000 pages of troubleshooting documentation. When a user says “it’s not working,” retrieval pulls generic getting-started guides. The actual solution sits unreachable in the database. (RAG)
The “Ghost in the Machine”: A healthcare agent helps multiple doctors. During Patient A’s visit, it recommends a dosage based on Patient B's lab results. Memory contamination across sessions creates physical risk. (Memory Isolation).
These are context failures. Without the right patterns, an agent is either a “goldfish” with no past, a “hoarder” overwhelmed by irrelevant data, or a “leaky pipe” mixing contexts that should stay separate.
If you’ve read the base architecture post, you already have the foundations. This post focuses on the Orchestration layer, which governs context management by deciding what information is retrieved, surfaced, and combined for reasoning at inference time.
When AI Acts: The Architecture Behind Agentic AI
The post architecture post establishes the foundational layers of an agentic AI system: how context is assembled, how plans are formed, how actions are orchestrated, and how outcomes flow back into the system.
Why Context Degrades
Context quality erodes through three mechanisms:
Compression artifacts: Each summarization loses something. Over repeated cycles, small distortions compound into significant drift.
Semantic drift: Models and embeddings change over time. Historical context may be misinterpreted by current system versions.
Attention decay: New content buries old content. Important information stays in the window but stops influencing output.
Effective context engineering is about managing inevitable degradation.
The Four Context Management Strategies
Context engineering breaks into four fundamental strategies. This post covers the first three as they apply to single-agent systems. The next part in this series extends Shared State and Memory Isolation to multi-agent coordination.
Compress what doesn’t fit → Context Windowing
Select what’s needed on demand → RAG
Isolate what shouldn’t cross boundaries → Memory Isolation
Write what must persist for coordination → Shared State
For single-agent systems, the core question is: how does one agent manage what it can see while serving multiple users safely?
Pattern 1: Context Windowing
The finite context management pattern
Context windowing governs what information occupies an agent’s context window at inference time. Given the finite window size, new information forces old information to be compressed, summarized, or discarded. The windowing strategy determines the effective world the agent can reason over in any given step.
The key insight: Context windowing is inherently lossy. Every approach trades off recency, relevance, and fidelity. No strategy preserves all three. The engineering question is which losses are acceptable for the task at hand.
Strategies
Sliding Window: Drop the oldest content as new input arrives. Simple and predictable, but early constraints and objectives disappear silently.
Summarization: Compress prior turns into a high-level gist. Preserves continuity, but loses nuance and irreversibly shapes future reasoning.
Hierarchical Summary: Maintain nested summaries across time horizons (recent detail, older abstractions). Offers the best long-term retention, at the cost of complexity and compounding abstraction error.
Compaction: Strip large tool payloads while preserving the conversational skeleton. Ideal for tool-heavy agents, but risky if discarded outputs later become relevant.
Context windowing strategies are explicit decisions about what you are willing to forget.
Pinning Critical Context
Regardless of strategy, some content must never be evicted:
System prompts
Safety and policy constraints
Key user facts
Task objectives and success criteria
Implement a reserved context budget that shields critical information from windowing pressure:
History Budget = Window Size - (System Prompt + Tools + Pinned Facts + Response Buffer)This prevents the “Vanishing North Star” problem, where objectives silently fall out of context. Critical constraints should be pinned by design, not left to compete with conversational history for space.
At a Glance
Governance Implications: Context windowing determines what evidence the agent can reason over at inference time. If critical constraints or earlier instructions fall out of the window, the agent may violate them—not maliciously, but because it literally cannot see them. Pinning critical content is therefore a governance control, preventing silent objective and policy drift.
Pattern 2: Retrieval-Augmented Generation (RAG)
The external knowledge access pattern
RAG extends agent memory beyond the context window by retrieving relevant information from external sources—vector databases, document stores, knowledge bases—at inference time. Retrieval is driven by the user’s query or by system-level query analysis, and the retrieved content is injected into the context before generation.
The key insight: RAG decouples knowledge from model training. The model performs reasoning; the retriever supplies facts. This separation allows knowledge to be updated without retraining the model and grounds responses in authoritative, up-to-date sources.
Variants
Standard RAG: Retrieve once based on user query, then generate. Simple and predictable, but the initial query may miss what's actually needed.
Conditional Retrieval: Retrieve only when the agent decides it needs external information. Reduces unnecessary calls, but the agent may misjudge when retrieval is needed.
Iterative Retrieval: Multiple retrieval rounds, each informed by previous results. Better for complex questions, but latency compounds and chains can drift.
Agentic RAG: Agent decides what to retrieve, from where, and when. Most flexible, but retrieval decisions become an attack surface, and are harder to audit and govern.
RAG variants are explicit decisions about who controls access to knowledge and how much risk you are willing to accept.
At a Glance
Governance Implications: RAG retrieval is a trust boundary. Retrieved content is injected directly into the context window and influences generation as authoritative input. A poisoned or unverified knowledge base therefore becomes poisoned context, regardless of model quality. Retrieval filtering, source verification, and content validation are necessary security controls, not optional features.
Pattern 3: Memory Isolation (Single-Agent)
The session and user boundary pattern
When a single agent serves multiple users or concurrent sessions, memory isolation enforces boundaries that prevent context from leaking across those boundaries. This is access control applied to agent context—ensuring User A’s inputs, history, and derived state never influence User B’s responses.
The key insight: single-agent deployments are almost always multi-tenant. Without explicit isolation, context can leak across sessions through shared state, cached embeddings, logs, or improperly scoped memory stores—creating privacy, safety, and trust failures by design.
Isolation Levels
Session Isolation: Memory clears when the session ends. Provides the strictest isolation with no cross-session leakage, but users must re-establish context every time.
User Isolation: Memory persists across sessions for the same user while remaining invisible to others. Enables personalization and continuity, but widens the exposure window.
Persistent Memory with TTL: Long-term memory with automatic expiration. Useful for preferences and learned patterns, but stale context can influence future behavior if retention policies are not explicit.
Isolation levels are explicit decisions about how long context is allowed to persist and how much exposure risk you are willing to accept.
Storage Tiering (Hot/Warm/Cold)
Not all context has equal access frequency or cost. Storage strategy should match how often information is needed and how expensive it is to surface.
Hot context is always injected into the context window and incurs token cost on every inference.
Warm context is frequently accessed but selectively injected, balancing token cost and latency.
Cold context is stored externally and retrieved only when needed, incurring latency while minimizing constant exposure.
Design memory tiers around actual access patterns. Poor tiering leads to bloated context windows, unnecessary retrieval, or silent influence from stale information. Storage tiering is a cost and risk management decision.
At a Glance
Governance implications: Session isolation is your first line of defense against context contamination. When Patient A's medical history appears in Patient B's session, that's not a bug in the model—it's a failure in memory isolation. The enforcement layer must be explicit and tested.
Pattern Interactions
Production systems rarely use one pattern in isolation; failures emerge when patterns are combined without intent.
When to Use What
Use context windowing when conversations exceed context limits and continuity matters. Accept that information will be lost and pin constraints that must remain visible.
Use RAG when the agent requires knowledge beyond its training data. Treat retrieval as a trust boundary and enforce filtering, source verification, and validation.
Use memory isolation whenever a single agent serves multiple users or sessions. Enforce isolation at the storage and retrieval layers to prevent cross-session contamination.
These patterns address different failure modes and are commonly required together.
In Closing
Choosing how context is managed is choosing what an agent can see and what it is allowed to combine. Window size, retrieval, and isolation shape the agent’s effective world, determining how information enters reasoning and how errors propagate when visibility is incomplete.
Single-agent systems expand governance surface area quietly. Every retrieval, eviction, and memory boundary is a trust decision, and failures tend to appear as undetected drift rather than explicit errors.
The next part extends these concerns to multi-agent systems: how agents share state through coordination, how isolation prevents cross-agent contamination, and why governance workflows determine whether multi-agent systems remain safe over time.
