How we solved Context Management in Agents — Sally-Ann Delucia

Sally-Ann DeLucia, Head of Product at Arize AI and core contributor to their internal agent Alyx, delivers a practical account of how the Arize team solved context management after nearly a year of building and iterating on a production AI agent. The talk is grounded in a real-world problem: Alyx operates on top of Arize's own observability platform, meaning the agent must analyze the very trace and span data it generates — a self-referential loop that made context management uniquely challenging.

The central problem is framed not as a pure engineering challenge but as a product and UX issue. DeLucia argues that context engineering — choosing strategically what the model sees — matters more than prompt engineering for agent reliability. She references Andrej Karpathy's framing of context engineering as the successor to prompt engineering, noting that the Arize team encountered this lesson the hard way through a vicious failure loop: Alyx would run on span data, the spans would grow, context limits would be hit, Alyx would fail and retry, adding more data to an already-overfull context.

Three solutions are described in sequence. First, naive truncation — taking only the first 100 characters of context — worked for simple queries but caused Alyx to treat every follow-up as a new conversation, breaking multi-turn reasoning entirely. Second, LLM-based summarization was tried and rejected because it gave the model too much control over what to retain, producing inconsistent and unreliable outputs with no principled notion of what was important. Third, and the approach currently in production, is a head-plus-tail smart truncation strategy combined with a retrievable memory store. Alyx keeps the first and last 100 characters of conversation context, discards the middle, and stores discarded content (including lengthy tool call results) in a memory store with IDs and positional metadata. The agent can invoke a retrieval tool at any point to pull back earlier context it deems relevant, maintaining access without carrying the full history in every prompt. This approach has been stable for several months without requiring changes.

For long-session challenges — users increasingly running 20-plus-turn conversations as they navigate across the Arize application — the team built long-session evals that load 10 turns and test the 11th, making context degradation a measurable and testable failure mode rather than something discovered only via user complaints.

Sub-agents are the third major architectural lever. The team found that data-heavy operations such as multi-query search over hundreds of trace spans did not need to live in the main conversation context. They restructured Alyx so the primary agent holds only chat history and light context, and delegates heavy data operations to sub-agents that carry the intensive intermediate reasoning. Results are passed back to the main agent. This pattern has become the team's default answer when context accumulates beyond what a single agent context window can cleanly handle.

DeLucia notes that an analysis of the recently open-sourced Claude Code source code revealed that Anthropic uses a similar truncation-and-compression strategy, providing external validation of the approach. Open challenges include true long-term memory (currently absent — the memory store is session-scoped), principled context budgeting beyond the current first-100/last-100 heuristic, and cache invalidation hygiene (a question from the audience about prompt caching drew acknowledgment that the team stores context off in a database by ID but has not yet built sophisticated cache-preservation logic). The core takeaway is that agents fail because of context, not prompts, and the winning strategy is one that lets agents remember what they need and forget what they do not.