Anthropic Workshop: Build Agents That Run for Hours — Ash Prabaker & Andrew Wilson
Summary
Ash Prabaker and Andrew Wilson, engineers on Anthropic's Applied AI team, delivered a workshop at AI Engineer Europe 2026 on building agents capable of running for 5–12 hours or more. The talk traces both the model and harness evolution over the past year and then introduces state-of-the-art harness patterns currently in use internally at Anthropic.
Andrew opened with a historical tour. He identified three core challenges that prevent agents from running long: context limitations (context rot, amnesia across sessions, and 'context sense anxiety' near window limits), poor out-of-box planning (agents try to one-shot everything or abandon half-built features), and inability to self-evaluate (models exhibit sycophancy toward their own output, marking incomplete features as done). He showed an internal benchmark metric — the minimum harness score at 50% task completion — rising from roughly 1 hour with Claude Opus 3.7 to 12 hours with Opus 4.6. Key releases he traced: Sonnet 3.5 artifact iteration as a precursor; Claude Code research preview (February 2025) on Sonnet 3.7; the Ralph Loop technique (Jeffrey Huntley, July 2025) — prompt fed into Claude Code CLI on a loop with a fixed plan.md, stop hooks, and a safe-word exit condition; Claude Code 2.0 with checkpoints and the renamed Agent SDK; Sonnet 4.5 with context self-awareness; Opus 4.5 enabling economical multi-agent setups with Opus for planning and Sonnet as worker; skills with progressive context disclosure and programmatic tool calling; and Opus/Sonnet 4.6 with server-side compaction, Agent Teams (sub-agents that communicate peer-to-peer), and 1-million-token context window GA.
Ash then presented the current state-of-the-art harness. The central pattern is a GAN-inspired generator–evaluator architecture: a generator builds, a standalone adversarial evaluator critiques, and this loop repeats. The evaluator uses Playwright MCP to open live pages, click through the app, take screenshots, and score output — not merely read diffs. The key insight is that tuning a dedicated critic to be harsh is tractable; tuning a builder to be self-critical is not. The evaluator grades on four weighted criteria: design, originality, craft, and functionality. Design and originality are weighted higher because Opus 4.6 already handles functionality well; the harness targets preventing 'AI slop' aesthetics. The rubric is calibrated with few-shot reference site examples.
A third role, the Planner, was added to complete the loop. The Planner converts a vague one-line prompt into high-level sprint specs, deliberately avoiding granular technical detail to prevent early specification errors from cascading. Critically, before any code is written, the generator and evaluator negotiate a 'done contract' via files on disk — the generator proposes acceptance criteria, the evaluator pushes back on scope and test weakness, and both iterate until they agree. The evaluator then grades against this negotiated contract, not the original spec. This contract layer is cited as the key innovation absent from the Ralph Loop.
The team demonstrated a 'retro game maker' prompt. Without the harness, play mode was broken — arrow keys did nothing, the agent had no mechanism to verify game behavior. With the harness (approximately $200, 6 hours), the app self-named 'Retro Forge,' included a 54-color sprite palette, a debug HUD for the evaluator, working physics, collision detection, and an AI-level generation assistant — none of which were in the original prompt. The evaluator caught FastAPI route ordering bugs, Boolean logic errors on the delete key, and untested game mechanics.
As models improve, harness components are actively removed. Context-window session resets were dropped for Opus 4.6 because context anxiety was eliminated in post-training. Sprint-level evaluator cadence was relaxed from every sprint to end-of-generation. A DAW (music app) built with the simplified harness ran at roughly half the token cost of earlier runs while still producing a fully functional application.
Practical guidance: use file system for shared state (JSON files rather than markdown, which agents overwrite); pipe traces into files and use a secondary agent to identify divergences; embed timestamped learnings JSON as breadcrumbs for future model sessions; use auto-mode instead of --dangerously-allow-all; and continuously prune harness components as new model generations render them unnecessary.
Key Takeaways
- •Self-evaluation is a trap — use a dedicated adversarial evaluator agent with a separate context window and system prompt tuned to be harsh, not the generator reviewing its own output.
- •Context compaction does not equal coherence; use structured handoffs and clean context boundaries between agent roles rather than relying on lossy compression to maintain coherent long-running sessions.
- •Require the generator and evaluator to negotiate and agree on a 'done contract' before any code is written — this converts vague user stories into specific, testable assertions and prevents the evaluator from rubber-stamping incomplete work.
- •Subjective quality (design taste, originality) is gradable: write a weighted rubric with explicit criteria and calibrate it with few-shot reference examples so the evaluator's taste converges on yours.
- •Use Playwright MCP (or Claude for Chrome MCP) in the evaluator — the evaluator must interact with the live app, not just read diffs, to catch behavioral bugs like broken arrow-key physics or nonfunctional game loops.
- •Use a Planner role that produces only high-level sprint specs, not granular technical plans — over-specification at the start causes early errors to cascade across the entire multi-hour run.
- •Read traces by hand as your primary debugging loop; empathize with the model's perspective (limited vision, no persistent memory) to understand why it made each decision before adjusting prompts.
- •Use JSON files rather than markdown for persistent state (feature lists, progress logs, learnings) because models are less likely to overwrite JSON files arbitrarily.
- •Prune harness components aggressively as models improve — context resets, per-sprint evaluation cadence, and other scaffolding that was necessary for Opus 4.5 became unnecessary with Opus 4.6.
Transcript
Nice meeting you guys. Um I'm Ash. Uh this is Andrew. We both work in uh as engineers in our applied AI uh team here at Anthropic. Um and the kind of topic for this session was uh inspired by a blog post we put out uh just a couple weeks ago actually about how to think about building uh agents that can actually run for really long extended periods of time. You know, we're talking 5 6 hour plus kind of runs. Uh I think we've all seen these kind of demos, you know, of like companies being like, "H...
Related Sessions
