Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

Tejas Kumar, AI Developer Advocate at IBM, delivers an 18-minute deep dive into agent harnesses — arguing that the core problem with unreliable AI agents is not the prompt, but the absence of a proper harness. He opens by distinguishing two uses of the word: the ML-world definition (a glorified test suite for model quality evaluation) versus the AI engineering definition he focuses on — everything around the model that gives it grounding in reality.

Kumar defines an agent harness through five canonical components: a tool registry (the set of callable functions available to the agent), a model (which may be swapped or degraded without the developer's knowledge, underscoring why harnesses must be model-agnostic), context management primitives (including automatic context compaction), guardrails (such as max iteration counts and max message thresholds), and a verify step (deterministic post-run logic that checks whether the agent actually did what it claimed).

The talk centers on a live-coded demo building a browser automation agent against Hacker News with the explicit goal of upvoting a post. Crucially, Kumar chooses GPT-3.5 Turbo — a deliberately old and cheap model from 2023 — to demonstrate that harness engineering, not model capability, is the primary lever. The agent is implemented using Playwright (not Playwright MCP) for browser control and OpenAI's tool-calling SDK. In its initial harnessless state, the agent hits the Hacker News login page, panics, and reports success despite never upvoting — a case of agent hallucination about its own actions.

Kumar then incrementally layers the harness through four successive diffs without modifying the prompt at any point. First, he adds guardrails: a max-iterations cap (six steps) that kills runaway loops, and a max-messages threshold that triggers naive context compression — always preserving the system prompt, user prompt, and the two most recent messages while discarding the middle. Second, he refactors the flat index.ts into a formal harness module (run_harness), collapsing the agent logic into run_harness_attempt and wrapping it in a retry loop capped at three attempts. Third, he implements a verify_successful_upvote function that reads the full tool-call trace deterministically: it checks whether a browser click on the upvote element actually occurred and succeeded, and returns early with failure if a harness_auto_login tool message indicates a failed login, or if the agent ended up on the login URL without the login handler having run. This eliminates the lying. Fourth, he adds a login_handler that fires on every agent loop iteration, checks the browser's current URL, and — if it detects the login page — programmatically fills credentials from environment variables and submits the form, then injects a message into the agent queue informing the model that login succeeded. The final run completes in six iterations: the agent navigates to Hacker News, the harness handles login transparently, and the upvote is confirmed.

Kumar argues that 2025 was the year of agents, 2026 is the year of harnesses, and predicts 2027 will bring dynamically generated harnesses — where an agent first constructs its own harness (analogous to plan mode but more structurally rigorous) before executing a task. He connects this to IBM's open-source OpenRAG project, which uses enterprise-grade harnesses to enable RAG over sensitive internal data (Teams calls, PDFs, invoices) in air-gapped enterprise environments. The core message: with a well-engineered harness, a cheap commodity model can reliably complete tasks that a more expensive model without a harness will fail.