Fighting AI with AI — Lawrence Jones, Incident

Lawrence Jones, founding engineer at Incident.io, presents three concrete patterns his team developed to use AI tooling to debug and improve their AI-powered Site Reliability Engineering (SRE) product. The core problem: Incident.io's AI SRE runs hundreds of telemetry queries per investigation — across logs, metrics, traces, historical incident data, and codebases — to produce automated root cause analyses (RCAs). With systems comprising hundreds or thousands of chained prompts, it becomes humanly intractable to determine where reasoning fails when an RCA is wrong. A full manual investigation of a single incident takes approximately one hour before a human can evaluate whether the AI output was correct.

The first pattern addresses eval dataset management. Incident.io stores evals as YAML files co-located with Go-language prompts, where each eval specifies input data, expected output, and grading criteria. As these YAML files grew very large (multi-megabyte), coding agents hit context limits and could no longer work with them directly. The team built a small CLI called 'eval tool' that exposes eval suite operations (list, add, edit, replace test cases) to coding agents. This enabled a red-green runbook workflow: the agent writes a failing eval case proving the bug exists, modifies the prompt until the eval passes, then runs the full suite to confirm no regressions. A final consolidation step compresses prompt growth to keep prompts maintainable over time. Jones demonstrated this against a real production prompt that converts natural language queries into log queries for a logging system.

The second and biggest unlock was serializing UI debugging views as downloadable file systems. The team had built rich browser UIs showing prompt hierarchies, tool call graphs, and trace visualizations — but coding agents cannot effectively navigate browser UIs. The insight, which Jones attributes to Anthropic's discovery when building Claude Code, is that coding agents excel at file system traversal. Incident.io now packages every AI interaction as a downloadable file system snapshot, including full trace data rendered as ASCII-friendly text files. Engineers drop this snapshot into a Claude Code session, describe the bad behavior, and the agent traverses the prompt hierarchy to identify exactly which prompt to modify. Because the codebase is also in scope, the agent can suggest and implement the fix in the same session, validated by the eval runbook.

The third pattern handles fleet-scale analysis. Incident.io runs daily backtests across hundreds of customer accounts, producing a single rolled-up accuracy score (e.g., 86% RCA accuracy). This number does not explain why accuracy rises or falls. To address this, the team built an analysis pipeline in a repository called 'scrapbook' that uses structured markdown playbooks to orchestrate a two-stage agent workflow: 25 Claude Code agents run in parallel, each independently analyzing one investigation from the downloaded file system corpus. A second clustering stage aggregates these per-investigation analyses to surface systemic failure patterns — identifying not just how the AI went wrong but which prompt hierarchy changes would address root causes across customer cohorts. The output is an actionable report that drives specific code changes and pull requests.

Jones concludes with three generalizable principles: prioritize internal debugging tooling so it works well with coding agents; prefer downloadable file systems over MCPs or browser-use agents as agent context — bulk file downloads enable grep-based traversal that outperforms interactive UIs; and replace ad hoc analysis with AI runbooks to turn days of debugging into minutes. Incident.io is hiring in London following a significant funding raise.