The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

June 4, 202623:251,586 views

Summary

Vincent Chen, research fellow and co-founder at Snorkel AI, presents a framework for building effective AI benchmarks derived from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants program. The central argument is that AI's ability to measure agents has fallen behind its ability to build them — enterprises are hesitant to deploy agents in high-stakes settings (finance, insurance, healthcare) not because capabilities are absent, but because robust evaluation infrastructure does not yet exist.

Chen structures the framework across two axes: the "science" of rigorous empirical measurement and the "art" of building benchmarks that actually shape the field.

On the science side, he identifies four pillars. First, individual task quality: tasks must be rigorously validated by domain experts with adversarial quality control. He cites GPQA (Graduate-Level Google-Proof Q&A) as a model, noting its multi-reviewer protocol with original authors, reviewers, and adjudicators, plus novel payout incentives tied to inter-reviewer agreement. Second, distributional diversity: benchmarks must define a clear taxonomy and intentionally distribute tasks across it — including rare but disproportionately important failure modes (analogous to motorcyclists in self-driving). MMLU's taxonomy of 57 academic and professional domains across STEM and humanities is cited as a durable example. Third, task difficulty and model headroom: benchmarks must remain unsaturated to reliably separate frontier models. ARC-AGI-2 stayed unsaturated for months and correctly anticipated the o1-style reasoning surge 18–24 months ago. ARC-AGI-3, launched weeks before the talk, launched with frontier models under 1% on every human-solvable task, making it the benchmark the field watches most closely. Fourth, robust eval methodology: benchmarks should measure beyond accuracy — capturing cost, latency, reasoning trace quality, tool use, and policy adherence. Tau-Bench is highlighted for evaluating multi-turn agents on both task completion and constraint adherence (e.g., a model that books the correct flight but violates fare class rules still fails).

On the art side, Chen highlights three differentiators. First, a thesis about where the field is going: Terminal Bench bet on the CLI as the core abstraction for general-purpose agents before coding agents made this obvious, and that bet has been validated as Claude and Codex build enterprise capabilities on CLI-based tools. Second, roadmapping for the field: SWE-Bench's simple idea of using GitHub PRs to measure coding agents spawned an entire family of benchmarks (SWE-Bench Lite, Verified, Pro, Multilingual, Multimodal) and continues to shape how coding agents are evaluated. Third, researcher UX as a first-class citizen: HELM (Stanford CRFM) pioneered standardized modular evaluation harnesses; Terminal Bench 2.0 shipped with Harbor, which has become the de facto evaluation infrastructure for agent builders. Easy contribution, extension, and adoption are treated as core product principles.

Chen closes with Snorkel's internal roadmap for the next generation of benchmarks across three axes: environment complexity (real codebases have org-specific policies, Slack context, flaky CI, parallel contributors — benchmarks capture only a fraction of this), autonomy horizon (testing agents over very long operational windows where context, integrations, and organizational priorities shift), and output complexity beyond plain text (capturing nuanced reward signals for strategic reports, trustworthy uncertainty expression, and new agent-to-agent artifact types that do not yet exist).

Key Takeaways

•The evaluation gap — measuring agents in production — is now one of the most critical unsolved problems in AI engineering; enterprises hesitate to deploy not for lack of capability but for lack of trustworthy measurement.
•Great benchmarks require adversarial quality control with multiple expert reviewers and adjudicators, not just single-author task creation (GPQA's appendix protocol is a replicable model).
•Define a deliberate task taxonomy before building a benchmark; rare but high-stakes failure modes must be over-represented, not sampled proportionally.
•Keep benchmarks unsaturated by targeting capabilities humans have but frontier models don't yet — ARC-AGI-3 launched with models under 1% on human-solvable tasks, preserving years of headroom.
•Measure beyond accuracy: include cost, latency, reasoning trace quality, and policy constraint adherence as first-class eval signals (a model that books the right flight but violates fare rules must still fail).
•A benchmark's thesis about where the field is going matters as much as its empirical rigor — Terminal Bench bet on the CLI before it was obvious and shaped how enterprise coding agents were built.
•The best benchmarks produce new research roadmaps; SWE-Bench's simple PR-based framing spawned Light, Verified, Pro, Multilingual, and Multimodal variants and continues steering the coding agent field.
•Researcher UX is severely underrated: making it trivially easy to run models, contribute tasks, and reuse reward signals for RL fine-tuning drives the adoption that determines a benchmark's real-world impact.
•The next wave of benchmarks must address environment complexity (flaky CI, org policies, parallel contributors), autonomy horizon (agents operating across weeks with shifting context), and output complexity (non-text artifacts, uncertainty expression, agent-to-agent interactions).

Transcript

Hey everybody. How's it going? Lovely. Well, I'm very excited to be here hailing from San Francisco. It's a little bit of a trek over, but I'm super excited to to uh chat with you all um today after the talk uh and beyond. Uh my name is Vincent. I'm a research fellow and a co-founder at Snorkel AI. And uh you know, today I'm going to be talking about um some meta evaluations for building benchmarks. The art and science of what we found to be really useful when building effective benchmarks. I ha...