Agentic Evaluations at Scale, For Everybody — Nicholas Kang & Michael Aaron, Google DeepMind

Nicholas Kang (Product Manager, Kaggle Benchmarks) and Michael Aaron (Software Engineer, Kaggle) from Google DeepMind present a diagnosis of the current AI evaluation landscape and detail four concrete infrastructure products Kaggle is building to address it. The talk opens with a sharp illustration of how benchmark results can be simultaneously correct and meaningless: on SWE-Bench Pro, six frontier models cluster within a couple of percentage points of each other, yet the harness they run inside shifts performance by 22%. A competing lab reran a Kaggle-hosted benchmark with their own API compaction settings and published substantially better results — neither number was fraudulent, both were uninformative. This harness-versus-model ambiguity is a central challenge the speakers return to throughout.

Kang identifies three systemic failures in today's eval ecosystem. First, benchmarks are scattered and stale: over ten new benchmarks are published daily on arXiv, authors move on after publication, and leaderboards go unupdated. Second, evaluations are opaque and non-reproducible, with model-release notes presenting benchmark charts whose configurations are undisclosed. Third, the supply of benchmarks is dangerously narrow: approximately 30,000 AI researchers are creating evals for a world of 30 million technical practitioners, producing coverage gaps that will deepen cognitive jaggedness — superhuman performance in some domains paired with mediocre performance in others.

To anchor the equity argument, the speakers describe a real Kaggle user: a wastewater treatment plant engineer from Turkey with 20 years of field experience who built a novel safety benchmark after incidents in his country caused fatalities due to violated protocols. That dataset exists nowhere else on the web and falls outside the economic incentives of any AI lab. It exemplifies why democratized, open-source benchmark contribution matters.

The four Kaggle products presented are: (1) Hackathons — a platform for channeling community expertise toward benchmark creation, currently running a live hackathon with the Google DeepMind AGI team focused on five of the ten cognitive faculties from a recently published AGI measurement paper; (2) Standardized Agent Exams (originally called SATs until a trademark issue arose) — an MVP launched the week before the talk that lets anyone paste a one-line prompt describing their agent and receive a leaderboard score; it attracted 500-plus submissions in its first week without active promotion, and community-created "exam prep" content appeared organically on social media; (3) Game Arena — a PvP benchmark where models compete at poker, chess, and Werewolf for an ELO rating computed via Bradley-Terry pairwise comparisons, with game transcripts published as open Kaggle datasets; running 400,000 poker hands was required to achieve statistical significance, illustrating the cost challenge; distinct model personalities emerged (Grok goes all-in frequently; newer generation models are more risk-averse and perform worse at poker); (4) the Benchmarks Platform — a community-facing eval platform where users write assertion-based tasks (supporting both rule-based checks and LLM-as-judge scoring), group them into benchmarks, and run them against any collection of models, with results shared publicly.

Key engineering details: Game Arena runs on OpenSpiel (a reinforcement learning framework), uses an LLM model proxy available on Colab for uniform model access, and is built on Kaggle's legacy simulation platform. The Bradley-Terry pairing algorithm reduces the number of games needed to establish statistical significance. All code is open source on GitHub. Challenges candidly acknowledged include high compute cost, difficulty sustaining community engagement at scale, model endpoint opacity (endpoints may quietly swap underlying models), deprecation cycles making longitudinal comparison hard, and calibrating exam difficulty so it provides signal without being too costly or too easy to complete.