Why building eval platforms is hard — Phil Hetzel, Braintrust

April 28, 202625:398,621 views

Summary

Phil Hetzel, Head of Solutions Engineering at Braintrust, delivers a technical talk on the hidden complexity of building evaluation platforms for AI agents. Drawing on his experience working with Braintrust's customer base — which includes companies like Notion — Hetzel maps the maturity stages teams go through when building eval infrastructure and explains why what appears simple on the surface becomes a deeply challenging systems engineering problem.

Hetzel begins by framing why evals matter: LLMs have extreme variability, agents are becoming the norm for customer-facing experiences, and without confident quality assurance, teams face brand, compliance, and cost risks in production. He references conversations where engineers confidently claim they can 'vibe code Braintrust,' using this as a jumping-off point to expose the iceberg beneath the surface of what eval platforms must actually do.

He walks through four maturity stages. Stage one is the spreadsheet-and-for-loop approach: easy to start, zero barrier to entry, but fundamentally a documentation tool rather than an experimentation engine. It doesn't scale for team collaboration, can't easily compare experiments over time, and relies on slow manual human scoring. Stage two is the 'vibe-coded UI' — teams build a bespoke interface with a real database (e.g., Neon) for persistence, bringing non-technical stakeholders into the loop, but still primarily a reporting tool. Stage three introduces a playground for experimentation: users can tweak system prompts and compare agent configurations side-by-side with automated scoring functions, which is where 'the rubber meets the road.' Stage four connects the offline eval loop with production observability — what Hetzel calls 'the flywheel.' He cites a real customer example: a Braintrust user was piping all production traffic into the eval database and running evals against it on an hourly basis, which directly prompted Braintrust to build native tracing and observability features.

Hetzel devotes significant attention to the data engineering challenges that make eval platforms genuinely hard. Agent traces are semi-structured to unstructured, can reach 10–20 megabytes per span (versus a few kilobytes for conventional application spans), arrive at high velocity, and require both low-latency read patterns for real-time observability and aggregate read patterns for analytics. Braintrust originally stitched together an open-source data warehouse, a custom domain-specific query language called BTQL (which, Hetzel admits, no one liked, including the team itself), and DuckDB in-browser for a third aggregation layer. This approach broke down at scale — particularly with customers like Notion sending large volumes of unstructured text — because none of the underlying technologies were optimized for full-text search across LLM trace data.

Hetzel argues the problem is not primarily a UX challenge: building a nice-looking eval UI is tractable with current AI-assisted coding tools, but building the correct data layer — one that supports full-text search across millions of traces, handles multi-megabyte semi-structured spans, and serves both high-velocity writes and complex aggregate queries — is a genuinely novel systems problem. He also highlights emerging headless use cases where engineers integrate coding agents (such as Claude Code or OpenAI Codex) directly against the eval platform's SQL interface to run automated quality improvement loops without any human in the loop.

Looking ahead, Hetzel identifies three areas where eval platforms will need to evolve: (1) surfacing unknown unknowns through topic modeling to direct engineering effort toward real failure modes; (2) making platforms agent-native so that coding agents can programmatically interact with evals; and (3) adding non-functional requirements at scale such as role-based access control, data masking, and automatic tracing via an AI proxy or gateway for centralized governance. The session closes with a brief Q&A covering multimodal trace support (Braintrust stores audio and video in object storage and renders them inline) and prompt management capabilities.

Key Takeaways

•An eval platform is far more than a test runner — it requires shared definitions of quality, reliable data pipelines, versioning, labeling workflows, and cross-team trust in results.
•Start with a spreadsheet and for-loop, but recognize this is documentation, not experimentation; plan to graduate to a dedicated platform before scale demands it.
•Connecting offline evals to production observability (the flywheel) is the single most impactful maturity step: real production traces reveal failure modes that synthetic datasets miss.
•Agent spans can reach 10–20 MB (versus a few kilobytes for traditional application spans), making off-the-shelf databases a performance bottleneck if not architected for LLM-specific data shapes.
•Eval platforms need two distinct query patterns: ultra-low-latency writes/reads for real-time observability and aggregate analytics queries — these have conflicting storage optimizations.
•Full-text search across millions of LLM traces is a hard requirement that most traditional analytics stacks (including data warehouses and DuckDB) cannot fulfill efficiently.
•Building a nice eval UI is tractable; building the correct data layer underneath it is the genuinely hard, novel systems engineering problem.
•Headless eval use cases are rising: teams increasingly want to plug coding agents (Codex, Claude Code) directly into the eval SQL interface for automated quality improvement loops.
•Non-technical SMEs and product stakeholders should be first-class users of eval platforms — playground features with system-prompt comparison UIs are essential to bring them in.
•Topic modeling over production traces helps surface unknown unknowns — what users are actually doing with an agent — so engineering effort targets real failure modes.

Transcript

All right, it's 11:15. We're going to go ahead and get started. Before we do, everyone say evals. Evals. I was telling my colleague Rose, who is at at the door that I was a adjunct professor for a number of years. And um the first year that I did it, I thought I was going to have this full class of 130 people every single week eager to learn. And then as the weeks went on, 130 became 60, became 30, became 10. So I always tell myself that whenever I give a talk that only about four or five people...