How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust

Phil Hetzel, Head of Solutions Engineering at Braintrust, presents a structured argument for why agent observability is a fundamentally different systems problem from traditional observability — not just a more complex version of the same thing. The talk is framed around three core problems that make agent o11y distinct.

Hetzel opens by characterizing traditional observability as answering a single question: is the system up? Tools like Grafana and Datadog are well-suited for tracking latency, error counts (400/500-level), uptime, and technical performance metrics. Braintrust itself uses Datadog for exactly this purpose. The issue, he argues, is that this framing is categorically insufficient for agentic applications.

Problem one is non-determinism. Traditional applications follow known, deterministic control flows. Agent applications are inherently non-deterministic due to LLM variability — the same input can produce different reasoning paths and tool-call sequences. This means observability must capture qualitative signals: Was the response grounded in the retrieved context? Were the expected tools invoked? Is the output aligned with the brand standard defined in the system prompt? These questions cannot be answered by traditional metrics alone.

Problem two is data scale and shape. Agent traces are described as "really nasty" along several dimensions. A single trace can exceed one gigabyte; an individual span can reach 20 megabytes. The data is semi-structured with large volumes of embedded unstructured text, yet must be ingested and surfaced in real time. Hetzel explains that Braintrust built a custom database from scratch to handle this, noting that prior to this build they had used ClickHouse but migrated away because it lacked robust full-text indexing capabilities. The custom database incorporates: a write-ahead log for instantaneous trace visibility, analytical indexes for fast filtering and aggregation queries, and a forked version of Tantivy — an open-source Rust-based full-text search library analogous to Apache Lucene — enabling engineers to query every trace containing a specific word or phrase. The unified query interface is SQL or a SQL-similar language. Hetzel credits Braintrust's founder, a former early employee at SingleStore, as the driving force behind the decision to build a proprietary storage engine.

Problem three is the persona problem. Traditional observability is operated by systems engineers and platform engineers. Agent observability, done well, requires domain experts: clinicians grading whether a medical agent responded correctly, wealth advisors reviewing financial guidance outputs, lawyers evaluating legal reasoning. These non-technical users can now interact with traces directly, writing natural-language annotations that justify their quality scores. Hetzel explains that these human annotations become the training signal for automated scoring functions — the human catches failure modes, the scoring function scales the detection of those failure modes across all production traffic.

On the topic of evals, Hetzel frames observability and evaluation as the same underlying system, differentiated only by whether inputs are known in advance (evals, run in batch) or unknown (observability, run in real time). Braintrust recently rolled out an LLM-powered clustering feature that runs lightweight embeddings over incoming traces to perform topic modeling, intent classification, and sentiment analysis — with the goal of compressing the feedback loop between a production failure and an experimental fix.

The talk closes with audience Q&A covering integration with offline optimization pipelines, the decision to build a custom database versus using OLAP solutions, and the relationship between functional quality observability and technical performance observability.