Eval & Observability Practitioner Brief | AI Engineer Europe 2026 — ConferenceDigest
Eval & Observability Practitioner Brief
For: ML, evaluation, and observability practitioners measuring agent quality
Eval & Observability Practitioner Brief
AI Engineer Europe 2026 — Intelligence Report
*For: ML, Evaluation, and Observability Practitioners Measuring Agent Quality*
Executive Summary
AI Engineer Europe 2026 surfaced a decisive signal for eval and observability practitioners: the evaluation problem has fundamentally changed shape. Where 2024 focused on building evaluation infrastructure, 2026 is dominated by the realization that traditional eval paradigms — golden datasets, offline batch runs, pass/fail unit-test-style checks — are structurally insufficient for production agents that are non-deterministic, multi-step, and increasingly running for hours or days. The conference was saturated with concrete evidence of this gap: Ioana Ghita (Accesa) documented a manufacturing agent that correctly identified an anomaly but triggered a strategically disastrous decision worth hundreds of millions of euros; Lawrence Jones (Incident.io) described an AI SRE requiring one hour of manual investigation to evaluate a single root cause analysis; and Phil Hetzel (Braintrust) cataloged the database-scale challenges of storing gigabyte-sized agent traces with full-text search requirements that no standard OLAP tool can satisfy.
The most technically consequential finding from the conference is the emergence of a multi-layer observability stack. Practitioners are converging on a pattern that combines explicit signals (tool error rate, latency, cost, user regeneration events) with implicit signals that require semantic interpretation: classifier-based signals for refusals, task failure, and user frustration; regex-based keyword matching for cheap real-time alerting; and a novel "self-diagnostics" primitive where agents are instructed to report their own failure modes via a dedicated tool. Danny Gollapalli and Zubin Koticha (Raindrop) demonstrated this architecture live, showing a 37%-to-9% drop in user frustration after deploying prompt v2.4, measured through production A/B experiments on classifier signals — not a pre-production golden dataset. The practical implication is that practitioners need two linked systems: an offline experimentation loop anchored to production traces, and a real-time monitoring layer that captures semantic quality signals at inference time.
Benchmark integrity itself was a major theme. Nicholas Kang and Michael Aaron (Google DeepMind / Kaggle) opened with a stark demonstration: on SWE-Bench Pro, six frontier models cluster within a couple of percentage points of each other, yet the harness they run inside shifts measured performance by 22 percentage points. Peter Gostev (Arena.ai) added a complementary data point from 5.5 million human preference votes: even by Q1 2026, approximately 9% of all model responses are rated as unsatisfactory by two independent judges simultaneously — a "dissatisfaction floor" that benchmark hillclimbing cannot close because narrow evals concentrate improvement at the top of the distribution while harder, judgment-intensive tasks improve more slowly. Samuel Colvin (Pydantic) provided the most concrete optimization result: GEPA (Genetic Evolution of Prompts via Agents) lifted a manually engineered prompt from 92% to 96.7% accuracy on a 65-case golden dataset, but only after 400 evaluation calls, demonstrating the computational cost of eval-driven optimization at scale and the overfitting risk when test distributions are narrow.
For practitioners, the strategic posture emerging from the conference is a shift from "build the eval system" to "close the eval flywheel." The highest-leverage investment is not in more sophisticated graders or larger golden datasets, but in the instrumentation that connects production failures to offline experimentation — and the tooling that allows agents themselves to participate in identifying their own failure modes. Every failing production trace is a test case; every agent complaint is a bug report. The teams shipping the most reliable AI products at AIE Europe 2026 are the ones treating observability and evaluation as the same system, differentiated only by whether inputs are known in advance.
Key Findings
Finding 1: The Harness Confound Is Larger Than the Model Difference
Nicholas Kang and Michael Aaron (Kaggle / Google DeepMind, "Agentic Evaluations at Scale, For Everybody") opened with the single most important benchmark integrity finding at the conference: on SWE-Bench Pro, the harness configuration shifts measured performance by 22 percentage points, while six frontier models cluster within a couple of percentage points of each other. A competing lab reran a Kaggle-hosted benchmark with their own API compaction settings and published substantially better results — neither number was fraudulent; both were uninformative.
This harness-versus-model ambiguity has direct practical consequences for practitioners who rely on public leaderboards to make model selection decisions. Bertrand Charpentier (Pruna AI, "20 days of compute vs 7 hours") added a complementary finding from image generation leaderboards: three major platforms (LM Arena, Design Arena, Artificial Analysis) produce different rankings for the same models, and the top-ranked model on any aggregate leaderboard still loses roughly 40% of head-to-head battles — meaning it is the wrong choice for nearly half of real-world use cases. Task-specific sub-leaderboards show completely different leaders per category.
Kaggle's operational response is the Standardized Agent Exams (SAE) — a one-line prompt submission that returns a leaderboard score, attracting 500+ submissions in its first week without active promotion. Their Game Arena PvP benchmark required 400,000 poker hands to achieve statistical significance, illustrating the compute cost of reliable agent evaluation. Distinct personality differences emerged (Grok goes all-in frequently; newer models are more risk-averse) that aggregate scores completely obscure.
Implication for practitioners: Treat any public benchmark number as a harness artifact until the exact evaluation configuration is disclosed. Internal benchmark reproducibility — locking harness versions, compaction settings, and tool definitions — is as important as the evaluation metric itself.
Finding 2: Production Traces Beat Golden Datasets as Ground Truth
Phil Hetzel (Braintrust) made this argument across three separate sessions at the conference ("Does GenAI Belong to Data Scientists?", "The Maturity Phases of Running Evals", "Why Building Eval Platforms Is Hard"), and it was independently validated by practitioners from Incident.io, Raindrop, Lovable, and Granola. The core claim: golden datasets constructed pre-production miss the long tail of real user behavior that causes the most consequential failures.
SallyAnn DeLucia (Arize AI, "AIE Singapore Day 2") distilled the lesson from building Alyx: production traces — not hand-authored golden sets — are the best eval ground truth. Lawrence Jones (Incident.io, "Fighting AI with AI") operationalized this with a downloadable file system pattern: every AI investigation is packaged as a downloadable file system snapshot with full trace data rendered as ASCII-friendly text, dropped into a Claude Code session so the agent can traverse the prompt hierarchy to identify exactly which prompt to modify. His team runs 25 parallel Claude Code agents analyzing one investigation each from a corpus of production failures, then clusters results to surface systemic failure patterns — turning a one-hour manual investigation into a minutes-long automated analysis.
Benjamin Verbeek (Lovable, "How Lovable Self-Improves Every Hour") demonstrated the most complete version of this flywheel: an LLM judge monitors sessions for stuck-user signals (repeated requests, session abandonment), flags the transition from stuck-to-unstuck as a high-signal training sample, asks "what context should have been injected at the start of this query?", and stores the answer as a knowledge entry. A holdout mechanism injects blank context for a small sample of matching queries, enabling a controlled A/B test of whether each entry actually improves project completion rates. Stale entries are pruned based on ongoing eval data. Early data shows measurable reduction in stuck-session messages and increased project deployments.
The data engineering implications are severe. Hetzel disclosed that Braintrust custom-built its database from scratch after ClickHouse failed on full-text search across unstructured trace data. A single agent span can reach 20 MB; a complete trace can exceed 1 GB. The database incorporates a write-ahead log for real-time visibility, analytical indexes for aggregation, and a forked Tantivy (Rust-based full-text search) for phrase and keyword search across all traces.
Implication for practitioners: Architect eval systems to ingest production traces from day one, not as a phase-two feature. The data engineering investment — especially full-text search across large semi-structured payloads — is larger than most teams anticipate and must be scoped early.
Finding 3: Implicit Observability Signals Outperform Explicit Metrics for Quality
Danny Gollapalli and Zubin Koticha (Raindrop, "Everything You Need to Know About Agent Observability") presented the most operationally detailed observability architecture at the conference, distinguishing two classes of signals with materially different diagnostic value.
Explicit signals — tool error rate, latency, cost, user regeneration events — are objective and measurable, but they capture mechanical failures, not quality failures. Ioana Ghita (Accesa, "Architecturally Ready, Behaviorally Unvalidated") illustrated this gap starkly: an interview agent hallucinated a non-existent action, the orchestrator caught the exception and executed a fallback, and the HTTP response returned 200 with normal latency. Infrastructure dashboards showed zero errors while a candidate was trapped in a conversational loop for over 40 minutes. "Silent failures masked by infrastructure-level observability" is the term Ghita uses for this entire failure class.
Implicit signals have three sub-categories in the Raindrop framework:
- Classifier signals: Binary ML classifiers trained for refusals, task failure, user frustration, jailbreak attempts, and positive wins. A real customer case shows user frustration falling from 37% to 9% after prompt v2.4, measurable with a few hundred events.
- Regex signals: Keyword matching for sentiment indicators, inspired by the leaked Claude Code `user_prompt_keywords.ts` file that the Claude Code team used as a cheap frustration detector.
- Self-diagnostics: A single tool definition plus a system prompt instruction encouraging the agent to report failure modes before returning a final answer. During a live demo, a coding agent that hit permission errors bypassed them via bash heredoc syntax and then self-reported the workaround. Key implementation insight: framing the tool as giving "feedback to its creators" significantly increases firing rate compared to naming it something defensive like `unsafe_bash_use`, because models resist self-incrimination.
The Granola case study (Mehedi Hassan, "Feedback Loops Are All You Need") confirmed the value of custom observability tooling over generic SaaS: they built an internal tracer capturing the full agent loop — tool calls, reasoning traces, search trails, per-request cost — specifically shaped to enable non-engineers to use it. The Granola founder traces agent runs front-to-back to diagnose failures directly.
Implication for practitioners: Instrument for semantic quality signals from launch. Tool error rate and latency are table stakes; frustration classifiers, self-diagnostic tools, and regeneration rate are the signals that surface what users actually experience.
Finding 4: LLM-as-Judge Requires Calibration Against Human Labels
LLM-as-judge (LLM-as-a-judge) featured in nearly every eval discussion at the conference, but was consistently presented with caveats that pure practitioners may underweight. Phil Hetzel (Braintrust) was the most systematic: "just because you put a robe and cloak on an LLM does not make it trustworthy by default." His prescription is to evaluate the judge itself against a human-labeled ground truth dataset — since judge outputs are discrete, this ground truth set is straightforward to construct and calibrate.
Peter Gostev (Arena.ai, "What Do Models Still Suck At?") contributed a concrete failure mode for LLM-as-judge: his BullshitBench (155 nonsense questions designed to test whether models push back on invalid premises) was graded by an LLM-as-judge pipeline he manually validated. The results reveal a model behavior invisible to standard benchmarks: Claude models push back on nonsense well above 50% of the time; GPT and Gemini accept nonsense roughly 50% of the time. Enabling extended thinking makes this worse — GPT-5.4 traces show the model noting a premise is questionable in one line, then spending 20 paragraphs attempting to solve the non-problem anyway. A judge that doesn't test for sycophancy cannot detect this.
Samuel Colvin (Pydantic, "Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops") showed GEPA's dependency on eval quality: after running GEPA with 400 evaluation calls on a 65-case golden dataset (generated with Claude Opus 4.6 and manually verified), the optimizer achieved 96.7% accuracy — but demonstrated a concrete overfitting case when it systematically excluded aunts and uncles from political dynasty classification because they were absent from the training split. The lesson: genetic prompt optimization amplifies whatever biases exist in the eval distribution.
Laurie Voss (Arize AI, "AIE Miami Day 2") ran 500 structured evaluations comparing MCP server vs. CLI tool integration using Claude Opus 4.6 via the Claude Agent SDK. The methodology — graded correctness scoring with LLM-as-judge, stratified by task type — produced a clear finding: MCP used 12 tool calls versus 5 per task on complex queries and incurred 6x higher cost. The eval methodology itself became the finding, with Voss noting that verbose JSON payloads overflowed context windows in ways that only showed up through systematic measurement.
Implication for practitioners: Treat the judge as a model requiring its own evaluation. Calibrate LLM judges against human labels on a representative sample before using them at scale. Pay particular attention to sycophancy failure modes — the judge accepting nonsense because the underlying model was trained to complete tasks, not refuse them.
Finding 5: Behavioral Validation Is Categorically Different from Unit Testing
Ioana Ghita (Accesa, "Architecturally Ready, Behaviorally Unvalidated") delivered the most operationally grounded argument for behavioral validation at the conference, backed by concrete production failures. Her central distinction: "testing" runs scenarios you designed; "behavioral validation" explores what you did not think of. A communication agent passed hundreds of test conversations and failed in production because real users sent long, multilingual, emoji-laden messages that combined edge cases only evaluated in isolation. A document processing agent collapsed when non-technical users interrupted workflows — scenarios never modeled during MVP design.
Ghita defines a new failure class: "unmodeled semantic and decision states" — situations where the agent's output is technically correct and monitoring shows no errors, but the business decision taken is wrong because the agent lacked a fallback for a valid-but-unmodeled state. This failure class is invisible to technical monitoring and only surfaces through human feedback or domain expert review.
Her engineering prescriptions for practitioners:
- Log agent reasoning, not just agent actions. Capture what the agent observed, what plan it formed, and why it chose a specific action, with end-to-end trace IDs.
- Maintain separate instruction space, data space, and forbidden-action space for every agent. Blurring these creates prompt injection vulnerabilities from external inputs.
- Every added agent exponentially increases failure combinations. Agents should only be added when simpler workflows are insufficient.
- Get agents in front of 10 real users as early as possible — this yields more behavioral insight than 500 automated tests.
The "amnesiac with permissions" phrase Ghita introduced — an LLM agent without an external state store that loses intermediate decisions while retaining the ability to act — captures a failure mode that becomes catastrophic in multi-step workflows. External state stores and decision logs are not optional for production agents in high-stakes domains.
Implication for practitioners: Behavioral evals must be designed by domain experts, not just engineers. The failure modes that matter most are the ones your engineering team cannot imagine in advance.
Finding 6: Eval-Driven Prompt Optimization Is Operationally Viable — With Caveats
Samuel Colvin (Pydantic, "Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops") provided the most technically detailed end-to-end eval-driven optimization pipeline at the conference. GEPA (Genetic Evolution of Prompts via Agents) is a genetic algorithm operating on prompt strings: it identifies a Pareto frontier of best-performing candidates, cross-breeds them, and iterates. Colvin's implementation used a second Pydantic AI agent (GPT-4.1 for speed) as the proposer, receiving Logfire-formatted context about failures and proposing new system prompt candidates. The result: 96.7% accuracy from a starting point of 87% (one-line prompt) and 92% (expert-engineered prompt), using 400 evaluation calls.
The operational constraints Colvin disclosed are as important as the headline number:
- Prompts are model-specific and must be re-optimized when the underlying model changes.
- GEPA can overfit narrow test distributions — the optimizer systematically excluded entire relationship categories absent from the training split.
- Variance reduction requires running each eval case many times; Colvin cites hedge funds spending $20,000 per night on evals.
- The technique is most valuable for private-data domains where models have no prior training signal (internal financial documents, legal databases, medical records).
- Shopify reduced annual model costs from roughly $5 million to $73,000 by combining GEPA-style prompt optimization with a smaller quantized model.
Logfire's managed variables feature extends this to production: A/B traffic splitting to roll out prompt or model changes incrementally, with an OpenFeature open standard backing. The roadmap toward "self-driving" managed variables — using evals and GEPA in a closed loop to autonomously hill-climb toward optimal configuration — is the logical endpoint of this architecture.
Implication for practitioners: GEPA-style optimization is ready for production use on well-defined tasks with stable evaluation criteria. The investment is in eval infrastructure quality, not in the optimization algorithm itself. Budget for ongoing re-optimization as models change.
Finding 7: Agentic Evaluation Infrastructure Requires New Data Primitives
The conference surfaced a consistent pattern: teams building serious eval infrastructure are building custom data systems, not assembling existing tools. Phil Hetzel (Braintrust) disclosed building a custom database after ClickHouse failed on full-text search. Danny Gollapalli (Raindrop) built a trajectory visualizer showing tool call order and errors, with data export to BigQuery and Snowflake. Lawrence Jones (Incident.io) built a CLI called "eval tool" that exposes eval suite operations (list, add, edit, replace test cases) to coding agents, because large YAML eval files caused context overflow. Pedro Rodrigues (Supabase, "Combine Skills and MCP to Close the Context Gap") used Braintrust as the eval harness for structured evaluations across six real Supabase project scenarios, testing four agent configurations across three conditions (baseline, MCP only, MCP plus skill) — and found the skill condition outperformed baseline across all four model/vendor combinations.
The scale challenges are non-trivial. The Raindrop self-diagnostics demo was inspired by OpenAI's December 2024 blog post on training models to self-report misalignment — a research finding now productized into a tool definition pattern that any team can implement. Google DeepMind (KP Sawhney, "How Google DeepMind Runs Agents at Scale") maintains a separate agent trajectory store specifically for coding agents, enabling engineers to diagnose exactly where looping or off-rails behavior began. Their skills library uses a Darwinian quality filtering process — thousands of engineers contribute, but only strongest survive — to prevent the skill library sprawl that degrades agent quality at scale.
Stage three of Hetzel's eval maturity model highlights the hardest problem in agentic eval: CRUD tool calls and external system state. Replaying a trace offline requires either recreating the exact state of all external systems at the time of the original trace, or using timestamp-based version queries against a vector database to retrieve historical state. Both approaches are expensive; the first is often infeasible. Hetzel's practical workaround: inject the full captured system state directly into the trace itself as context, trading storage cost for reproducibility.
Implication for practitioners: Treat eval infrastructure as a first-class engineering investment, not a configuration exercise. Plan for custom storage with full-text search, trajectory visualization, and production-to-offline state replay from the start of your platform build.
Finding 8: Agent Self-Improvement Loops Are Entering Production
Several teams demonstrated closed-loop systems where agents identify and report their own failure modes, triggering automated improvement cycles without human intervention. Benjamin Verbeek (Lovable, "How Lovable Self-Improves Every Hour") described the vent tool architecture in detail: the agent files complaints via `vent_send_feedback` only when platform limitations are actively degrading its work. In the first hour, the agent filed 20 complaints about a silent file-copy failure for filenames containing spaces — a real production bug previously invisible in logs. Vent volume also became a reliable incident detector: spikes in vent activity correlated directly with platform outages. A second agent monitors the vent channel, deduplicates reports, and automatically opens pull requests for engineer review.
Anant Dole and Asbjorn Steinskog (Play Magnus / Take Take Take, "Building a Chess Coach") demonstrated Claude Code Channels — a research-preview MCP feature that injects external events into a running Claude Code session. When a user flags bad commentary in the app, a "commentary triage" skill runs a structured investigation: re-generates commentary, modifies prompts or detectors, creates new detectors if needed, and posts clarifying questions back to Slack. The engineer can merge the resulting PR from a phone.
Incident.io's fleet-scale analysis pipeline (Lawrence Jones) runs daily backtests across hundreds of customer accounts. 25 Claude Code agents analyze one investigation each from the production corpus; a clustering stage aggregates per-investigation analyses to surface systemic failure patterns rather than individual case failures. The output is an actionable report driving specific code changes and pull requests — turning a one-hour per-investigation manual process into a minutes-long automated pipeline.
Implication for practitioners: Self-reporting and self-improvement primitives are production-ready. The vent tool pattern and self-diagnostics tool definition require minimal implementation effort and provide failure signal unavailable from any monitoring system that observes only from the outside.
Strategic Implications
The golden dataset era is ending. The conference was unanimous: pre-production golden datasets cannot cover the behavioral surface area of production agents. The eval system must be designed to continuously ingest production failures as new test cases, with the tooling to close that loop in near-real-time. Teams that built eval infrastructure in isolation from observability infrastructure will need to integrate them or rebuild.
Benchmark selection determines what you optimize for. With harness configurations shifting SWE-Bench Pro scores by 22 points, and aggregate leaderboards hiding 40% win-loss rates at the task level, practitioners who trust public benchmarks without replicating the harness configuration are making model selection decisions on noise. Internal benchmarks anchored to production traces and domain-specific success criteria are the only defensible baseline.
The LLM judge needs a judge. LLM-as-judge is the practical workaround for non-deterministic outputs, but the conference provided concrete evidence that judges sycophantically validate outputs that they should reject, fail on edge cases absent from calibration sets, and vary systematically across model families. Calibrating judges against human labels on a representative sample is not optional; it is the first step in deploying any judge at production scale.
Semantic observability is the new SLA. Infrastructure metrics — error rate, latency, cost — cannot detect the class of failures that matter most in agentic systems: the agent that confidently executes a wrong plan, the one that loops a user for 40 minutes while returning HTTP 200, the one that approves a strategically wrong decision because no fallback was modeled. Implicit semantic signals — frustration classifiers, self-diagnostic tools, domain-expert annotations — are the signals that actually correlate with business outcomes.
Eval infrastructure is a platform investment, not a project. The data engineering requirements for agent observability — gigabyte traces, full-text search, dual read patterns for real-time and analytics, trajectory replay with external system state — are categorically different from standard application monitoring. Teams that try to solve this with existing tools (ClickHouse, Prometheus, Datadog) will hit walls. Budget accordingly and invest in custom storage as early as the first production agent deployment.
Self-improvement loops compound. The teams with the highest-quality agents at AIE Europe 2026 are not the teams with the best evaluation frameworks — they are the teams whose agents are actively participating in finding and reporting their own failure modes. The vent tool, self-diagnostic tool definitions, and production trace flywheel are not advanced features; they are the baseline for maintaining quality as agent complexity grows.
Action Items
Priority 1 — Immediate (0–30 days)
Instrument a self-diagnostic tool in every production agent. Add a single tool definition (`report_feedback` or equivalent) with a system prompt instruction to call it before returning a final answer when platform limitations or capability gaps are degrading quality. Use a name that frames it as feedback to creators, not self-incrimination. Route output to a dedicated Slack channel with deduplication. Estimated effort: 1 day.
Add user frustration classification to your observability stack. Train or prompt a lightweight binary classifier on your production traces to detect user frustration signals (repeated requests, explicit complaints, regeneration events). Establish a baseline frustration rate before your next prompt or model change. This is the signal that actually correlates with product quality.
Audit your current benchmark methodology for harness confounds. For every public benchmark number you use to justify model selection: identify the harness configuration, compaction settings, and tool definitions used. If they differ from your production harness, treat the number as directionally useful at best. Run your top two models on your own eval suite with your own harness before making deployment decisions.
Establish end-to-end trace IDs across all agent actions. Ghita's prescription: every agent decision must be reconstructable after the fact. Implement a trace ID that propagates through all tool calls, sub-agents, and external system interactions for every production session. This is the prerequisite for all downstream failure analysis.
Priority 2 — Near-Term (30–90 days)
Build the production-to-offline eval flywheel. Instrument your production traces to flow into your offline eval dataset continuously. Start with a simple pipeline: capture traces flagged by frustration classifier or user regeneration events, route to a review queue, annotate with justification (not just thumbs up/down), and add passing cases to your eval suite. Phil Hetzel's four-stage maturity model suggests starting simple — even a spreadsheet-and-for-loop is better than nothing — and evolving toward the playground + production tracing integration.
Implement A/B testing for prompt and model changes using semantic signals. Before your next system prompt change, instrument an experiment that routes X% of traffic to the new prompt while measuring frustration classifier rate, task completion rate, and self-diagnostic tool firing rate. Statistical significance is achievable with a few hundred events per variant. Raindrop demonstrated this produces actionable results in days, not weeks.
Calibrate your LLM judges against human labels. Identify the 3–5 most important quality dimensions your judge evaluates. Collect 50–100 human-labeled examples with written justifications (not just scores). Measure judge-human agreement using precision/recall. Document the failure modes where the judge consistently disagrees with human raters — these are the blind spots in your quality measurement system.
Implement behavioral validation sessions before major releases. Following Ghita's prescription: assign 10 real users (not engineers) to use the agent on realistic tasks without scripted scenarios. Document every unexpected behavior, loop, or failure mode. Convert each one to an eval case. This is how you build coverage for "unmodeled semantic states" that automated testing cannot discover.
Priority 3 — Strategic (90+ days)
Plan eval storage infrastructure for agentic scale. If you are storing agent traces in a standard logging system (Datadog, CloudWatch, ClickHouse), prototype the full-text search and trajectory visualization requirements before you hit scale limits. Budget for a purpose-built store with write-ahead log, analytical indexes, and full-text search. Braintrust's architecture (forked Tantivy on Rust) is one reference implementation; others are emerging.
Build a fleet-scale analysis pipeline for systemic failure patterns. Once you have enough production traces, implement Lawrence Jones's pattern: parallel subagents each analyze one trace from a corpus, a clustering stage surfaces common failure patterns, and the output drives specific code changes. This converts the "why did accuracy drop this week" question from a one-hour manual investigation to a minutes-long automated analysis.
Sessions to Watch
"Everything You Need to Know About Agent Observability" — Danny Gollapalli & Zubin Koticha, Raindrop. The most complete production observability framework at the conference: explicit signals, three implicit signal types, self-diagnostic tool architecture, A/B experiment methodology, and a live demo showing 37%-to-9% frustration reduction. Essential viewing for anyone building a monitoring stack.
"The Maturity Phases of Running Evals" — Phil Hetzel, Braintrust. A four-stage practical framework from vibe-checking through LLM-as-judge to production flywheel, with concrete guidance on CRUD tool call replay, timestamp-based vector database queries for historical state, and CI/CD eval automation. Complements the broader "Why Building Eval Platforms Is Hard" session.
"Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops" — Samuel Colvin, Pydantic. The most end-to-end eval-driven optimization pipeline at the conference: 65-case golden dataset, GEPA genetic optimization from 87% to 96.7% accuracy, Logfire managed variables for live A/B prompt testing, and concrete disclosure of cost ($20k/night eval runs at hedge funds) and overfitting risks.
"Agentic Evaluations at Scale, For Everybody" — Nicholas Kang & Michael Aaron, Google DeepMind/Kaggle. The definitive benchmark integrity session: 22-point harness confound on SWE-Bench Pro, 400,000 poker hands for statistical significance in Game Arena, and four concrete infrastructure products (SAE, Game Arena, Benchmarks Platform, Hackathons) addressing the democratization of eval creation.
"Architecturally Ready, Behaviorally Unvalidated" — Ioana Ghita, Accesa. Concrete production failure cases — silent failures masked by infrastructure metrics, unmodeled semantic states triggering hundred-million-euro decisions — with engineering prescriptions for reasoning logging, instruction/data/forbidden-action space separation, and external state stores. Essential for teams working in regulated or high-stakes domains.
"Fighting AI with AI" — Lawrence Jones, Incident.io. Three patterns for using agents to debug agent failures: eval YAML CLI tooling, downloadable file system debugging views, and fleet-scale parallel subagent analysis with cohort clustering. The most operationally specific session on eval tooling design for engineering teams.
"What Do Models Still Suck At?" — Peter Gostev, Arena.ai. BullshitBench reveals that extended thinking modes make sycophancy worse, not better; 5.5 million human preference votes show a persistent 9% dissatisfaction floor; and expert-category dissatisfaction rates reveal that gaming, finance, and law improve far more slowly than quantitative tasks. Critical for calibrating expectations about what current eval frameworks measure.
"How Lovable Self-Improves Every Hour" — Benjamin Verbeek, Lovable. The most mature production self-improvement loop at the conference: LLM judge for stuck-user detection, holdout A/B testing for context injection quality, vent tool for agent-to-engineer complaint routing with automated PR generation, and incident detection via vent volume spikes. A complete reference architecture for autonomous quality improvement.