Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

Laurie Voss, Head of Developer Experience at Arize AI (and co-founder of npm), delivers a hands-on two-hour workshop on building a complete evaluation pipeline for agentic applications. The session centers on what Voss calls the "vibes problem": most teams test agents by running a few queries and eyeballing the output, which does not catch regressions, does not run in CI, and breaks silently when prompt changes affect unrelated behaviors. The workshop builds a financial analysis agent using the Claude Haiku model (deliberately chosen for being "reliably dumb" to generate interesting failures) and the Claude Agent SDK, then progressively layers evals on top of it using Arize Phoenix, an open-source AI observability platform.

The workshop begins with tracing fundamentals. Phoenix captures every LLM call, tool call, and agent step as spans in an OpenTelemetry-compatible format extended by the Open Inference standard. Instrumentation requires only two lines of code: importing `phoenix.otel` and calling `register(auto_instrument=True)`. The agent itself is a two-turn multi-agent system: one sub-agent performs web searches for a given stock ticker, and a second sub-agent writes a concise financial report from the research output.

Before writing a single eval, Voss emphasizes reading the traces. In the workshop's twelve test runs (Tesla, Apple, Nvidia, Rivian, Coca-Cola, Apple vs. Microsoft, and others), he surfaces three systemic failure categories: (1) the agent trying to write output to disk because the prompt said "write a report" without specifying the destination; (2) the Apple report covering only AWS, not all of Amazon; and (3) the Rivian report including suspiciously specific 2026 delivery figures (62,000–67,000 vehicles) that the model likely hallucinated.

The workshop implements three eval types. First, a deterministic code eval using regex checks whether the expected stock ticker appears in the output — passing 11 of 13 traces, with Tesla and Amazon failing. Second, a built-in LLM-as-a-judge correctness eval using Claude Sonnet scores 0 out of 13, because the judge's training cutoff is mid-2025 and the agent is producing forward-looking 2026 financial analysis the judge cannot verify. Switching to a faithfulness eval — which checks whether the written report is grounded in the agent's own research output rather than world knowledge — scores 13 out of 13. This result demonstrates the workshop's sharpest lesson: choosing the right eval matters more than tuning it. Third, a custom actionability eval is built from scratch with a five-part rubric: judge role definition, explicit criteria for actionable vs. non-actionable reports, clearly labeled data blocks using XML tags, labeled positive and negative examples, and a constrained binary output ("actionable" or "not actionable") with chain-of-thought reasoning before the label. The initial actionability eval finds roughly half the reports non-actionable because they summarize data without giving a buy/sell/hold recommendation.

The final section covers Phoenix experiments: after updating both agent prompts to require specific financial ratios, news from the last six months, and an explicit buy/sell/hold recommendation, all six previously failing actionability tests pass. Voss frames this as data-driven prompt engineering — every prompt change maps directly to a failure surfaced by an eval explanation.

Additional topics covered include meta-evaluation (comparing LLM judge outputs against human-annotated golden data sets), precision vs. recall trade-offs for classifiers, known LLM judge biases (position bias, length bias, confidence bias, self-preference bias), the Swiss cheese model of layered defenses, and the impact hierarchy for agent improvement: data quality fixes yield the highest ROI, followed by prompt engineering, then model selection, with hyperparameter tuning (temperature, top-P) at the bottom. Voss closes with practical guidance on sample sizes: 12–20 examples give directional signal for workshop experiments, while 200–400 are needed for production shipping decisions at a 5% defect rate threshold.