Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop

Danny Gollapalli and Zubin Koticha, co-founder/CEO and back-end engineer at Raindrop, deliver a practical workshop on agent observability for AI Engineer Europe 2026. The session argues that agent failures are fundamentally different from traditional software failures: agents are non-deterministic, unbounded, multi-tool, multi-step, and increasingly deployed in high-stakes domains including healthcare, finance, and the military — making monitoring a critical unsolved problem.

The presenters contend that the conventional evals paradigm — golden datasets with fixed input/output pairs — is insufficient for production agents. As agents call exponentially growing tool sets, access multiple memory sources, and orchestrate their own sub-agents recursively, the combinatorial input space makes exhaustive testing impossible. The Raindrop team positions production monitoring as the successor to eval-centric thinking, calling agent observability "humanity's last problem" before agents fully outpace human oversight.

The framework is organized around two signal types. Explicit signals are objective and measurable: tool error rate, latency, cost, and user regeneration events. Spikes in any of these serve as immediate health indicators. Implicit signals capture semantic meaning and are described as more valuable but harder to detect. These are further broken into three categories: (1) classifier signals — binary ML classifiers trained for refusals, task failure, user frustration, content moderation, jailbreaking, and positive wins; (2) regex signals — keyword matching for sentiment indicators, inspired by the leaked Claude Code source file `user_prompt_keywords.ts` that the Claude Code team used as a cheap frustration detector; and (3) self-diagnostics — a novel technique that exploits models' capacity for self-introspection.

The self-diagnostics section, demonstrated live by Gollapalli, is inspired by OpenAI's December 2024 blog post on training models to self-report misalignment. The implementation requires only a single tool definition (a "report" tool) and a system prompt instruction encouraging the agent to call it before returning a final answer. During the live demo, the write tool was intentionally broken to throw permission errors; the coding agent bypassed this via bash heredoc syntax and then self-reported the workaround. Key implementation insight: framing the tool as giving "feedback to its creators" significantly increases firing rate compared to naming it something like "unsafe_bash_use", because models are trained to appear polished and resist self-incrimination. Self-diagnostics surfaces tool failures, user frustration, capability gaps (e.g., a user requesting an alert feature the agent lacks — functioning as a built-in feature request signal), and self-correction behaviors that may carry security implications.

The session also covers experiments/A/B testing: shipping a prompt or model change to a percentage of users and measuring whether classifier signal rates improve. An example showed user frustration dropping from 37% to 9% after deploying prompt v2.4. Statistical significance is achievable with a few hundred events, and short multi-day runs can be skipped when sample sizes are sufficient to detect meaningful regressions.

Raindrop's additional features discussed in Q&A include a triage agent that automatically investigates signal spikes by scanning all traces, a "deep search" feature for creating new binary classifiers via natural language, a trajectory visualizer showing tool call order and errors, data export to BigQuery and Snowflake, and integrations with Statsig for experiment analysis. A real customer case was cited: Raindrop automatically created an issue when a database provider failure caused a sudden spike in user frustration, enabling rapid triage without human intervention.