Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft
Summary
Amy Boyd (Microsoft Foundry Developer Relations Lead) and Nitya Narasimhan delivered a hands-on workshop at AI Engineer Europe 2026 on closing the observability gap in agentic AI systems. The session title draws a deliberate analogy to London Underground signage: just as the gap between a train and the platform varies by station and changes over time, the gap between an agent's actual behavior and its intended requirements widens silently as models change, prompts are tweaked, and edge cases accumulate. The core argument is that developers must evaluate, monitor, and optimize agents continuously throughout their lifecycle, not just at launch.
The workshop used a fictitious Contoso Travel Agent as the hands-on scenario. Attendees were guided through a GitHub repository containing a four-hour workshop compressed into a 90-minute session, delivered as a 'cooking show' with pre-baked code and live demos. The repo uses GitHub Codespaces with a dev container so participants need no local tooling setup, and costs roughly 10 USD for two days of Azure usage.
On the platform side, the presenters focused on Microsoft Foundry (ai.azure.com). Key capabilities demonstrated included: creating a new Foundry project and deploying GPT-4.1 as a default model; adding App Insights for tracing; configuring OpenTelemetry-based traces that can be viewed locally or pushed to Azure Monitor; and selecting built-in evaluators covering quality metrics (coherence, fluency, groundedness, relevance), risk and safety, and agent-specific metrics such as intent resolution, task adherence, and tool call accuracy. A live groundedness failure was shown where a response cited August 2024 instead of 2025, illustrating how trace-linked evaluations let teams pinpoint exactly which step in a multi-agent workflow caused a regression.
The workshop progressed through four notebook labs: (1) environment connectivity check; (2) building a simple prompt agent via the Azure AI SDK; (3) adding function tools for car, flight, and hotel search; and (4) composing a workflow agent using Foundry's declarative YAML workflow capability, which chains specialist sub-agents (flight, hotel, car) under a concierge orchestrator. The workflow agent traces reveal token costs and per-agent performance, enabling surgical optimization.
Red teaming was framed as adversarial evaluation where a second AI attacks the first. The workshop covered agentic risk categories such as prohibited actions, sensitive data leakage, and task adherence violations, and demonstrated attack strategies from simple leetspeak substitution up to crescendo attacks that escalate incrementally to evade guardrails. The distinction between safety evaluations (normal-user behavior) and red teaming (malicious-user behavior) was emphasized using a 'building inspector vs. hired burglar' analogy.
The headline demo was the Foundry 'observe skill', released just two weeks before the conference. Activated via GitHub Copilot Chat with Claude as the underlying coding-agent model, the skill accepts a bare agent endpoint with no eval dataset and autonomously: generates an evaluation dataset from the agent's instructions; runs a baseline batch evaluation; identifies failures (in the demo, two task adherence failures out of a small test set); optimizes the system prompt using a built-in prompt optimizer; deploys a new versioned agent; re-runs evals; and rolls back to the best-performing version if a regression is detected. The skill iterated through ten prompt versions, found version five optimal, and surfaced a clear before/after comparison table. Nitya stressed that the skill's visible reasoning chain is its most valuable feature: it exposes failure modes developers did not know to look for. A model-swap path (e.g., GPT-4.1 to GPT-4 Mini for cost reduction) with automatic regression detection was also highlighted. The observe skill is built on a Foundry MCP server and uses trigger-phrase routing across subskills including dataset creation, insight analysis, and prompt optimization.
Key Takeaways
- •Instrument tracing from day one: connect App Insights to your Foundry agent before any evaluation work so traces and evals are linked and co-navigable.
- •Use trace-linked evaluations to shorten the detect-to-diagnose loop: seeing which trace step caused an eval failure eliminates guesswork when a model or prompt changes.
- •Evaluate at every node of the agent workflow, not just the final response: intent resolution, tool call correctness, and task adherence can fail independently.
- •Treat task adherence as the metric most likely to need ongoing tuning; it is the one most sensitive to prompt and model changes.
- •Red teaming is distinct from safety evaluation: safety evals cover normal users while red teaming simulates adversarial manipulation; both are required for production agents.
- •The Foundry observe skill can bootstrap observability from zero: give it only an agent endpoint and it generates the eval dataset, runs batch evals, and optimizes the prompt automatically.
- •Always version agent deployments so rollback is a simple identifier swap; the observe skill automates this and reverts to the highest-scoring version.
- •Model swaps (e.g., GPT-4.1 to GPT-4 Mini) should be paired with an immediate eval run to detect quality regressions before the cheaper model reaches production.
- •Multi-agent workflow traces expose per-agent token costs and accuracy, enabling targeted optimization of only the underperforming sub-agent.
- •Use Azure Monitor to correlate AI telemetry with dependent service telemetry (e.g., Azure AI Search) so performance root causes are visible across the stack.
Transcript
Hi everyone. Welcome. Thank you for joining us this morning. Um my name's Amy Boyd. I lead our Foundry developer relations team at Microsoft. I'm joined by my colleague Nitya on the team who is an absolute expert in observability. And today we thought we'd have a nice catchy title uh given where we are in London of mind the gap in your agent observability. We'll talk a little bit through what we actually mean by mind the gap cuz it's got a deeper reason than just the nice sort of uh title there....
Related Sessions
