Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

Samuel Colvin, creator of Pydantic and founder of Pydantic the company, leads a hands-on workshop at AI Engineer Europe 2026 covering two core techniques for improving production AI agents without redeployment: GEPA (Genetic Evolution of Prompts via Agents) optimization and Logfire managed variables.

The session opens with context on the Pydantic ecosystem — Pydantic validation, Pydantic AI agent framework, and Logfire observability platform — and Colvin's view that "AI observability" is a feature, not a standalone category, ultimately to be absorbed by general observability or AI tooling.

The main case study involves analyzing UK Members of Parliament for political dynasty connections, replicating a real analysis Colvin submitted to The Rest Is Politics podcast, which found roughly 24% of MPs have an ancestor who was also a politician. The core challenge is getting models to correctly exclude same-generation and descending relations (spouses, children, siblings) and include only ancestors — a distinction that consistently confused models when simply instructed.

Colvin demonstrates a Pydantic AI structured-output agent that strips Wikipedia HTML via BeautifulSoup and returns typed `PoliticalRelation` objects. He runs evals against a golden dataset of 65 test cases (generated with Claude Opus 4.6 and manually verified) using Logfire's evaluation harness. A simple one-line prompt scores 87% accuracy; a manually crafted expert prompt reaches 92%.

The workshop then introduces GEPA, described as a state-of-the-art prompt optimization library created by Lakshya, a first-year PhD student at Berkeley. GEPA is a genetic algorithm that operates on a string (or a key-value dict) — it identifies the Pareto frontier of best-performing prompt candidates, cross-breeds them analogously to breeding racehorses, and iterates toward an optimal value. Colvin implements a custom GEPA adapter using a second Pydantic AI agent (GPT-4.1 for speed) as the proposer, which receives Logfire-formatted context about failures and proposes new system prompt candidates. After running GEPA with 400 evaluation calls, the optimized prompt achieves 96.7% accuracy — a significant lift over the 92% human-engineered prompt — at the cost of increased verbosity.

Colvin addresses GEPA limitations directly: prompts are model-specific and must be re-optimized when the underlying model changes; variance can be reduced by running each eval case many times (he cites hedge funds spending $20,000 per night on evals); GEPA can overfit narrow test distributions (demonstrated when the optimizer systematically excluded aunts and uncles because they were absent from the training split); and the technique is most valuable for private-data domains where models have no prior training signal, such as internal financial documents, legal databases, or medical records. He contrasts GEPA with DSPy's longer-standing optimization approach and with fine-tuning, noting that Shopify reduced annual model costs from roughly $5 million to $73,000 by combining GEPA-style prompt optimization with a smaller quantized model instead of GPT-5.

The second half covers Logfire managed variables. Colvin demos a FastAPI web server where a Pydantic model (containing `instructions`, `model`, and `max_tokens` fields) is registered as a managed variable backed by the OpenFeature open standard. Without restarting the server, he updates the system prompt via Logfire's targeting UI to reply first in French, then in German, then switches the underlying model from Anthropic Sonnet to OpenAI GPT-4.1 — all live. The system supports A/B traffic splitting to roll out prompt or model changes incrementally. Colvin describes the roadmap toward "self-driving" managed variables: using evals and GEPA in a closed loop to autonomously hill-climb toward an optimal configuration without human intervention.

Audience Q&A covers implicit user feedback (observing what users say next in a chat as a feedback signal, analogous to Google's dwell-time ranking), handling sensitive data (recording only categorical performance metrics to avoid exfiltrating PII, or self-hosting Logfire within a VPC), compaction strategy as an additional optimization axis, and the trade-offs between optimization, fine-tuning, and simply waiting for the next model generation.