Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI
Summary
Mahmoud Mabrouk, co-founder and CEO of Agenta AI, delivers a workshop on building calibrated LLM-as-a-judge evaluators using the GEPA (Genetic-Evolutionary Prompt Optimization) algorithm. The talk opens with a common anti-pattern: teams deploy off-the-shelf hallucination judges, see green dashboards, yet still get customer complaints — because prompts like "rate whether this output is a hallucination" cannot work without domain knowledge the app itself lacks.
The core argument is that miscalibrated evaluators are worse than none: they produce false confidence and stall the development loop. A calibrated judge aligned with human annotations directly accelerates the prompt-engineering iteration cycle (experiment, eval, improve) and enables reliable online production monitoring. Mabrouk frames this as the foundation for a "data flywheel" where a system auto-improves from observed traces.
The practical use case is an airline customer support agent from the TaoBench benchmark (Sierra), with 599 annotated conversation traces (62% compliant, 38% non-compliant). Annotation reasoning labels — e.g., "non-compliant because the agent approved a cancellation without verifying the reservation met the airline's cancellation rules" — are critical for the optimization algorithm to learn from.
Metric design is emphasized as harder than optimization. Mabrouk recommends: (1) deriving metrics from the specific business use case rather than generic hallucination scores; (2) splitting evaluation into multiple narrow binary judges (policy adherence, response style, information delivery, tool call correctness) rather than one composite; (3) using binary pass/fail labels over 1–5 scales, since even human annotators struggle to agree on numeric scores.
The GEPA algorithm is a genetic/evolutionary approach: starting from a seed prompt, it iterates by (a) mutating candidates via LLM reflection — the judge observes failures and proposes a revised prompt — and (b) merging multiple candidates to combine policy rules. Candidate selection uses a Pareto frontier strategy: rather than picking the highest average-scoring prompt, it identifies the best candidate per individual test case, ensuring full training-set coverage. The open-source `gepa` library's `optimize_anything` API is used, which can optimize any parameterized configuration beyond just prompts.
Key engineering decisions: Mabrouk wrote a custom reflection template injecting domain priors (airline policy context, annotation format, instruction to surface policy rules) instead of using the GEPA default — the default failed to converge on this complex task. The seed prompt assumes compliance by default; seeds starting neutral produced near-random outputs that the algorithm could not recover from.
Results: the seed judge achieved 61% accuracy with 98% compliant recall but near-zero non-compliant recall. After GEPA optimization, accuracy rose to 74% on the 112-trace validation set — a 14-point gain — with non-compliant recall and precision improving substantially. The Pareto frontier reached 100% on training data, though merging all candidates into one deployable prompt remained a bottleneck.
Failed experiments: older/smaller models (GPT-4o mini, nano) failed as judge or refiner for complex policy. Best combination was Gemini for reflection and Grok for judging. Including the full agent policy in the seed prompt performed worse than omitting it — Mabrouk hypothesizes it traps the algorithm in a local minimum. Cost note: experiments ran $200–$300 in API tokens due to long trajectory inputs. Workflow advice: start with one GEPA iteration, inspect candidates and reasoning traces in an observability tool like Agenta, overfit to training data to confirm convergence, then scale sampling.
Key Takeaways
- •Miscalibrated LLM judges are worse than no judges — they generate false confidence and waste optimization iterations without producing signal.
- •Design evaluation metrics from the specific business use case, not generic measures like hallucination; involve subject matter experts in defining failure modes.
- •Break complex evaluation into multiple narrow binary judges (e.g., policy adherence, tool correctness, response style) rather than one composite score — it is far easier to calibrate each independently.
- •Use binary pass/fail labels with natural-language reasoning rather than 1–5 numeric scores, since even human annotators struggle to agree on fine-grained numeric scales.
- •Include annotation reasoning (why a trace is compliant or not) in training data — without it, GEPA cannot infer policy rules and the judge fails to converge.
- •Engineer the seed prompt to assume a 'passing' baseline by default; a neutral or adversarially biased seed leads to near-random initial outputs that the algorithm cannot recover from.
- •Write a custom GEPA reflection template with domain priors rather than relying on the default — this significantly improves optimization quality for complex policy-heavy judges.
- •Do not include the full agent policy in the seed prompt; it creates a local minimum and prevents exploration of the prompt space during optimization.
- •Start GEPA experiments with micro-iterations (one round, small batch) and inspect candidates and reasoning traces before scaling up — diagnose before spending compute budget.
- •Smaller/cheaper models (GPT-4o nano, older GPT-4) fail as judge or refiner for complex tasks; use a capable model (GPT-4o mini or better) and accept the cost as a one-time investment.
Transcript
Hello everyone and welcome to my talk slashworkshop judge the judge and today we're going to talk about LLM as a judge quite sure you know this scenario you have an agent in production and someone from the team says we need to monitor the reliability so you go to one of the libraries and maybe use the hallucination LM as a judge you put it in production within your observability platform and things looks fine But customers are actually saying that the agent is not working and you look at the tra...
Speakers
Topics
Related Sessions
