Spec-Driven Testing for Agents With A Brain the Size of A Planet — Steven Willmott, SafeIntelligence

Steven Willmott, CEO of Safe Intelligence, presents a framework for spec-driven validation of AI agents at AI Engineer Europe 2026. Safe Intelligence has operated for three years, initially applying formal verification techniques to vision models and tabular data by examining entire regions of input space under perturbations. At the conference, they released a new product applying analogous techniques to language models — generating edge cases and adversarial test cases without direct model access.

Willmott opens with a counterintuitive challenge to the assumption that a smarter agent is always a better agent. He uses the example of jailbreaks embedded in poems: small models fail to understand the poem and do not execute the embedded malicious instruction, while large models comprehend the poem and comply. This demonstrates that bigger models have more attack surface and broader capability to act on harmful instructions. He identifies two compounding risks with large, broadly capable agents: (1) they accept a wider variety of instruction formulations, including adversarially crafted ones, and (2) they have more infrastructure access to abuse — for example, a banking agent capable of wiring large sums is dramatically riskier than a read-only Q&A agent.

The core proposal is spec-driven validation: defining what an agent is supposed to do independently of the implementation, using a structured specification that survives model swaps and drives both security testing and iterative improvement. Willmott identifies five components that should appear in an agent spec beyond a simple evaluation dataset:

1. **Ground truth dataset** — examples of correct inputs and outputs (the traditional eval). 2. **Rules** — explicit business constraints such as "never offer more than 10% discount" or "no refunds after 30 days." He notes that reliably testing rule compliance is non-trivial. 3. **Ontologies and domain dictionaries** — the relevant universe of entities for the specific agent. An airline chatbot's spec should enumerate only the destinations that airline actually flies; internal company terminology must be explicitly defined because no external LLM will know it. 4. **Domain knowledge** — agent-specific substitution rules. For example, in business finance, "gross profit" and "gross sales" are not interchangeable, whereas a general-purpose LLM might conflate them. 5. **Rights and roles** — the agent's behavior should vary by authentication state and user permissions, and specs should capture these variations. 6. **Robustness requirements** — how much variation in input can the agent tolerate? This mirrors vision model testing (e.g., runway detection through varying fog levels) applied to language: how many typos, how many rephrasings, how much linguistic noise before the agent fails?

With a complete spec, Safe Intelligence's platform does two things: security testing (pulling spec knowledge to identify edges of vulnerability, since agents are most vulnerable exactly in the domains they are designed to act in and have the most infrastructure power there) and robustness testing (varying inputs systematically to map the envelope of reliable performance). Willmott frames the spec as enabling a "backyard RL" loop — running the agent automatically, collecting results, and iterating to close robustness gaps — without modifying the underlying model.

He connects this to emerging standards: the A2A (Agent-to-Agent) spec's concept of agent cards, which describe agent capabilities but are insufficient alone for evaluation. He also references his own prior work as a co-author of the OpenAPI Specification, positioning future work on an open, version-controlled format for agent specs that could be stored in a GitHub repo and consumed by any evaluation or testing tool.