What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH

Bilge Yücel, Senior Developer Relations Engineer at Deepset, delivers a practical engineering walkthrough of what it means to build AI systems under sovereignty constraints — and, critically, what breaks when organizations try to retrofit sovereignty onto existing production systems.

Yücel opens by grounding the policy concept of sovereign AI into a concrete technical definition: having explicit control over data flow, model choice, infrastructure, observability, and operations. She structures this around four sovereignty pillars. Data sovereignty covers where data is stored and processed — a key example being that sending EU citizen data to an embedding API hosted in Virginia is already a GDPR violation, regardless of where the application itself runs. Infrastructure sovereignty concerns the compute layer: a spectrum ranging from fully air-gapped environments (EU AI Act compliant), to private VPCs (GDPR safe), to sovereign cloud, to SaaS products where the US Cloud Act creates legal exposure even when infrastructure is physically in Europe. Model sovereignty addresses vendor lock-in at the model level — both tight API coupling and the subtler problem of codebases that are technically provider-agnostic but require weeks of rewrites to switch models. The fourth pillar, operational sovereignty, covers monitoring, tracing, versioning, audit trails, and human-in-the-loop requirements for high-stakes domains like HR or finance.

The talk's most actionable section describes what actually breaks when engineers are asked to make an existing system sovereign on short notice. Swapping a frontier API for a self-hosted model requires re-evaluating system performance from scratch and rewriting prompt logic for a different model architecture. Moving private data into the correct jurisdiction creates multi-database complexity — cross-jurisdiction search queries alone become a significant engineering challenge (e.g., deciding whether to run query classification first or fan-out requests to both databases). Replacing managed cloud infrastructure with on-premises compute suddenly exposes how much Kubernetes management, GPU/CPU network bridging, and hardware provisioning had been invisibly handled by hyperscalers. Incorporating observability after the fact reveals that most AI application layers are effectively black boxes, with no version-controlled audit trail of component inputs and outputs.

Yücel then makes a case for Haystack (Deepset's open-source orchestration framework) as a structural solution to several of these problems. Key capabilities highlighted: a consistent interface enabling cloud-to-self-hosted migration with minimal code changes; fully typed and declared data flows so every input and output is traceable even in agentic architectures; YAML serialization for deterministic versioning (pipelines can be committed to git and reproduced from any hash); and being fully open source with no hidden assumptions or black-box components. She demonstrates a sovereign agent architecture in code, showing guardrails implemented via an Nvidia chat generator and LLM message router, dynamic BM25 tool search across hundreds of tools (to avoid filling context windows with tool definitions), MCP server integration via MCPToolset with selective tool registration, custom on-prem chat generator components, and confirmation strategies for human-in-the-loop approval on specific tool calls (e.g., payment request submission).

The talk closes with a three-question sovereignty checklist: (1) Can you swap models without changing application logic? (2) Do you have reproducible run logs stored in a compliant location? (3) Can your team respond to an incident without calling a hyperscaler? Deepset's enterprise customers include Airbus, Bosch, Siemens, the European Commission, and several German federal ministries, giving the sovereignty concerns a grounded, production context.