Building Conversational Agents — Thor Schaeff and Philipp Schmid, Google DeepMind
Summary
Thor Schaeff (developer experience, Google DeepMind) and Philipp Schmid (developer experience, Google DeepMind) delivered a hands-on workshop at AI Engineer Europe 2026 covering how to build conversational agents using the Gemini API, spanning text-based coding agents, the new Interactions API, and the Gemini Live API for real-time multimodal voice and video applications.
The workshop opened with attendees creating free-tier Gemini API keys via AI Studio (ai.dev), then moved into building a tool-calling coding agent. Philipp demonstrated using Gemini 3 Flash as the coding model — noting that with well-structured agent skills and context, the model performs reliably despite not being the largest available — and walked through constructing an agent class with a constructor, run method, and GenAI client using the Python google-genai SDK. The agent skeleton was scaffolded using agentic coding tools (Cursor, Gemini CLI, and Anti-Gravity were the primary IDEs tested), with a dedicated Gemini Interactions API skill installed via NPX to provide the agent with up-to-date API documentation fetched at runtime rather than baked into training data.
Philipp introduced the Interactions API as the evolution of the generate content API, explaining the core benefits: server-side state management (eliminating client-side history array management), support for asynchronous/background agent execution (critical for long-running tasks taking 1-4+ minutes), implicit caching with 2-3x higher cache hit rates observed in startup deployments (cached input tokens are ~90% cheaper), fork-able conversation history via interaction IDs, and support for remote MCP servers plus tool combination (e.g., combining Google Search with custom function tools). The API stores interactions for 1 day on free tier and 55 days on paid; a store=false flag allows disabling persistence for PII scenarios. The Interactions API is also coming to Vertex AI, though timing was not confirmed.
Thor then demonstrated the Gemini Live API, a stateful WebSocket API that supports streaming real-time text, audio, and video (up to 1 frame per second) to the model, enabling low-latency voice-first assistants. The model powering this is Gemini 3.1 Flash Live — the same model deployed in the Gemini mobile app and Google Search Live. A Live Jukebox demo was shown, where a conversational DJ agent used tool calls to invoke the Lyra 3 music generation model (supporting full songs with lyrics, or 30-second clips) to generate audience-requested songs in real time. The implementation architecture involved a FastAPI server proxying the WebSocket connection between the client browser (capturing audio/video) and the Gemini Live API back end, with an auth token endpoint for secure API key handling.
Two implementation patterns were covered for the Live API: a server-proxied approach (recommended for production) and a direct browser-to-API approach using ephemeral tokens. Thinking tokens from the model are available as text events on the WebSocket channel but are not spoken aloud. The session acknowledged that native audio mode offers more natural conversation flow but provides less observability than cascading text-based pipelines, making it less suited for complex enterprise use cases requiring fine-grained logging or response rewriting.
Audience Q&A surfaced important practical considerations: speaker identification is not reliably supported (a limitation highlighted when one attendee mentioned using Nvidia Parakeet for 10-second voice enrollment); personalization requires explicitly feeding context into the session setup or via function calls during conversation; evaluation and productionization of voice agents require structured system prompts, compliance layers (SOC 2, HIPAA), and partner integrations such as LiveKit or Pipecat for transcription storage, observability tooling, and additional guardrails. Shopify Sidekick was cited as a real-world deployment using the Live API for screen-aware technical support.
Key Takeaways
- •Use the Gemini Interactions API instead of generate content for agents: server-side state eliminates client-side history management and enables 2-3x better implicit cache hit rates, reducing costs significantly (cached input tokens are ~90% cheaper).
- •Install the Gemini Interactions API agent skill via NPX to give your coding agent up-to-date API knowledge fetched at runtime — this avoids model knowledge cutoff issues with new APIs like Interactions that postdate training.
- •Gemini 3 Flash is a viable coding model for well-scoped agentic tasks; with good skills, context, and system instructions it performs reliably and consumes far fewer tokens than larger models.
- •The Gemini Live API uses a stateful WebSocket accepting real-time audio (chunked buffers) and video (up to 1 fps) — use a FastAPI proxy server to avoid exposing API keys in the browser, and handle auth via short-lived ephemeral tokens.
- •For long-running agent tasks (1-4+ minutes), use asynchronous execution with polling or webhooks rather than keeping HTTP connections open — the Interactions API supports background execution natively.
- •Interaction IDs are stored server-side (1 day free tier, 55 days paid) and can be used to branch conversation history at any prior turn, enabling parallel agent fan-out from a shared context base.
- •Setting store=false in the Interactions API disables server-side persistence, which is necessary for PII-sensitive deployments but precludes using server-side state management.
- •Native audio mode in the Live API offers natural conversation flow but lacks the observability of cascading text pipelines — for production business use cases requiring response rewriting or fine-grained logging, a text-based pipeline may be more appropriate.
- •Speaker diarization / voice enrollment (identifying a specific user and ignoring background voices) is not reliably supported in the Live API today; evaluate third-party solutions like Nvidia Parakeet for voice-fingerprint-based filtering.
Transcript
Hello everyone. It's just that Philip and I were both Germans so we thought it was funny maybe we we can do it in German. Actually looks like there's a there's a German crew there which is nice. Uh no no worries. We'll we'll we'll do it in English. We'll do it in a couple different languages maybe we'll find out. Do we have other languages in the room? Other nationalities? Yeah, what do we have? Shout it out. >> Italian. >> So Italian, Spanish. Any Icelandic? No. Then okay, close close. Romanian...
Topics
Related Sessions
