Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind

Patrick Löber, a member of technical staff at Google DeepMind working on the Gemini API and AI Studio, presents a practical walkthrough of building native multimodal agents using the any-to-any capabilities of the Gemini ecosystem. The talk is structured around four core capabilities—multimodal understanding, native image generation, native speech generation, and real-time audio interaction—culminating in the building blocks for a NotebookLM clone implemented as an agent rather than a hardcoded workflow.

Löber opens by clarifying the current state of the Gemini model landscape. Despite the aspirational 'any-to-any' framing, the architecture is not yet a single unified model. Instead, there is a primary Gemini model series (currently Gemini 3) that handles multi-modal input—text, code, image, audio, video, URLs, and Google Search—but outputs only text. Separate specialized native generation models handle image and speech output; these are named Nano Banana for image generation and are based on Gemini's world understanding. A distinct text-to-speech model currently runs on Gemini 2.5. The local path uses Gemma 4, which supports text, image, and video input with audio input available on smaller variants.

For multimodal understanding, Löber demonstrates the Google AI SDK, available in multiple languages and free via ai.studio. Files—PDFs, video, MP3s—are uploaded via the file API and passed into a `client.models.generate_content` call targeting Gemini 3 Flash. A key practical note: one minute of audio equals 1,920 tokens, and with a 1-million-token context window, Gemini can ingest over nine hours of audio or approximately one hour of video in a single call. Context caching is built into the API and can reduce costs by up to 90% for repeated queries over the same large files. YouTube URLs can be passed in directly without pre-downloading.

The agentic generation phase uses Gemini as a reasoning model in a loop. Function declarations define tool calls for image generation (using Gemini 3.5 Flash Image Preview, also known as Nano Banana 2) and speech generation (the text-to-speech model). An agent prompt instructs Gemini to act as a 'research agent partner,' decide which concepts are complex enough to warrant a visual diagram, and which sections benefit from audio summaries. The agent reasons iteratively about whether generated assets are sufficient before terminating the loop. The result is a system that produces text, infographics, and podcast-style audio from a heterogeneous set of source documents.

Löber highlights the practical value of native generation: because Nano Banana is trained on top of Gemini's world model, it understands semantic content rather than just pixel patterns. He demonstrates this with an example from Twitter where drawing arrows on a map and asking Gemini to generate a picture yields the Golden Gate Bridge—the model infers the subject from geospatial understanding rather than visual matching. Native image generation also enables math homework correction with annotated diagrams and code generation rendered inside images.

The speech model is multilingual with accent control. Löber demos a British accent and a Bavarian accent, noting that accent selection is supported for a range of options though not exhaustively. The model supports two-speaker podcast-style audio files.

For real-time interaction, Löber introduces the Live API and Gemini 3.1 Flash Live, described as a true audio-to-audio model: a single architecture where audio enters and audio exits, eliminating cascaded ASR-LLM-TTS pipelines. A video demo shows the model identifying the speaker's appearance in real time. The Live API is available for experimentation at ai.studio/live.

Löber also briefly mentions a new multimodal embedding model that maps text, image, audio, and video into a unified vector space—enabling multimodal search applications—as a complement to the pipeline capabilities described.