Building Generative Image & Video models at Scale - Sander Dieleman, Google DeepMind

Sander Dieleman, a research scientist at Google DeepMind with over a decade of experience on the generative media team (responsible for models like VEO), delivers a technical behind-the-scenes overview of everything involved in training large-scale generative image and video diffusion models. The talk spans eight interconnected areas: data curation, representation, diffusion modeling mechanics, network architecture, training at scale, sampling and guidance, distillation, and control signals.

Dieleman opens by emphasizing that data curation is chronically underrated — time spent improving data often yields better returns than model tweaking, yet academic incentives historically discouraged it by rewarding use of standard benchmarks. He argues this mindset must be unlearned for production-scale model training.

On representation, he explains why raw pixels are impractical for high-resolution video: 30 seconds of 1080p video at 30 FPS occupies several gigabytes for a single training example. The solution is learned latent compression via auto-encoders rather than traditional codecs like H.265, which obscure too much structure for generative modeling. Latents preserve spatial grid topology but at reduced resolution (e.g., a 256×256 RGB image compresses to a 32×32 latent grid with extra channels), reducing tensor sizes by up to two orders of magnitude. The EQ-VAE paper is cited as an example of improving auto-encoder training.

The diffusion modeling section provides an intuitive explanation of the forward corruption process (adding Gaussian noise gradually) and backward denoising process. Dieleman frames diffusion as "spectral autoregression": because natural images and video follow a power-law frequency spectrum, adding noise progressively destroys high-frequency components first. This means denoising models implicitly generate content coarse-to-fine — sketching global semantics before adding texture detail — which is a natural fit for visual media.

For network architecture, U-Nets dominated early diffusion models (including early Stable Diffusion), but transformers have since replaced them. Because bidirectional attention is appropriate for image/video denoising (unlike causal LLMs), diffusion transformers are slightly more expressive than standard LLM architectures. Knowledge from LLM scaling transfers directly. For video, a hybrid approach between full autoregression and joint 3D denoising exists: autoregressing in time while using diffusion per-frame, as exemplified by Google's Genie model.

Training at scale requires both data parallelism and model parallelism. Google DeepMind uses JAX with Pjit for automatic model sharding across TPUs, minimizing inter-chip communication. Diffusion models remain somewhat smaller than frontier LLMs, partly because the guidance mechanism lets them punch above their parameter weight.

Classifier-free guidance is described as the single most impactful technique in diffusion sampling. By computing the difference between a conditional prediction (given a text prompt) and an unconditional prediction, then amplifying that delta, guidance dramatically improves sample quality at the cost of diversity. Citing the 2021 GLIDE paper from OpenAI, Dieleman notes that today's models would look surprisingly poor without guidance — it is universally applied. He also notes that guidance works best when there is a semantic abstraction gap between the conditioning signal and the generation target, which is why text-to-image guidance is so effective but low-level signals like segmentation masks work less well.

On distillation, the goal is reducing sampling steps rather than shrinking model size. Consistency models are highlighted: once a deterministic sampling path is known, a student model learns to predict the endpoint directly rather than tracing the trajectory. In practice, one-step consistency sampling yields lower quality, so practitioners often use multi-interval consistency modeling (e.g., three steps) as a practical compromise.

Finally, Dieleman covers control signals beyond text prompts — reference images for identity preservation, explicit camera motion controls, and event timing for video. He advocates introducing advanced conditioning in post-training rather than pre-training, and discusses three architectural strategies for injecting conditioning into transformer-based denoisers: extra tokens, broadcast signals, and other mechanisms. Post-training preference tuning via RLHF or DPO is also mentioned as a key lever for model behavior.