Personalization in the Era of LLMs - Shivam Verma, Spotify

Shivam Verma, Tech Lead of the User Representations team in Spotify's AI Foundation org, presents how Spotify engineers personalization at scale for its 750 million users and 100 million-plus track catalog by integrating user embeddings, Semantic IDs, and soft tokenization into open-weight LLMs — without full fine-tuning of the base model.

Verma opens by contrasting Spotify's traditional recommender system ("trad rec") pipeline — a multi-stage candidate generation and ranking architecture where each product surface has its own siloed model — with the emerging generative, unified approach the team is building. The new paradigm treats recommendations as a sequential token prediction problem, directly paralleling how LLMs generate text.

The first pillar is foundational user modeling. Spotify generates dense user embeddings daily for over one billion users (including non-MAUs), capturing streaming history across sessions to produce a compressed vector representation of taste. Earlier work used autoencoder architectures; the team has since moved toward transformer-based sequential models. These embeddings are visualized on a hypersphere where users, tracks, and podcast episodes share a single joint embedding space — Verma's own embedding, for example, clusters near a large technology podcast given his ML engineering background. This cross-content embedding space forms the foundation for all downstream recommendation models.

The second pillar is catalog understanding via Semantic IDs, a technique originating from a Google paper in the context of YouTube. Rather than feeding raw high-dimensional catalog vectors into an LLM, Spotify tokenizes each item's vector into a compact sequence of four to six discrete tokens using a hierarchical tokenization scheme. Ariana Grande and Bruno Mars, for instance, are each represented as six tokens, with the first two shared (both are pop artists) and the remaining tokens diverging to capture genre and stylistic distinctions. This compression allows LLMs to reason over Spotify's catalog autoregressively — generating the next song or podcast episode exactly as the model would generate the next word. The domain adaptation process involves continual post-training of open-weight models (Llama, Qwen, and similar) on Spotify's catalog data formatted with Semantic IDs, enabling the model to predict the next Spotify URI as its output token.

The third pillar is soft tokenization for user-level personalization. Because fine-tuning a frozen LLM on every individual user is computationally infeasible at 750 million-user scale, Verma's team uses a vector projection layer to map a user's pre-computed embedding into the token space of the LLM. This creates a "soft token" — a user-specific, contextually updated representation that the model can attend over during inference without any weight updates to the base LLM. This approach keeps the base model frozen while injecting per-user context directly into the prompt's token stream.

The three components — user embeddings, Semantic IDs, and soft tokenization — are assembled into a steerable, generative recommender pipeline currently running in production for podcast next-episode recommendations on Spotify. Verma reports positive results on internal metrics from this stack. The team also recently launched a "Taste Profile" feature (currently available in select markets, with broader rollout planned) that surfaces a textual representation of what Spotify knows about each user and allows users to edit, confirm, or remove attributes — effectively injecting user-controlled steering signals into the generative model. A prompted playlist feature, newly extended to podcast episodes at the time of the talk, also runs on related infrastructure. Verma positions this architecture as the future of Spotify's entire recommendation stack, moving from siloed per-product models to a single unified generative backbone.