Agents on the Canvas in tldraw — Steve Ruiz, tldraw

Steve Ruiz, founder of tldraw, presents a talk at AI Engineer Europe 2026 covering the evolution of AI agents integrated directly into tldraw's infinite canvas, culminating in a December 2025 experiment called Fairydraw. The talk is delivered with live demos and traces tldraw's AI journey through three distinct phases.

Ruiz begins by introducing tldraw as both a free online whiteboard and a commercial SDK used by products like Replit's agent canvas and Luma AI's canvas. He emphasizes that building AI-powered demos is part of his job to prove the SDK's capabilities.

The first phase discussed is Make Real (2023), an early viral experiment where users could sketch a UI on the canvas and have a vision model convert it into a working HTML prototype. Ruiz contextualizes this as historically significant because vibe coding had not yet been coined as a term, and it was one of the first experiences allowing non-technical users to create functional prototypes without writing code. He also notes a fundamental technical challenge: vision models have less training data than text models, and spatial data conflicts exist — for instance, the mathematical Y-axis (up = positive) conflicts with the web DOM Y-axis (down = positive), and linguistic ambiguities like 'left' (viewer's left vs. stage left) caused unpredictable model behavior.

The second phase involved structured-output drawing agents. Rather than generating images via diffusion models, tldraw used LLM structured outputs to produce the same primitives (circles, shapes, connectors) a human user would create. The agent was placed in an agentic loop — produce, review, iterate — following conventions from coding agents like Cursor. Demos included generating a butterfly lifecycle diagram and auto-completing a D&D character sheet by reading existing form context.

The third and most ambitious phase is Fairydraw (October–December 2025, live at fairies.tldraw.com). Fairydraw introduces multiple AI agents — represented as animated sprite characters ('fairies') directly on the canvas — that users can interact with spatially. Key multi-agent coordination mechanisms implemented include: a leader-election protocol where one fairy scouts the canvas, generates a to-do list, and delegates subtasks to the other fairies; shared canvas state visible to all agents simultaneously; and prevention of task overlap between concurrent agents. Ruiz notes this was developed in parallel with the broader industry's exploration of agent orchestration patterns.

Ruiz then describes a further extension: wrapping tldraw in an Electron desktop app over the holidays and opening an HTTP endpoint that accepts arbitrary JavaScript and executes it against the live canvas runtime — essentially a code-execution backdoor. While acknowledging this is unsafe for web apps ('don't do this'), he argues it is acceptable for a local, file-based, offline-first desktop application. This unlocked bidirectional interactions such as: visualizing code as a canvas diagram, editing the diagram, then having the agent update the underlying code to match; and making canvas elements interactive by having the agent inject event-handler scripts directly. He describes Claude willingly rewriting minified Spotify bundle code to remove podcast UI elements when asked by a team member, illustrating the model's comfort with aggressive local file manipulation.

Ruiz closes by framing the local-first, file-over-app philosophy as newly practical: agent script injection against a local runtime transforms previously idealistic offline-first ideas into a concrete architectural choice for maximizing agent agency. He draws a parallel to tools like Open Interpreter (which he calls 'Open Claude'), and encourages the audience to experiment with Fairydraw, noting that real-world usage continues to surface surprising behaviors.