How Transformers Finally Ate Vision – Isaac Robinson, Roboflow

Isaac Robinson, Research Lead at Roboflow, delivers a technical walkthrough of how Vision Transformers (ViTs) came to dominate computer vision despite being a seemingly poor architectural fit for image data. The talk traces the evolutionary arc from CNNs through the key architectural milestones that ultimately cemented transformer supremacy: ViT, Swin, ConvNeXt, Hiera, SAM, and RF-DETR.

Robinson opens by contrasting CNNs and ViTs on their fundamental properties. CNNs carry strong inductive bias — translation invariance and locality — motivated by how biological vision works, and scale as O(n²) with image resolution. ViTs, by contrast, apply a set-to-set attention operation over 16×16 image patches with no built-in spatial assumptions, scaling as O(n⁴) with resolution. On the surface, CNNs should win. In practice, they lost.

The core argument Robinson advances is that massive ViT-specific pretraining — not architectural superiority — explains the transformer's victory. He traces this through four intermediate architectures. Swin Transformer introduced shifted local attention windows, recovering O(n²) complexity and locality bias, seemingly vindicating the idea that vision-specific inductive biases still matter. ConvNeXt then went the opposite direction: it took the transformer design formula (mixer → feed-forward → mixer → feed-forward, layer norm, 4×4 patches instead of 16×16) and applied it to a pure convolutional backbone, outperforming both ViT and Swin on standard ImageNet benchmarks. This appeared to restore CNN credibility.

Hiera (from Meta) provided the decisive insight. Researchers systematically stripped inductive biases from a well-performing model one at a time, finding that each bias removal yielded a speedup, and that pretraining — specifically Masked Autoencoder (MAE) pretraining — could recover the lost accuracy. MAE, conceptually equivalent to BERT for vision, masks random patches and trains the model to reconstruct them. Critically, MAE only works for ViTs: you cannot drop patches in a convolution that is applied uniformly across all positions. This ViT-exclusivity of MAE gave transformers a pretraining moat that CNNs cannot access.

Robinson also highlights DINOv2 and DINOv3 as further examples of ViT-specific self-supervised pretraining producing semantically rich feature maps — illustrated by PCA decompositions where cat paw segmentation and satellite imagery structure emerge from purely unsupervised objectives. Linear probe accuracy from frozen DINOv3 features now approaches the ceiling of fully supervised methods.

The O(n⁴) speed penalty, once a major concern, was largely neutralized by FlashAttention and related infrastructure developed for LLMs. When Hiera's paper reported speed gains over ViT, it explicitly excluded FlashAttention from the benchmark — once FlashAttention is re-added, the claimed advantage disappears.

The SAM series (SAM → Mobile SAM → SAM 2 → SAM 3) mirrors this trajectory almost exactly: each attempted architectural deviation (Mobile SAM's TinyViT hybrid, SAM 2's Hiera backbone with MAE) eventually gave way in SAM 3 to simply using the best massively pretrained ViT backbone available, abandoning further architecture ablations.

Robinson then pivots to the deployment challenge this creates. SAM 3 is 800 million parameters and takes 300ms on a T4 GPU — unusable for edge or resource-constrained scenarios that historically defined vision deployments. Roboflow's response is RF100VL, a benchmark for measuring foundation model transfer to diverse downstream object detection tasks, and RF-DETR, which uses Neural Architecture Search (NAS) over a shared foundation model backbone to generate an entire family of deployment-flexible models. Robinson reports approximately 40× speed improvement for the same accuracy versus fine-tuning SAM 3, with a 15× speed improvement yielding a meaningful accuracy gain. All models in the RF-DETR family share the same foundation model, with NAS-derived architectural knobs enabling hardware-aware and data-aware specialization.

In the Q&A, Robinson addresses video and multimodal pretraining (SAM 3's perception encoder does video tracking), and V-JEPA, noting it has not yet demonstrated strong downstream transfer for video tasks.