20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

Bertrand Charpentier, cofounder and chief scientist at Pruna AI, challenges the dominant assumption that state-of-the-art means a single top-ranked large foundation model. He opens with a striking compute audit: the Design Arena leaderboard for image editing models runs 26,000 battles at 62 seconds of generation time each, totaling 20 days of compute, $5,000 in cost, and approximately 556 kWh of energy — roughly equivalent to 400 marathons of human energy expenditure. A fast compressed model from Pruna completes the same evaluation in 7 hours at $265. This contrast anchors the talk's core thesis: efficiency is a first-class dimension of state-of-the-art, not an afterthought.

Charpentier identifies two common approaches practitioners use to pick a model — checking public leaderboards and running internal benchmarks — and systematically exposes the flaws in each. On public leaderboards, he notes that three major platforms (LM Arena, Design Arena, and Artificial Analysis) produce different rankings for the same image editing models. The Elo score ranges differ drastically across platforms (e.g., 1,100–1,300 on one vs. a completely different range on another), making cross-leaderboard comparisons unreliable. One model, "Human," ranks 10th on Artificial Analysis but 5th on Arena. Moreover, even the top-ranked model on any aggregate leaderboard loses roughly 40% of head-to-head battles, meaning it is the wrong choice for nearly half of real-world use cases. Task-specific sub-leaderboards (e.g., object removal, background change, text editing) show ChatGPT Image is never top-ranked on any individual task — different specialized models lead each category.

On internal benchmarks, Charpentier runs a live audience poll showing attendees select different preferred images from the same set, illustrating how manual inspection is doubly biased — by individual preference and by the specific samples chosen. He then presents automated metric results for eight image models scored across multiple CLIP score variants, showing the rankings change across dataset variants with differences so small they fall within noise. He contrasts this with text-rendering-specific metrics where models span the full 0–1 range clearly, concluding that metric selection must match the target use case precisely.

His proposed solution is the Pareto front: plot models on a two-axis chart with quality score (Elo) on the Y-axis and an efficiency metric (latency or cost-per-generation) on the X-axis. The Pareto frontier typically contains three to four models clustered within a narrow quality band (e.g., all between 1,100 and 1,200 Elo) but varying up to 20x in generation speed or cost. He shows a task-specific Pareto front for text rendering using optimized Flux 2 and Flux 2 Flex models (developed in collaboration with Black Forest Labs), demonstrating these smaller specialized models sit on the frontier while being significantly faster than large generalist models.

Charpentier closes by describing Pruna AI's approach: serving fast image and video models that generate in 1–5 seconds via hosted endpoints, alongside an open-source compression library. He outlines the compression techniques employed: per-module quantization (applying different quantization to different model components), structural pruning, and denoising step reduction via distillation or caching (reducing 20–50 diffusion steps down to 4–20). He argues that practitioners who apply multi-benchmark, use-case-targeted, efficiency-aware evaluation will naturally discover small specialized models rather than defaulting to large foundation models.