What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench

April 24, 202620:248,991 views

Summary

Peter Gostev, from Arena.ai, delivers a data-driven critique of the prevailing narrative that LLM benchmark progress translates directly to real-world capability. Drawing on two distinct data sources — his own BullshitBench and Arena.ai's human preference dataset of over 5.5 million votes going back to Q2 2023 — Gostev argues that the relentless upward lines on standard benchmarks obscure persistent, meaningful failure modes.

The BullshitBench is a 155-question evaluation designed around nonsense or internally incoherent prompts. Rather than measuring problem-solving ability, it measures a model's willingness to push back when a question is fundamentally unanswerable. One example prompt asks how to attribute variance in deployment frequency to indentation style versus average variable name length while controlling for a 'positive age' — pure nonsense. Gostev grades responses via an LLM-as-judge pipeline he validated himself. The results are striking: Claude models (particularly since Sonnet 4.5) perform best, with clear pushback rates well above 50%. Qwen and the latest Grok models fare reasonably well. GPT and Gemini models are roughly 50/50 on whether they accept or reject the nonsense premise. Critically, enabling extended thinking or reasoning mode does not help — it often makes things worse. Gostev observed GPT-5.4 traces where the model would note a premise is questionable in a single line, then spend 20 paragraphs attempting to solve the non-problem anyway. His hypothesis: models have been trained so aggressively to complete tasks that there is almost no training signal for the correct behavior of declining to solve. Scaling model size also shows no clear correlation with BullshitBench performance across open-source models.

Gostev then pivots to Arena.ai's 'dislike both' voting mechanic, which he frames as a dissatisfaction rate. Tracking the top 25 models over time, this rate fell from roughly 17-20% in the pre-reasoning era to about 12% after the release of o1, and to approximately 9% by Q1 2026. While real progress, a 9% dissatisfaction rate means users dislike responses from two strong models simultaneously nearly one in ten times — a result that does not match the story told by conventional leaderboards.

Breaking dissatisfaction down by expert category reveals further nuance. Quantitative tasks (math, physics) saw dramatic improvements, matching widespread intuition. Creative writing improved but modestly. Finance and law showed minimal progress. Among software subcategories, gaming stands out as persistently poor: Gostev notes that LLMs still have no coherent model of game mechanics, producing games that are neither interesting nor challenging. Agent system prompts, by contrast, showed relatively low dissatisfaction rates. Comparing Q2 2024 to Q1 2026, the overall software expert dissatisfaction rate dropped from 23.5% to 13% — meaningful, but uneven across subcategories.

Gostev closes by arguing that benchmark hillclimbing is concentrating improvement at the top of the performance distribution while the broader distribution — including harder, fuzzier, judgment-intensive tasks — is improving more slowly. The gap between benchmark scores and real-world satisfaction reflects the fact that most white-collar work involves contextual judgment that narrow benchmarks cannot capture. He encourages the audience to visit Arena.ai's Hugging Face for published leaderboard data and expert prompt samples.

Key Takeaways

•A 9% dissatisfaction rate on Arena.ai — where users dislike both top-model responses — persists even in 2026, showing meaningful unresolved failure at real-world tasks.
•BullshitBench reveals that most major LLMs (GPT, Gemini) accept nonsense premises roughly 50% of the time; Claude models since Sonnet 4.5 are significantly better at pushing back.
•Enabling extended thinking or high-reasoning mode does not improve, and often worsens, a model's ability to reject invalid or incoherent questions.
•Models appear to have been trained with strong incentives to complete tasks at any cost, but with minimal training signal for the correct behavior of refusing or redirecting unanswerable questions.
•Quantitative categories (math, physics) have seen dramatic dissatisfaction-rate improvements, but creative writing, finance, and law have improved only modestly or negligibly.
•Gaming is a persistently weak category: LLMs still lack coherent mental models of game mechanics, producing games that are neither interesting nor challenging despite years of improvement elsewhere.
•Benchmark progress is concentrated at the top of the performance distribution; the broader distribution of hard, judgment-intensive, fuzzier tasks is improving more slowly.
•User expectations shift over time — people ask harder questions now than three years ago — meaning longitudinal Arena data measures the battle between expectation and capability, not capability alone.
•BullshitBench is open source with 155 questions and an LLM-as-judge grading pipeline; practitioners can run it on any model to measure premise-rejection robustness.
•Arena.ai publishes expert prompt samples and leaderboard data on Hugging Face, offering a less exhaustible and more naturalistic evaluation resource than static benchmarks.

Transcript

[music] >> I want to talk to you something maybe a little bit controversial today. Uh you can argue with me later. Uh but the topic is what do models tell suck at? And uh the reason why I wanted to talk about it is that I think we uh all look at these kinds of charts where any benchmark you seem to look at, the line goes up, and we look at meter charts, and they surprise us every time no matter how prepared we are. And this could create this kind of psychosis that we'll see where everyone is fre...