Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

June 2, 202620:401,365 views

Summary

Kobie Crawford, Developer Advocate at Snorkel AI, presents empirical evidence for what he terms "task fidelity scaling laws" — the thesis that data quality in agentic fine-tuning tasks has an outsized, measurable impact on model training outcomes. Snorkel, which traces its origins to a Stanford AI research lab and was founded in 2019, has consistently centered its business on the premise that data quality is the critical variable in AI development. This talk extends that premise into the agentic domain, specifically TerminalBench-style tasks involving containerized execution environments.

Crawford defines task quality across four criteria: achievability (the task can actually be completed), non-triviality (the task is genuinely difficult), functional correctness (test logic behaves as specified), and environment reliability (the containerized environment executes predictably). Tasks passing all four criteria are labeled "accepted"; those failing any are "rejected." These two buckets serve as the experimental proxy for high- vs. low-quality training data.

To establish that accepted tasks are actually harder — not just categorized differently — Snorkel ran accepted and rejected tasks through Claude 3.5 (Sonnet 4.5) and Codex (including GPT-4o, GPT-4.1, and GPT-5.1). Accepted tasks required twice as many tool calls on average, exhibited lower pass rates, and demanded more output tokens, all consistent with genuinely higher difficulty. Critically, the failure mode analysis showed a qualitative difference: accepted tasks failed with clean, informative signals — the model reached a dead end on a real logical challenge. Rejected tasks failed in degenerate ways — mismatches between what the task description requested and what the tests actually verified, or implicit dependencies never surfaced in the model's context window.

With that validation in hand, Snorkel ran a controlled reinforcement learning training experiment: same base model, same compute budget, same number of tasks — but one run used accepted (high-quality) tasks and one used rejected (low-quality) tasks. The result was stark. Low-quality tasks yielded roughly 1% performance improvement over the base model; high-quality tasks yielded approximately 6% improvement. That is a 5x difference in training uplift attributable entirely to task quality, with all other variables held constant.

Crawford notes that ambiguous task specifications are particularly damaging. Under-specified tasks — where the desired outcome is not clearly encoded in the test definition but tests nonetheless expect specific behaviors — create noise rather than signal. They appear hard because they produce failures, but those failures do not indicate that the model lacks a useful skill. They indicate that the task was poorly constructed. This distinction matters enormously for benchmark design: saturation of benchmarks may be masking whether models are actually improving or whether poorly constructed tasks are simply creating irreducible noise floors.

Snorkel's production approach to maintaining data quality combines human domain experts with LLM judges, using structured rubrics to measure inter-annotator agreement both between individual humans and between human annotators and LLM judges. Crawford acknowledges that verification becomes harder outside of coding and mathematics domains, and highlights Snorkel's open benchmark grants program for partnering on evaluations in fuzzier, less verifiable domains including emotionally complex and human-centric tasks. Future directions include longer-horizon multi-step tasks where partial completion and iterative problem-solving matter more than binary pass/fail outcomes.

Key Takeaways

•High-quality agentic tasks produce 6% RL training uplift vs. 1% for low-quality tasks — a 5x difference with identical compute and task count.
•Task quality can be operationalized into four verifiable criteria: achievability, non-triviality, functional correctness, and environment reliability.
•Ambiguous or under-specified task definitions do not produce harder tasks — they produce noise that masks whether model improvement is real.
•Accepted (high-quality) tasks require roughly twice as many tool calls, have lower pass rates, and need more output tokens compared to rejected tasks.
•Clean failure modes are a signal of task quality: when a model fails on a well-specified task, it fails for a genuine reasoning or skill gap reason.
•Benchmark saturation may be an artifact of poor task quality, not actual model capability limits — tasks that can never be completed create irreducible noise floors.
•Tests that check behaviors never requested in the task description create spurious failures that pollute training signal.
•Expert-in-the-loop data generation is important for maintaining quality at scale in production RL pipelines.
•LLM judges should be calibrated against human annotator agreement using rubrics, not used as a standalone quality signal.
•Verification difficulty scales with domain fuzziness — coding and math are tractable; emotionally complex tasks require multi-outcome scoring rubrics.
•When building agentic benchmarks, implicit dependencies must be made explicit in the model's context or excluded from test assertions.

Transcript

My name is Kobe Crawford. I'm a developer advocate at Snorkel. We are the frontier AI data lab. And what that means is that we produce data sets for foundation models to help climb on. So our research team is highly integrated with the work that we do in terms of our production work. And we put a lot of emphasis on how we integrate research in that. This company's origins actually begin from a Stanford University AI research lab and uh the work that they were doing there actually was part of one...