Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face
Summary
Ben Burtenshaw from Hugging Face presents a structured argument for using coding agents to tackle AI systems engineering — historically deep specializations like CUDA kernel writing and distributed ML training pipelines. The core thesis is that skills (file-based context loaded by agents at runtime) can compress expertise timelines from years to hours, turning zero-shot failures into few-shot workflows by providing structured examples and references.
The talk is organized around three progressively more autonomous challenges framed as 'bosses' from video games. Boss 1 covers writing and distributing CUDA kernels. Burtenshaw explains that GPU efficiency bottlenecks are predominantly memory-bound rather than compute-bound: an H100 delivers one petaflop per second of computation but only 3 TB/s of memory bandwidth, meaning the GPU often idles waiting for tensor data. Custom kernels — such as Flash Attention — increase arithmetic intensity by batching more computation per read/write cycle ('keeping the GPUs warm'). Hugging Face's `kernels` library manages kernel distribution through TOML-defined hardware compatibility manifests versioned as Hub repos. An agent using the kernels skill — which includes benchmarking scripts and worked examples — generated a Qwen3 8B kernel for H100 achieving a 94% speedup, noting this represents low-hanging fruit on compatibility gaps rather than state-of-the-art optimization. The `upskill` library is introduced as a tool for evaluating skill quality by generating evals and comparing different models on the same skill, enabling cost-conscious model selection (e.g., Haiku vs. GPT-4o-class models).
Boss 2 demonstrates end-to-end LLM fine-tuning from a single zero-shot prompt. Burtenshaw references a November/December blog post where Claude fine-tuned Qwen3 6B on a chain-of-thought dataset end-to-end using HF CLI skills and Hub compute. A related approach using Onslaught (a compute-optimized variant) is also mentioned as available with free credits. A separate talk by colleague Merve Noyan covered this topic in greater depth.
Boss 3 is AutoLab, a multi-agent autonomous research lab inspired by Andrej Karpathy's Auto Research project (which iterated on nanoGPT training scripts using a single Claude Code agent). Burtenshaw's extension distributes the task across four specialized agent roles: a Researcher that scouts HF Papers and arXiv for hypotheses using the HF Papers CLI; a Planner that maintains a job queue; Workers that implement hypotheses as modified training scripts (changing architecture or hyperparameters); and a Reporter that monitors all jobs and pushes results to a live Trackio dashboard. The system operates on a Git repo with a main branch tracking training scripts and a shared data structure for scores. Workers submit jobs to Hugging Face compute via HF Jobs. Implementations in OpenCode, Claude, and Codex are provided in the public repo. The Trackio dashboard is highlighted as ideal for agentic workflows because it uses a fully open Parquet data layer, enabling agents to generate arbitrary visualizations (Gantt charts, tables) without being constrained by a fixed dashboard schema. The talk shows a Gantt chart derived from the raw Trackio data illustrating parallel agent activity over time.
Burtenshaw's closing argument emphasizes two principles: agents work best with fully open primitives (not abstracted APIs that create ceilings), and the Hugging Face Hub — with integrated storage, tracking, and compute — is ready for agentic ML engineering workloads at scale. The description notes a finetuned Qwen3 0.6B reached 35% on LiveCodeBench and an RMSNorm kernel achieved 1.88x speedup on H100s, both produced by agents without a dedicated systems engineer.
Key Takeaways
- •Skills (file-based context with examples and benchmarking scripts) transform zero-shot agent failures into reliable few-shot workflows for complex tasks like CUDA kernel writing.
- •GPU performance bottlenecks are usually memory-bound, not compute-bound — custom kernels improve arithmetic intensity to keep the GPU computing rather than waiting for tensor data.
- •The Hugging Face kernels library lets agents publish optimized kernels as versioned Hub repos with TOML-defined hardware compatibility, enabling easy distribution and reuse.
- •Use the upskill library to evaluate whether a skill works reliably and to benchmark cheaper models (e.g., Haiku) against more expensive ones on the same skill before committing to a model choice.
- •A multi-agent research lab with distinct Researcher, Planner, Worker, and Reporter roles parallelizes hypothesis generation and experiment execution, enabling overnight autonomous ML research.
- •The HF Papers CLI enables agents to programmatically search and retrieve papers as a literature scout within a multi-agent pipeline.
- •Trackio's open Parquet data layer makes it the best agent monitoring dashboard because agents can generate arbitrary visualizations without being constrained by a fixed schema.
- •Agents work best with fully open primitives — abstracted APIs create ceilings that limit agent autonomy and capability.
- •The Hugging Face Hub's combination of storage, model hosting, compute (HF Jobs), and tracking infrastructure makes it production-ready for agentic ML engineering workloads.
- •Both a 1.88x speedup on H100 RMSNorm kernels and 35% LiveCodeBench on a finetuned Qwen3 0.6B were achieved by agents without a dedicated systems engineer, demonstrating practical AI systems engineering at scale.
Transcript
[music] >> Hi everyone. As you heard, I'm Ben from Hugging Face and the talk that I'm going to present to you today is called your coding agent should do AI systems engineering. So, there are two main takeaways that I want you to get from this talk. One, and probably the fun part, is that we can use coding agents to tackle the hardest engineering problems in AI, so systems engineering and machine learning engineering. And maybe the boring part is that in order to do this, we're going to need sta...
Speakers
Topics
Related Sessions
