Self-Training Agents: Hermes Agent, HF Traces, Skills, MCP & Finetuning — Merve Noyan, Hugging Face

Merve Noyan, an open-source team member at Hugging Face, delivers a rapid-fire walkthrough of the Hugging Face ecosystem for building and self-training agents, framing it around the thesis that open-source models have fundamentally caught up with closed models. She opens by citing the Artificial Analysis intelligence index, where GLM 5.1 — a commercially licensed open model — has taken the top position over closed proprietary models, illustrating that the traditional performance gap argument no longer holds. Beyond benchmarks, she emphasizes that weight access enables quantization, fine-tuning, edge deployment, and guaranteed data privacy, all without data leaving user infrastructure — a consideration she notes is increasingly important given recent security breaches in cloud AI services.

Noyan walks through several layers of the Hugging Face Hub ecosystem relevant to AI engineers building agents. The Hub now hosts nearly 3 million models, datasets, and Spaces, with dedicated filtering for agentic and vision-language models. She points to a notable trend: major labs are now releasing models with vision capabilities from day zero — examples include Gemini 4 (an omni model), Qwen 3.5, and Chimera 2.5 — predicting this will become standard practice. The inference providers feature routes requests to the fastest or cheapest provider (Groq, Cerebras, Novita, and others) and includes a tool-use column to aid model selection for agentic workflows. A new benchmark datasets feature on Hub surfaces popular evals — SWE-Bench Pro, AIME, Humanity's Last Exam — with models ranked directly in the dataset viewer, making it easier to pick the right open model for a specific task.

For local agent deployment, Noyan recommends tools such as Pye (simple setup, works with inference providers or local llama.cpp), llama.cpp's llama-agent binary (launches a model directly from a Hugging Face Hub ID), LM Studio, and MLX. She highlights Hermes agent as her top recommendation for memory management with open models, noting GLM 5.1 autonomously debugged and fixed a broken Slack integration during a live experiment. Hub has also introduced a new traces repository type, allowing agent session logs from tools like Codex, Claude Code, or Pye to be uploaded, visualized in the dataset viewer, and used for subsequent model fine-tuning.

The core of the talk focuses on Hugging Face Skills — modular capabilities that plug into coding agents via MCP. Key skills include: the Hugging Face CLI skill for managing repositories and launching jobs; the LLM Trainer skill for kicking off fine-tuning jobs remotely or locally across LLMs, VLMs, object detectors, and segmentation models; the Gradio skill for building demos; and a dataset exploration skill using the dataset viewer API. A live demo shows Claude Code receiving the prompt to fine-tune Qwen-2-VL on the Lava-Instruct-MiX vision-language dataset. The agent autonomously calculates VRAM requirements, selects an appropriate compute instance, asks clarifying questions about validation splits, and launches the job on Hugging Face infrastructure — all without any manual napkin math. The resulting model appears directly on Hub when complete.

On the MCP server front, Hugging Face exposes models, datasets, Spaces (described as "the App Store of AI"), semantic search, and a Jobs product for one-off compute tasks. Noyan demonstrates querying the Qwen image generation Space via MCP with the prompt "generate image of a baklava made of yarn," retrieving the output inline. A closing case study from colleague Niels describes OCR-processing 30,000 AI papers on Hugging Face Hub using open OCR models, Codex, and the Jobs infra — entirely through agent prompting. The agent selected the model via the OCR-Bench benchmark dataset, wrote the processing script, computed instance costs, and launched the batch job, storing outputs in the new Buckets storage product (described as cheaper and faster than S3). Noyan concludes by noting a newly shipped model recommendation skill that advises on the best OCR model for fine-tuning use cases, reducing the barrier further.