From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

Cormac Brick, a software engineering tech lead on Google's AI Edge team, presents a practical deep-dive into building on-device agents and deploying tiny language models (TLMs) on mobile hardware. The talk covers the full stack from model selection through fine-tuning, runtime deployment, and production applications.

Brick frames the motivation for on-device AI around five pillars: latency, privacy, offline capability, reliability, and cost savings. He then distinguishes two developer paths Google provides. The first is system-level GenAI via Gemini Nano through AI Core, where the model is pre-installed on the device (using Gemma 4 E2B and E4B as base models for Nano), meaning the app carries no size overhead. The second is app-level GenAI via LiteRT-LM, where developers bundle a custom or fine-tuned model directly inside their application, enabling full customization at the cost of more engineering effort.

The talk then pivots to the Google AI Edge Gallery app — open source on Android, built on LiteRT-LM — which serves as both a demonstration platform and reference implementation. The centerpiece demo is a "restaurant roulette" skill built on Gemma 4, illustrating how an agent skill harness works: a system prompt describes available skills at a high level, and a dedicated `load_skill` tool call is triggered lazily when the model decides a specific skill is needed. Individual skills can include JavaScript for in-app UI rendering, enabling custom visual components like the roulette wheel without any native code changes. Skills can be authored externally, published to GitHub, and loaded into the app via URL, creating a community skill ecosystem. Brick notes that within a conversation, chaining three or four sequential skill calls (e.g., Wikipedia lookup followed by Google Maps navigation) works robustly, while triggering multiple skills in a single response is still an active area of development.

The second half focuses on Function Gemma, a 270-million-parameter model released in December (based on Gemma 3 architecture) specifically designed for robust function calling at tiny scale. On a Pixel 7, Function Gemma achieves approximately 2,000 tokens per second prefill and 140 tokens per second decode — fast enough for real-time text-to-function or voice-to-function pipelines on legacy hardware. Out of the box on a fixed set of app intents, Function Gemma achieves only 46% accuracy. After fine-tuning on a synthetically generated dataset — where Gemini Flash is used to generate training examples — accuracy jumps to over 90% on eight of ten target functions, with the remaining two functions achieving somewhat lower scores. Brick emphasizes that for sub-200-million-parameter models, fine-tuning on a narrow, focused task is essentially mandatory for production reliability. A public fine-tuning lab is available as a Hugging Face Space (searchable as "Function Gemma fine-tuning lab"), allowing developers to define functions, upload data, and run the fine-tuning flow without custom infrastructure.

The export and inference workflow runs from PyTorch via the `litert-torch` package to the LiteRT-LM file format — a single-file container bundling the model weights, tokenizer, and inference configuration. This format replaces older `.task` files for LLM workloads, with Swift and JavaScript APIs forthcoming in addition to the existing C++ and Java APIs.

Finally, Brick presents Eloquent, a production iOS transcription app built entirely on chained tiny models. It chains a custom ASR engine and a text-polishing engine, both built on Gemma 3 technology and both under a few hundred million parameters each. The polishing model removes filler words, applies a user-supplied personal dictionary for domain-specific terms, and corrects transcription errors. Eloquent serves as a proof-of-concept that production-grade applications can be built entirely from fine-tuned sub-billion-parameter models.