TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google
Summary
Cormac Brick, Tech Lead for Google AI Edge, presents a workshop on bringing language models to edge devices using LiteRT-LM, with a focus on two main tracks: agent skills built on Gemma 4 medium-small models, and fine-tuned tiny language models (sub-1B parameters) for in-app deployment.
Brick opens by distinguishing two deployment paradigms. "System-level GenAI" places 2–5B parameter models at the OS layer — Google's AI Core and Apple Intelligence both follow this pattern — where third-party apps access models via prompt APIs. "In-app GenAI" uses custom-fine-tuned tiny models (100–500M parameters) shipped directly with an application, targeting broader device reach beyond premium hardware.
The Gemma 4 launch (E2B and E4B variants) enables the system-level track. The E2B model requires only approximately 2GB of RAM resident at inference time, with additional per-layer embedding parameters memory-mapped and streamed at roughly a few hundred bytes per token — avoiding the need to load the full parameter table. Gemma 4 introduces built-in function calling and chain-of-thought thinking, which together unlock on-device agent skills. On a high-end Android phone GPU, the E2B model achieves thousands of tokens per second; on a Raspberry Pi, the E2B reaches roughly 133 tokens per second. The E4B model delivers proportionally lower throughput but runs on a wide device set.
The agent skills architecture is demonstrated through the Google AI Gallery app (available on iOS and Android, open source). Skills follow a "progressive disclosure" pattern: the system prompt contains only one-line skill descriptions; when the model determines a skill is relevant, it invokes a `load_skill` tool call whose response is the full `skill.md` instructions. This keeps context windows compact for edge models — critical for reliability in 2–4B parameter models. Skills are composed of a `skill.md` metadata file plus optional JavaScript assets. Three core orchestrator tools — `load_skill`, `run_javascript`, and `run_intent` — are sufficient to drive the full skill system. Constrained decoding is applied specifically at tool-call generation time, restricting outputs to the known finite tool set; this yields especially large reliability gains for small models. Brick reports that about 80 skills were vibe-coded internally using Gemini CLI or similar tools, with the ADB integration enabling automated on-device validation.
For the tiny-model track, Brick highlights Function Gemma (270M parameters), which achieved 85–90% reliability on 10 mobile function-calling tasks after fine-tuning, compared to a baseline of roughly 40–50% pre-fine-tune — a 20–40 point improvement typical for sub-500M models. The recommended workflow is: start from a Gemma 3 checkpoint (e.g., Gemma 3 270M), generate synthetic training data with a large cloud model, fine-tune with LiteRT-Torch (which includes quantization), and export a single `.tflm` file deployable across CPU and GPU on Android, iOS, macOS, Linux, Windows, web, and IoT. NPU deployment requires ahead-of-time vendor compilation producing a device-specific artifact, but the app-level API is consistent across NPU, CPU, and GPU.
A production example is AI Edge Eloquent (iOS), a transcription app using two separate fine-tuned Gemma-derived models: an ASR engine and a text-polishing engine. The modular design enables reuse of the transcription model across features and simplifies debugging. The polishing model is instruction-fine-tuned to remove filler words and apply a user-supplied biasing dictionary for proper nouns and technical terms (e.g., preventing "LoRA" from being transcribed as the name "Laura").
Third-party model support is also highlighted: Fast VLM from Apple (500M parameters, vision-language) runs on a Qualcomm S25 with NPU acceleration at very high frame rates. LiteRT-LM supports LoRA hot-swapping at runtime, making it practical for robotics and IoT platforms to maintain one base model with multiple task-specific adapters (8–100MB depending on rank). The Gemma 4 models carry an Apache 2.0 license, expanding commercial usability. Context windows are 32K for E2B/E4B and 128K for larger Gemma 4 variants; Gallery defaults to 8–12K for performance reasons. Fine-tuning documentation with Colab notebooks exists for Gemma 3 270M and Function Gemma; Gemma 4 fine-tuning recipes are available on Vertex AI.
Key Takeaways
- •Use the progressive disclosure pattern for edge agent skills: expose only one-line skill descriptions in the system prompt and load full instructions on-demand to keep context windows compact and improve reliability on 2–4B models.
- •Apply constrained decoding specifically at tool-call generation time, restricted to the known finite tool set — this yields 20–40 point eval gains for sub-500M models and remains useful up to ~2B parameters.
- •For in-app GenAI with sub-500M parameter models, fine-tuning is essentially mandatory for production reliability; expect a 20–40 point improvement on task-specific evals versus a base model.
- •The recommended tiny-model workflow is: large cloud model generates synthetic training data, fine-tune a Gemma 3 270M or 327M checkpoint with LiteRT-Torch, quantize, export a single .tflm file deployable across CPU/GPU on Android, iOS, macOS, Linux, and web.
- •A modular multi-model architecture (separate ASR + text-polishing engines) is often more pragmatic than one unified tiny model: easier to debug, weights are reusable across features, and each model has a narrower safety surface.
- •LiteRT-LM supports LoRA hot-swapping at runtime without unloading the base model, making it efficient for robotics and IoT deployments that need multiple task-specific adapters (8–100MB each).
- •The Gemma 4 E2B model requires only ~2GB RAM resident at inference time because per-layer embedding tables are memory-mapped and streamed at ~hundreds of bytes per token during autoregressive decoding.
- •NPU deployment requires ahead-of-time vendor compilation to a device-specific artifact, but the app-level LiteRT API is identical to the CPU/GPU path — enabling consistent developer experience across hardware backends.
- •Vibe-coding with Gemini CLI or Claude Code plus ADB integration is a practical way to rapidly prototype and validate on-device skills, with the CLI able to install and test skills directly on a connected phone.
Transcript
[music] Hey, my name is uh Cormbrick. I work um on Google AI edge, which is a way of kind of bringing models to the edge. This is something we use internally for our own products. It's also something we make available as open source products. As part of this, we also work really, really close with the Gemma team because they publish a lot of models that are targeted to edge devices as well. Um, yeah, quick bit of background on me. Um, yeah, I've been focusing on Edgi for the last like, wow, like...
Speakers
Topics
Related Sessions
