Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

Chintan Parikh, Product Manager for LiteRT at Google AI Edge, presents a practical overview of the current state of on-device AI deployment, focusing on Google DeepMind's Gemma 4 Edge models and the LiteRT runtime stack. The session is aimed at engineers looking to move AI workloads off the cloud and onto end-user devices.

Parikh opens by establishing the four core motivations for edge inference: low latency (critical for real-time camera applications like AR filters and video calls), privacy (processing sensitive data locally without network round-trips), offline capability, and cost reduction by offloading token-heavy cloud calls. He frames edge as a hybrid strategy rather than a wholesale cloud replacement.

The main model focus is Gemma 4's two edge-optimized variants: the E2B (2-billion parameter model, approximately 1–2 GB RAM after quantization, suitable for voice interfaces and summarization) and the E4B (4-billion parameter, higher RAM footprint, better suited for laptops and IoT). Both models are Apache 2.0 licensed and available on Hugging Face with pre-quantized variants and platform-specific benchmarks. Parikh highlights three new capabilities introduced in Gemma 4 Edge over prior Gemma 3 variants: native function calling (tool calling against local or external APIs without prompt engineering), structured JSON output (baked into model architecture rather than achieved through prompting), and chain-of-thought / thinking mode (an explicit reasoning trace that can be surfaced in the UI).

Parikh demonstrates these capabilities through Google's Gallery app — an open-source playground available on Android and iOS — which bundles agent skill examples including a Wikipedia query skill, a mood/sleep tracker agent with trend analysis, an image-to-music pairing skill, and an animal vocalization workflow involving multi-app navigation. All inference runs entirely on device. The app includes a skill editor so developers can write and load custom skills without leaving the app, and community-contributed skills are published to a public GitHub repository.

The runtime underpinning these experiences is LiteRT (the rebranded TensorFlow Lite), which supports PyTorch and JAX models in addition to TFLite-format models. Parikh describes the full deployment pipeline: LiteRT Torch for model conversion, optional quantization via the Model Explorer graph visualization tool, and the AI Edge Portal — a cloud-based benchmarking service for testing models across diverse Android device fleets (addressing the "5-year-old phone" problem). LiteRT runs cross-platform across Android, iOS, macOS, Linux, Windows, web, and IoT (including Raspberry Pi), with a single TFLite file format usable across all targets.

Hardware acceleration is a central theme. CPU and GPU delegates are universally available today. NPU acceleration is completed for Qualcomm and MediaTek chipsets, with additional partner integrations in progress. Parikh claims NPU execution yields 3–10x performance improvement and significant energy savings for real-time AR/VR and TTS applications. Benchmark figures cited: up to 13x boost from NPU acceleration in specific scenarios; approximately 56 tokens per second on iOS; and LiteRT runtime running up to 35x faster than llama.cpp on mobile, roughly at parity on desktop, and 3x faster on IoT. A live demo using a Raspberry Pi robot (built the morning of the talk) illustrated LiteRT LLM running on a CPU, responding to a natural-language command to move the robot's antenna.

Parikh closes by noting that orchestration architectures — where an agent dynamically decides whether to run a sub-task locally or route it to the cloud — are an active area of development at Google, though no production examples were shared. He invites engineers working with open-weight models to explore the LiteRT conversion path, as the framework accepts any model that can be converted to TFLite format.