Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Adrien Grondin, developer of the Locally AI app, presents a concise technical demonstration of running Gemma 4 on iPhone using Apple's MLX framework, achieving 40 tokens per second on the latest iPhone hardware. The talk targets iOS developers and engineers interested in on-device inference as a viable production capability.

Grondin introduces MLX as a machine learning framework built by Apple and optimized for Apple Silicon — the same family of chips powering iPhones, iPads, and Macs. The primary iOS-focused library is MLX Swift LM, available on GitHub, which provides a straightforward API that Grondin claims can be integrated into an iOS app in under 10 minutes. The framework handles model downloading via Hugging Face integration: developers pass a Hugging Face model ID to the framework and it fetches the weights directly.

For model sourcing, Grondin points developers to the MLX Community on Hugging Face, which hosts approximately 4,000–5,000 quantized model variants. When a new model is released by a major lab, the community typically publishes a quantized MLX version within 30 minutes. Available quantization levels include BF16, MXFP4, 8-bit, 6-bit, 5-bit, 4-bit, and 3-bit variants. Grondin recommends staying in the 4-bit to 8-bit range: below 4-bit, output quality degrades noticeably, while 8-bit is appropriate for the smallest models.

The demo centers on Gemma 4 (Google's model family), specifically the 8-bit-quantized 4-bit variant running live on iPhone. The model streams output in real time at 40 tokens per second — described as well above the threshold for acceptable user experience. Grondin notes that even older iPhones can deliver around 20 tokens per second, which he argues is already useful for many application use cases. The Locally AI app, available free on the App Store, bundles this capability into a consumer-facing chatbot. The main friction for end users is initial model download size, typically 1–3 GB depending on the model chosen.

Beyond text generation, Grondin highlights that the broader MLX ecosystem now supports vision-language models (MLX VLM from 'Prince'), audio and speech-to-speech (MLX Audio), video generation, and omni models — positioning MLX as a comprehensive on-device AI runtime rather than a narrow inference tool.

Grondin also mentions the Locally AI app includes a very small liquid model at around 300–350 parameters, fast enough to run in iOS Shortcuts for text-processing automation tasks. On structured generation, he acknowledges it is not yet natively supported in MLX Swift LM, but third-party packages are emerging; tool calling, however, is already supported.

The talk closes with a significant announcement: Locally AI has been acquired by LM Studio, the desktop AI studio for running local models. LM Studio supports model downloads from Hugging Face, local server hosting, and OpenAI- and Anthropic-compatible response APIs, with both llama.cpp and MLX as selectable backends. Grondin frames this acquisition as expanding the reach of MLX-powered on-device inference across both mobile and desktop environments.