Why MLX — Prince Canuma, Neywa Labs

May 11, 202623:105,338 views

Summary

Prince Canuma, founder of Neywa Labs and a core contributor to Apple's MLX framework, delivers a talk at AI Engineer Europe 2026 making the case for on-device AI as a first-class deployment target rather than a fallback. The talk is grounded in a personal origin story: in 2020, Canuma's father lost his sight while Apple simultaneously released its first M1 chip, prompting Canuma to explore on-device vision models as an accessibility tool for someone in Africa with unreliable internet access. That accessibility-driven motivation led him to discover and contribute to MLX, an array framework for Apple Silicon that functions analogously to PyTorch or TensorFlow but is optimized for the unified memory architecture of Apple chips.

Three years of contribution have yielded tangible scale: MLX now has over 1.5 million downloads and more than 4,000 ported models. Neywa Labs maintains day-zero support for frontier open-source models — for example, Gemma 4 was available on MLX the same week it shipped from Google DeepMind. MLX powers inference inside LM Studio and Liquid AI's toolchain.

Canuma demonstrates several capability tiers live. Vision: the RFDetector model from Roboflow runs in real time on-device, detecting objects including a glass and performing background blur without any cloud call. The Gemma 4 26B vision-language model loads on a 96 GB VRAM Mac Studio and analyzes an uploaded image, reading and summarizing on-screen text. He also shows grounded visual reasoning for security and dash-cam analysis use cases.

Audio: Neywa's custom TTS model called Marvis generates speech in under 100 milliseconds. The stack supports automatic speech recognition (comparable to Super Whisper or WhisperFlow), text-to-speech, and a modular speech-to-speech pipeline in which each stage — ASR model, LLM, TTS model — can be swapped independently to match any hardware budget. Both Python and Swift APIs are supported.

Video generation: MLX Video can produce video from a text prompt on a MacBook with as little as 16 GB of VRAM. A community user chained sequential generations to produce a coherent story without a single cloud call.

The talk's technical highlight is Turbo Quant, a KV-cache compression method Canuma implemented publicly within 30 minutes of the paper's release (25 March, the tweet reached 700,000 views). Turbo Quant reduces KV cache memory by 4x, enabling 1 million token context to run fully on-device. At 300,000-token context, throughput nearly doubles while response quality matches the full-precision model on exact-match benchmarks.

On hardware, MLX uses the GPU — not the Neural Engine — because Core ML's private API surface makes Neural Engine access unreliable for developers today. Canuma signals that Apple may unify components at the M5 generation and expresses hope for improvements at WWDC. Real-time GPU monitoring can be done with Mactop, an open-source tool by Carson. For omni (image+audio+text) models, Canuma recommends Gemma 4 (E variants) for lighter workloads and Qwen3 Omni (~30B parameters) for higher capability, both runnable on-device.

The talk closes with a robotics demo: a Richie Mini robot powered by MLX Vision and MLX Audio performs real-time voice cloning of the Iron Man Jarvis voice and responds to spoken commands, illustrating the end-state vision — agents that can hear, see, and speak, running on an iPhone, iPad, Mac, or robot chassis without phoning home.

Key Takeaways

•MLX enables running models like Gemma 4 26B fully on a Mac or even an iPhone by leveraging Apple Silicon's unified memory, removing cloud dependency entirely.
•Turbo Quant reduces KV cache memory by 4x with near-identical response quality, making 1 million token context feasible on consumer Apple hardware today.
•A modular speech-to-speech pipeline (swap ASR, LLM, and TTS independently) lets you tune latency and quality to any hardware budget, from M1 to M3 Max.
•Marvis TTS generates audio in under 100 milliseconds on-device — comparable to cloud TTS latency but without a network round-trip.
•MLX uses the GPU, not the Neural Engine; Core ML's private API surface makes Neural Engine access unreliable for developers until Apple resolves it (expected WWDC).
•Day-zero model support means new frontier open-source models (e.g., Gemma 4) are available on MLX the same week they are released, so you are never waiting on a port.
•For omni (image + audio + text) tasks, Gemma 4 E-variants are the lighter option; Qwen3 Omni (~30B) offers higher capability — both run on-device.
•Video generation chains (using MLX Video) can be sequenced to produce a coherent narrative on a MacBook with 16 GB VRAM, no cloud call required.
•Use Mactop to monitor real-time GPU and CPU utilization during on-device inference — run `mactop` in terminal for an overlay view.
•On-device AI is not merely a cost optimization — for users in regions with poor connectivity, it is the only viable path; reliability arguments apply equally to robotics and local agents.
•Both Python and Swift APIs are available in MLX, enabling fully native iOS/macOS apps with on-device vision and audio intelligence.

Transcript

Good morning everyone. How many of you here have ever tried to run AI on your phone, on your MacBook? How was that experience? Was good? With MLX, yes. More or less? Okay. So, this talk today is for you. I'm going to show you how you can deploy and manage AI agents, or even voice agents if you will, completely on device using MLX. Today's agenda, of course, we're going to start with why on device. A lot of you use cloud code subscriptions and many other subscriptions. I want to convince you toda...