Run Frontier AI at Home — Alex Cheema, EXO Labs

Alex Cheema, co-founder of EXO Labs, delivers a technical workshop on running frontier AI models on local consumer hardware, arguing there is a 100x improvement still available across the full inference stack. The session is framed around a live demo running GLM 5.1 — a one-trillion-parameter model released the day before the workshop — across four Mac Studios connected by Thunderbolt 5, costing approximately $40,000 in hardware and achieving around 20 tokens per second.

Cheema opens by distinguishing inference from training, noting that while open-source efforts have focused on training transparency, the real barrier for democratizing AI is inference cost and capability. He argues that most AI inference is memory-bound at low batch sizes, not compute-bound, which has different hardware implications than the data-center training paradigm that favors raw FLOPS on Nvidia GPUs.

A central technical theme is kernel inefficiency. Cheema describes discovering that Qwen 3.5 running on Apple Silicon was running approximately 50% slower than theoretical maximum due to unnecessary separate kernel launches. By fusing these kernels, EXO recovered 30% performance — illustrating that even widely-used open-source inference stacks have significant untapped headroom.

On the networking side, EXO integrated low-latency RDMA to enable true tensor parallelism across Mac Studios. Prior to RDMA, inter-node latency was around 300 microseconds; with RDMA it dropped to single-digit microseconds. For a model like GLM 5.1 with 60 layers requiring two synchronizations per layer per token, the difference is roughly 36 milliseconds of saved communication overhead per token — enough to make tensor-parallel inference genuinely faster than single-node inference at this scale.

Cheema introduces heterogeneous hardware splitting as a key architectural strategy: running the compute-dense prefill phase on an Nvidia RTX Spark (approximately $4,000, ~4x more compute than a MacBook M4 Max) and the memory-bandwidth-intensive decode phase on a Mac with its larger, higher-bandwidth memory pool. The MacBook M4 Max has 546 GB/s memory bandwidth versus 273 GB/s on the Spark, but the Spark has roughly 4x more compute. The live demo shows this splitting achieving roughly a 2x end-to-end speedup for large prompts (around 100 KB), while providing no benefit for small prompts due to the quadratic prefill cost scaling with prompt size.

The EXO software architecture uses event sourcing and append-only logs across cluster nodes to handle dynamic node membership, ensuring in-flight inference requests survive node sleeps or disconnections. Cluster discovery is automatic over mesh networks, with Tailscale recommended for secure remote access.

On the broader market thesis, Cheema argues that within 18 months a $5,000 local inference box will deliver near-frontier performance for the majority of consumer use cases. He cites the "intelligence per joule" metric (improving ~5x over two years from hardware, ~3x from model improvements, compounding) and predicts use-case bifurcation: local hardware for the 90-99% of tasks with diminishing-returns intelligence requirements, cloud for high-compute scientific workloads. He dismisses on-disk inference as impractical and warns against treating auto-research loops as a scientific substitute, calling the slot-machine approach inadequate without a well-reasoned hypothesis and controlled experiments.

EXO Labs is planning to publish a public benchmark website within approximately one month, tracking thousands of configurations (different models, quantizations, pruning approaches) and plotting Pareto frontiers of quality vs. tokens-per-second by hardware budget, including the intelligence-per-joule metric.

Cheema also argues local inference economics improve further if test-time training (continual learning) becomes mainstream, since per-user personalized weights break cloud batching assumptions entirely, potentially giving local a 10x relative advantage over cloud pricing.