Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

Mozhgan Kabiri Chimeh, Developer Relations Manager at NVIDIA, presents a data-backed exploration of running large language models locally on the NVIDIA DGX Spark workstation. The talk challenges the default assumption that LLM workloads must live in the cloud, arguing that local compute is increasingly viable for production-grade prototyping. She frames the core engineering tension: developers either exhaust memory or lack the right software stack, forcing everything to the cloud — and then face scheduling delays, unpredictable costs, and data residency concerns.

The DGX Spark is powered by the GB10 Grace Blackwell Superchip, combining CPU and GPU in a unified memory architecture with 128 GB of unified memory and NVFP4 support. This enables models up to approximately 200 billion parameters to run locally on a form factor that fits on or under a desk. Critically, it runs the same NVIDIA AI software stack used in data center and cloud environments, meaning workflows can be promoted from desktop to production with minimal changes.

Kabiri Chimeh built a fully reproducible benchmarking harness using vLLM serving open-source models ranging from 1.5B to 14B parameters. The methodology enforces environment isolation via Docker containers, three mandatory warm-up runs before measurement, and background GPU metrics logging at one-second intervals. Each run produces a versioned artifact directory (timestamped with a sanitized model ID) containing full model endpoint responses and benchmark metrics. The streaming response timestamping logic is shown explicitly — capturing time-to-first-token by timestamping the very first chunk returned by the vLLM server.

The throughput benchmark results (completion tokens per second) reveal a clear hierarchy. The 1.5B instruct model delivers 61.73 tokens/sec. The 14B base model drops sharply to 8.40 tokens/sec. The 14B NVFP4 quantized model achieves 20.19 tokens/sec — more than twice the unquantized 14B and still faster than average human reading speed. This is the key result: NVFP4 4-bit floating-point quantization effectively doubles usable throughput at the 14B scale without sacrificing model intelligence.

On time-to-first-token, the NVFP4 14B model is 3.4 times faster than the unquantized 14B base model. Kabiri Chimeh draws a sharp distinction between memory capacity and memory bandwidth: having 128 GB of unified memory allows fitting large models, but actual throughput is governed by data movement efficiency. NVFP4 increases intelligence per byte, which is why it dominates the responsiveness metrics. She describes this as the 'critical engineering sweet spot' — a sophisticated model that still feels as responsive as a much smaller one.

The practical guidance distilled from these experiments covers three use cases where DGX Spark is the right choice: steady-state workloads requiring cost predictability, privacy-sensitive data that cannot leave local infrastructure, and rapid prototyping cycles where iteration speed depends on uncontested compute access. The recommended workflow is to build and fine-tune locally using the identical NVIDIA software stack, then promote to data center or cloud when ready to scale. Resources and playbooks are available at build.nvidia.com/spark.