What Lies Beneath the API — Benjamin Cowen, Modal

Benjamin Cowen, a forward deployed machine learning engineer at Modal, delivers a talk arguing that fine-tuning is not an edge case for mature AI products — it is the inevitable destination. Drawing on Modal's position as a general-purpose serverless compute platform, Cowen observes a consistent pattern across a wide range of AI applications, from physics simulations and quantum chemistry to voice processing, LLMs, and agents: as products specialize and customer requirements tighten, companies must move beyond frontier API prompting toward custom model training.

Cowen frames the landscape as a spectrum. On one end sits the frontier API — fast to build with, powerful, and responsible for unlocking an era of accelerated growth, but entirely non-customizable beyond prompt engineering. On the other end is fully self-managed training, which historically demanded large dedicated clusters, isolation from production workloads, and significant infrastructure engineering investment. The talk argues that a new middle ground is emerging through serverless compute platforms that provide algorithm-level control without the full infrastructure burden.

Three concrete signals indicate a company is approaching the fine-tuning inflection point. First, API spend exceeds customer revenue — including prompt-optimization tricks like "caveman mode" (using sparse language to reduce token counts) — which indicates unit economics are not scaling. Second, evaluation scores have plateaued, suggesting the frontier model's general optimization objectives no longer align with the company's specific business logic. Third, latency and throughput requirements exceed what any shared inference endpoint can reliably deliver, particularly under enterprise contracts.

Cowen cites real customer results: Intercom reports beating their frontier API at one-fifth the cost; Pinterest claims orders of magnitude of improvement; and Decagon, a Modal customer, articulates the core misalignment clearly — frontier labs optimize for winning at everything, while product companies need to win only at their specific business logic.

The practical case is deliberately demystifying. Supervised fine-tuning can be implemented in approximately 300 lines of Python using modern open-source libraries that provide full algorithm control. Cowen contends that any team that has already built an agent harness and is collecting eval data already possesses the prerequisites to begin reinforcement learning training — the agent harness constitutes the rollout environment, and the eval data constitutes the training signal. Modal's unified APIs for sandboxes and GPU containers make scaling RL rollouts straightforward; one Modal customer scaled to 50,000 sandboxes solely for RL rollout during the prior quarter.

For model serving after training, Cowen points to vLLM, SGLang, and Triton Inference Server as viable options, all deployable with auto-scaling on serverless platforms to match production traffic patterns. The overall message is not that companies should train immediately, but that the timeline is 6–12 months rather than years — and that data collection and eval maturity are the critical prerequisites to begin building toward that moment.