The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

Filip Makraduli, an AI researcher and engineer at Superlinked, presents the company's open-source small model inference engine — dubbed SAI (Superlinked AI Inference) — framing it as infrastructure that nobody else built, so his team did. The talk originated from a personal lesson: Makraduli published a Substack article explaining flash attention and memory- vs. compute-bound processes, but readers pointed out he had overlooked inference — the production runtime layer that makes models fast in the real world. This gap in his own understanding prompted him to join Superlinked's infrastructure team and actually build a system from scratch.

The core motivation centers on why small model inference matters for agentic workflows. Makraduli references the Chroma research on "context rot" — the empirically documented degradation of LLM output quality as context windows grow — and argues that small specialized models serve as preprocessing layers to combat this. Named entity recognition models (e.g., GLiNER), rerankers, and embedding models such as Stella can preprocess and filter input data before it reaches a large language model, reducing token counts and improving agent effectiveness. Andrej Karpathy's work on graph-based knowledge bases using NER-generated ontologies, and Chroma's own context-management model, are cited as community validation of this architectural pattern. Superlinked has already deployed this approach in production: the inference engine was used as a tool-calling solution for taxonomy classification for an e-commerce client.

Makraduli then systematically dismantles the "just add more GPUs" misconception. Small embedding and reranker models — Stella, GLiNER, ColBERT-family models — occupy only a few gigabytes of VRAM each. Provisioning a dedicated GPU per model leaves the hardware mostly idle. The solution is dynamic model loading with hot-swapping and a least-recently-used (LRU) eviction policy, allowing many models to share one GPU with high utilization and low cost.

The second gap is that existing tools such as vLLM handle the model-level runtime but leave production concerns — routing, auto-scaling, queuing, Prometheus metrics, Grafana dashboards, GPU provisioning — for teams to wire together themselves. No open-source end-to-end solution existed for small-model inference at this scope.

SAI's architecture is framed as the "yin and yang" of inference. The yin (model support) addresses the reality that Hugging Face hosts close to 3 million models with divergent architectures. BERT, Qwen, and ModernBERT all differ in attention implementations (standard vs. grouped query attention), positional embeddings (absolute lookup vs. rotary/RoPE), QKV fusion compatibility, normalization schemes, and output types (single vector vs. multi-vector late interaction as in ColBERT, or scalar scores as in cross-encoders). SAI re-implements the forward pass for each model family, adds variable-length flash attention to eliminate padding waste in token-based batching, and supports late interaction models and cross-encoders alongside standard bi-encoders.

The yang (infrastructure) provides a three-primitive API — encode, score, and extract — sitting behind a router and dynamic queuing layer that dispatches to pools of spot and full GPU instances. Auto-scaling is driven by KEDA with Prometheus metrics. Helm charts, Docker images, and Terraform configurations are published, so deploying a new model is a config change and a `terraform apply`. Integration tests with Chroma, Qdrant, Weaviate, and LanceDB have validated the system with vector database partners. The engine is fully open-sourced on GitHub under the Superlinked organization.