OpenRAG: An open-source stack for RAG — Phil Nash

Phil Nash, a developer relations engineer at IBM, presents OpenRAG — an opinionated, open-source RAG stack built from three existing open-source components: Docling for document processing, OpenSearch for vector and keyword search, and LangFlow for visual orchestration and agentic pipelines. The talk opens by reframing the "RAG is dead" narrative: Nash argues that while large context windows can technically replace retrieval, the cost of loading millions of tokens per query and the genuine complexity of real-world data make RAG an enduring practical necessity.

On the ingestion side, Nash walks through Docling's multi-pipeline architecture. A simple pipeline handles plain-text formats (Markdown, HTML, Word). An ASR pipeline processes audio and video. PDFs — identified as the hardest document type — are handled by either a standard pipeline composed of multiple small, focused models performing layout analysis, table extraction, OCR, and image description, or a newer VLM pipeline using IBM's Granite DocLing 258 million-parameter vision model that processes everything in a single pass. Docling outputs an intermediate DocTags XML-like representation, from which hierarchical chunks are generated using structure that the parser has already identified. This contrasts with naive fixed-size chunking by preserving semantic document hierarchy.

OpenRAG is intentionally non-prescriptive about embedding models. It supports OpenAI, WatsonX, and Ollama for local, air-gapped operation. Crucially, it supports indexing over multiple embedding models simultaneously — useful for zero-downtime migration between embedding models, though Nash notes this comes with a query-speed trade-off. By default, OpenRAG does not use OpenSearch's built-in HNSW or IVF vector indexes; instead it uses the JVector KNN plugin, an open-source vector index based on the DiskANN architecture that supports live indexing and does not require the entire index to reside in memory, which improves scaling characteristics for large datasets.

On the generation side, OpenRAG uses agentic retrieval within LangFlow rather than a single-pass nearest-neighbor lookup. An LLM agent receives the user query along with tools it can invoke to perform as many OpenSearch queries as needed, deciding autonomously what to search for and how to interpret results. The demo shows the system running entirely on local models via Ollama — specifically Granite 4 3B as the language model and Qwen3 0.6B for embeddings — demonstrating fully offline capability. The LangFlow interface exposes chat flows, knowledge ingestion pipelines, knowledge filters (leveraging OpenSearch's filtering API for document-scoped queries), and cloud connector integrations (Google Drive, SharePoint, OneDrive via Google OAuth).

Nash demonstrates live customization inside LangFlow's drag-and-drop editor, adding a guardrails component inline between the chat input and the prompt template, parsing the guardrail result, and routing failures to a dedicated error output — illustrating that the LangFlow visual layer makes pipeline modifications accessible without rewriting Python or modifying infrastructure. The agent also includes an MCP server (for URL ingestion as an internal LangFlow flow) and a calculator tool, since Nash argues language models should not be trusted for arithmetic.

OpenRAG additionally exposes an MCP server endpoint so the retrieval and agent capabilities can be consumed by external agents. The project is at version 0.4.0, built with a Next.js frontend and Python backend, and is fully open source on GitHub. Nash closes by reiterating that RAG is not solved universally — every dataset, user base, and interaction pattern demands tuning — but OpenRAG is designed to provide a high-quality, extensible baseline so teams can focus on that tuning rather than rebuilding foundational infrastructure.