Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci

April 8, 202640:356,095 views

Summary

Stefano Fiorucci, AI/Software Engineer at Deepset (maintainer of the Haystack open-source LLM framework), presents a practical software engineering walkthrough of building Reinforcement Learning (RL) environments for language model evaluation and training. The talk is motivated by the limitations of the supervised fine-tuning (SFT) paradigm — as Ilya Sutskever noted at NeurIPS 2024, pre-training is no longer sufficient to drive continued model quality improvements. OpenAI's O1 model demonstrated that reinforcement learning with chain-of-thought improves performance with more compute, and DeepSeek R1 shed implementation light by using RL with Verifiable Rewards (RLVR) and GRPO, a lighter algorithm compared to PPO, to avoid the high cost of curated SFT data.

Fiorucci introduces the core conceptual mapping: in RLVR, the LLM is the agent, the environment encapsulates data, scoring harnesses, and game logic, and rewards are verifiable outcomes (correct answers, won games, successful tool calls). He argues this enables models to exceed statistical expert imitation — discovering strategies that human-curated examples cannot teach.

The centerpiece tool is Verifiers, an open-source library by Prime Intellect. Verifiers provides base classes for single-turn environments, multi-turn environments, and tool-using agent environments (ToolEnv), with an OpenAI-compatible API abstraction supporting local models via vLLM, OpenRouter, or OpenAI. It integrates with training frameworks including Prime RL, TinkerRL, and SkyRL. The library also includes an MCPEnvironment class for connecting to Model Context Protocol servers, a StatefulToolEnv for persistent per-rollout state, and support for recursive language model (RLM) inference patterns.

The main experiment involves training LFM-2 (a small open model by Liquid AI) to play Tic-Tac-Toe. Baseline evaluation showed LFM-2 struggled severely with format compliance and valid moves, while GPT-4o-mini was a strong player. The training pipeline proceeded in two phases: (1) SFT warm-up — 200 synthetic games generated by GPT-4o-mini (filtered to remove losing games) to teach format and valid move syntax, run on a 96 GB GPU in minutes using Prime RL; (2) GRPO/CISPO reinforcement learning using the Verifiers RL trainer with two GPUs (one for inference, one for training).

Key engineering details include: opponent skill curriculum using minimax with configurable random-move probability (ranging 20–70% for training stability), stratified sampling to ensure balanced opponent difficulty across batches, per-turn deterministic seeds derived from example seed and board state to reduce noise across GRPO rollouts, and a soft invalid-move penalty (−0.1 per infraction, capped at 8 turns) rather than an immediate game-ending loss. A second RL phase with tighter opponent skill (0–25% random moves) and elevated temperature prompted an initial exploratory drop in rewards followed by recovery to new highs.

Results: after full training, LFM-2 dominates random opponents and draws 85% of the time against an optimal opponent — outperforming GPT-4o-mini (the teacher model used to generate SFT data) against optimal play. Invalid moves dropped to near zero.

Fiorucci shares hard-won lessons: batch size is critical for stable GRPO training (values below 256 caused collapse in this environment); hidden biases in environment logic (e.g., a deterministic tie-breaking in a minimax implementation) can cause a model to memorize a specific opponent rather than learn general strategy; starting from a non-reasoning instruct model is often better than a reasoning model when GPU budget is limited, since reasoning models produce long thinking traces that get truncated; and developers should resist prematurely stopping training runs — RL progress is slow, and a walk is recommended while training runs.

Key Takeaways

•Use Verifiers (github.com/primeintellect-ai/verifiers) to build modular RL environments as Python packages that plug into any OpenAI-compatible model endpoint.
•Run a SFT warm-up phase before RL training to teach a small model basic format compliance — even 200 filtered synthetic examples can dramatically reduce invalid outputs.
•Set batch size generously (256+ games per weight update) for GRPO/CISPO training; small batch sizes with diverse opponent skill cause unstable training or model collapse.
•Use stratified sampling over opponent difficulty ranges to ensure every training batch contains a balanced mix of easy and hard games, preventing reward variance from destabilizing optimization.
•Derive per-turn seeds from example seed + board state to make GRPO rollout comparisons fair — randomness in the environment should not be the source of reward differences between trajectories.
•Audit your environment logic for hidden biases: a deterministic tie-breaking rule in a minimax implementation caused the model to memorize one opponent type rather than learn generalizable strategy.
•Prefer starting from an instruct (non-reasoning) base model when GPU is limited — reasoning models emit long thinking traces that get truncated under tight token budgets, wasting compute and potentially degrading intelligence.
•Use temperature increases carefully to break out of learned strategy plateaus; expect an initial reward drop as the model explores, and do not prematurely stop the run.
•RLVR lets small, specialized models outperform much larger closed models on narrow tasks: after RL training, LFM-2 outperformed its GPT-4o-mini teacher against optimal Tic-Tac-Toe opponents.
•Always do qualitative rollout inspection after training — programmatic benchmark scores can be misleading if the environment has biases that inflate win rates without genuine generalization.

Transcript

Hello everybody and welcome to Let's Learn Wander engineering reinforcement learning environments. A few words about me. I am Stefano Fiorucci, a AI and software engineer. By day, I work on AI orchestration at Deepset, where I develop Haystack, an open-source LLM framework. By night, I love tinkering with small language models, fine-tuning, and reinforcement learning. Today, I'm going to talk about reinforcement learning environments for language models evaluation and training. This has been a h...