Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

Angelos Perivolaropoulos, a research engineer and speech-to-text team lead at ElevenLabs (responsible for the Scribe V2 transcription model), delivers a hands-on workshop on training a small GPT-2-style language model from scratch using only PyTorch, without any pre-trained weights or high-level Transformers library abstractions. The session is explicitly inspired by Andrej Karpathy's nanoGPT project and positions itself as covering roughly 80% of what real research engineers do in large AI labs.

The workshop is structured around four fundamental building blocks: the tokenizer, model architecture, training loop, and inference. For the tokenizer, the presenter opts for character-level tokenization over byte-pair encoding (BPE) because the Shakespeare training dataset contains only 65 unique characters, yielding just 4,225 possible bigrams — a tractable space for a small model. He explains that BPE tokenizers (used in production models like GPT-4) learn common character patterns from training data, but a 50,000-token BPE vocab would add ~19 million parameters to the embedding table alone — more than three times the rest of the model — making it impractical for a 10 million parameter training exercise.

The model architecture follows GPT-2 closely: 6 transformer layers, 6 attention heads, an embedding dimension of 384, and a sequence (block) length of 256 tokens. Total parameter count is approximately 1.8 million, broken down as roughly 25K for token embeddings, 98K for positional embeddings, and the majority in the transformer blocks (attention projection at ~590K per layer, MLP at ~1.2M per layer). The presenter walks through the GPT class structure in PyTorch, explaining multi-head self-attention, MLP feed-forward blocks, residual connections (X = X + attention rather than X = attention), and layer normalization as a mechanism to prevent activation explosion across layers.

The training loop covers next-token prediction via cross-entropy loss, using a batch size of 64 sequences of 256 tokens each. The learning rate schedule uses a warm-up phase over 100 steps followed by cosine decay to a maximum of 5,000 steps, implemented with the AdamW optimizer. The presenter describes expected loss milestones: starting at ~4.17 (natural log of 65, random baseline), dropping to ~3.3 when the model learns character frequencies, ~2.5 for common subwords like 'th', ~1.5–2.0 for real words, and ~1.0–1.2 for coherent Shakespeare-style text. Overfitting was observed at ~2,400 training steps on validation loss. Checkpointing every 1,000 steps and monitoring both train and validation loss are emphasized as critical debugging practices.

For inference, the presenter contrasts greedy decoding (deterministic, poor creativity, appropriate for ASR) with temperature sampling (typically 0.7 for text generation) combined with top-K sampling to prevent highly improbable tokens from being selected. Seeds are recommended even with greedy decoding for reproducibility.

The Q&A covers several advanced topics. On reasoning models: base models and reasoning models share the same architecture; the difference is post-training with high-quality chain-of-thought data sourced from labeling companies like Scale AI. On scaling: increasing context length is non-trivial — you cannot simply raise the block size parameter without architectural changes; this is the core challenge labs like Google DeepMind solved for Gemini's 1 million context window. On multimodal models: video or audio encoders produce hidden-state vectors of the same dimension as the language model's embedding space, which are then prefixed into the transformer's input sequence, bypassing tokenization for non-text modalities. On audio tokenization at ElevenLabs: mel spectrograms are the standard intermediate representation; audio tokenizers are trained on domain-specific corpora (voice data yields different tokens than music data); and losses like L2 on mel spectrograms or KL divergence for model distillation replace cross-entropy in audio settings.