← Back to the generator

About

LLM From Scratch

I'm a Computing and Mathematics (COMA) student at Queen's University. This project is a GPT-style language model I wrote without PyTorch or TensorFlow. Every layer, every gradient and the training loop are implemented in NumPy.

Why from scratch

In my earlier projects, fastai and PyTorch handled the hard parts for me. That made me productive, but it also hid how the models worked. I wanted to understand language models in a practical way, including the complexity that libraries abstract away. Implementing the forward pass, backpropagation and optimizer myself meant I had to understand each piece well enough to derive its gradient.

How I built it

I worked through the model one component at a time and kept a log as I went.

  1. Day 1

    Embeddings and the forward pass

    Character vocabulary, token and position embedding matrices, and getting a prediction out for an input character.

  2. Day 2

    Manual backpropagation

    Deriving and coding the gradient of every operation, then updating weights with stochastic gradient descent.

  3. Day 3

    Self-attention

    Query, key and value projections, causal masking, softmax and its Jacobian, wrapped into multi-head attention.

  4. Day 4

    Assembling the transformer

    Stacking blocks with residual connections, LayerNorm and a feed-forward layer into a model that actually trains.

  5. Day 5

    The interface

    A FastAPI streaming endpoint and this Next.js front end, with temperature, length and an attention heatmap.

Architecture

The model follows the decoder-only transformer from Attention Is All You Need (2017). Each block applies multi-head causal self-attention, a residual connection and LayerNorm, then a ReLU feed-forward layer with a second residual connection and LayerNorm. The output projection reuses the embedding matrix, and sampling divides the logits by a temperature before the softmax.

Parameters
368,832
Tokenization
character-level, 131 symbols
Transformer blocks
6
Attention heads
8 per block (head size 8)
Embedding size
64
Feed-forward
64 → 256 → 64, ReLU
Context window
1,000 characters
Normalization
post-norm LayerNorm, 2 per block
Output layer
tied to the token embedding
Dependencies
NumPy

Writing the backward passes was the core of the work. Attention needs the gradient through the softmax Jacobian and the masked score matrix. LayerNorm needs the gradient through the mean and variance. Each residual connection sends gradients down two paths that have to be summed. Each layer caches its activations on the forward pass so the backward pass can use them.

Training

I trained the base model on the source code of the Python requests library, about 377,000 characters, using random 128-character windows in batches of 32. Training ran on a CPU for over 24 hours. Cross-entropy loss fell from about 18 at initialization to 0.66, which means the model puts an average probability of roughly e−0.66 ≈ 52% on the correct next character.

The training loop uses a linear warm-up followed by either cosine or linear learning-rate decay, both implemented by hand.

LoRA fine-tuning

Just before this project, I implemented the LoRA paper (Low-Rank Adaptation of Large Language Models) on a pre-trained Hugging Face model. Here I rebuilt it for my own architecture: rank-8 adapters (α = 1) wrap the query and value projections in all 48 attention heads. The base weights stay frozen and only the adapters train, which is 55,296 parameters or about 15% of the model.

I fine-tuned on a 3.6 MB corpus of general Python code, with 100 warm-up steps and a learning rate decaying from 3×10−4 to 3×10−5. The demo runs the base model with these adapters applied.

Serving the model

The NumPy model runs in a FastAPI service in a Docker container on Hugging Face Spaces. Because the model generates one character at a time, the API streams each character to this page as soon as it's sampled, using Server-Sent Events. After the last character it runs one forward pass over the full text and returns the attention weights averaged across all 48 heads, which is what the heatmap shows.

What I learned

  • How attention, normalization and residual paths behave numerically, not just conceptually.
  • How to read a research paper and turn it into working code, twice: the transformer and LoRA.
  • Numerical stability is part of the math in practice: subtracting the max before a softmax and adding an epsilon inside the log.
  • Why frameworks make the design choices they do, such as weight tying and caching activations for the backward pass.
  • The cost of training without a GPU, and how much model size and context length matter in practice.

Limitations

At 369K parameters and character-level tokenization, the model learns Python's look (indentation, keywords, identifiers and punctuation) but not its meaning. Its output is plausible-looking code, not working code. A next step I'd planned was moving to a more modern design, starting with rotary positional embeddings (RoPE).

Where it fits

  1. 2025

    fast.ai Practical Deep Learning

    Course notebooks: image models, neural nets from scratch, random forests.

  2. Jun 2, 2025

    duck_or_desert

    First trained model: a ResNet-18 image classifier using fastai.

  3. Jun 2025

    LeanAI

    Scraped and LLM-labelled my own dataset; deployed an 11-model regression ensemble.

  4. Jun 17, 2025

    lora-from-scratch

    Implemented the LoRA paper on a pre-trained Hugging Face model.

  5. Jun–Jul 2025

    llm-from-scratch

    This project. Removed the frameworks and built the transformer myself.