spec · gpt-from-scratch · research · 2025
Decoder-only GPT, from scratch
A small transformer built and trained in raw PyTorch to learn the internals
- Role
- Sole engineer · learning project
- Domain
- Deep Learning / Fundamentals
- Year
- 2025
- Status
- research
stack →PyTorchNumPyTransformers (theory)
no interface capture on record
Notes ↗A from-scratch implementation of a decoder-only transformer — tokenizer, learned embeddings, multi-head scaled dot-product attention, pre-norm residual blocks, and the full logits→softmax→sampling loop — plus a skip-gram embedding model trained by hand.
01 · Problem
You do not really understand attention until you have written the backward pass and watched the loss move. Libraries hide exactly the parts worth learning.
02 · Approach
- 01Implemented Q/K/V as learned projections, softmax(QKᵀ/√dₖ)V attention, and multi-head concatenation without nn.Transformer.
- 02Pre-norm LayerNorm placement, residual connections as refine-not-overwrite, FFN as per-token refinement.
- 03Trained a skip-gram embedding matrix from scratch, verified with loss curves and cosine similarity between related tokens.
- 04Sampling controls — temperature, top-k, top-p — implemented as post-hoc transforms on the logits.
03 · Outcome
- Working generation loop on a small corpus.
- Self-corrected several misconceptions in the process — e.g. the FFN refines the hidden state, and the next layer computes fresh Q/K/V from that.