The transformer forward pass, token by token

A language model does not read text. Everything downstream of the prompt is arithmetic on vectors, and the first job is turning characters into vectors the model can work with.

Text becomes token IDs, then rows

The tokenizer splits text into known pieces and maps each to an integer from its vocabulary. "I love dogs." might become [2, 3, 5, 6]. Those integers carry no meaning — a token ID is just the row number for that token in the embedding matrix.

raw text
  -> tokenizer          split into known pieces
  -> token IDs          integer per piece
  -> embedding lookup   select one matrix row per ID
  -> vectors

The embedding matrix is vocab_size × embedding_dimension — one row per token, one learned number per dimension. A 50,000-token vocabulary with 768 dimensions is a 50,000 × 768 matrix. The lookup is not a similarity search; it is a plain row selection. Those rows started random and became useful only because gradients updated them during training. At inference the matrix is frozen and only read.

The model's token embedding layer and a standalone sentence-embedding model are different things. Token embeddings are internal model inputs; RAG/search embeddings are external vectors an application stores and queries.

Self-attention makes the vector context-specific

The row for "bank" is the same whether the sentence is "river bank" or "bank account". Self-attention is the learned mechanism that resolves this: it rewrites each token's vector as a weighted blend of the other tokens' contributions, so "bank" ends up geography-flavored in one sentence and finance-flavored in the other.

For every token the model computes three projections from its current hidden state:

• Q (query) — what this token is looking for
• K (key) — what this token offers for matching
• V (value) — the information this token contributes when attended to

Q, K, V come from learned weight matrices (Wq, Wk, Wv). The model never learns a fixed list of "relevant words"; it learns those projections, and relevance is computed fresh for each sentence.

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

QKᵀ            raw similarity scores
/ √dₖ          scale — keeps large dot products numerically stable
softmax        scores -> weights that sum to 1
· V            weighted mix of value vectors
= contextual vector

For "the river bank flooded", updating "bank": raw scores might be the 0.3 / river 4.2 / bank 1.0 / flooded 2.1. After softmax the weights concentrate on "river" (~0.75), and the new "bank" vector is 0.02·V_the + 0.75·V_river + 0.08·V_bank + 0.15·V_flooded. That output is the contextual embedding.

Multi-head attention runs this in parallel

One head learns one way of relating tokens. Multi-head attention gives each head its own Wq/Wk/Wv, all reading the same input. No head is assigned a job — each starts with different random weights, produces different attention patterns, and specialization (if it appears) emerges because useful patterns reduce loss. The head outputs are concatenated and passed through a learned linear projection.

hidden size 4096, 32 query heads  ->  head dimension = 4096 / 32 = 128
each head works with length-128 Q/K/V vectors
32 heads × 128 = 4096 back out

A block refines; it does not overwrite

A transformer block alternates communication (attention, across tokens) and private refinement (a feed-forward network, per token). Two structural pieces keep deep stacks trainable:

• Residual connections: output = x + F(x). The block adds a same-shaped learned correction to the original vector instead of replacing it. Information is preserved and gradients flow.
• Pre-norm LayerNorm: normalization is applied inside the block, before attention and before the FFN, because each transformation can change the representation's scale.

x
  -> LayerNorm -> Attention  -> + x        (cross-token update)
  -> LayerNorm -> FFN        -> + x        (per-token update)
  -> next block

The FFN expands then contracts (e.g. 768 → 3072 → 768): expand for richer intermediate features, non-linear activation, contract back to the model dimension. It does not act on Q/K/V — those are temporary projections used inside attention. The FFN refines the hidden state; the next layer computes fresh Q/K/V from that refined state.

Stacking many blocks means many rounds of the same operation under one objective (next-token prediction). Different abstraction levels emerge — basic token relationships, then phrase and sentence structure, then more abstract patterns — because each layer refines the previous layer's output, not because layers have separate goals.

text
  -> tokenizer -> token IDs -> embedding lookup -> + positional
  -> Q/K/V -> attention scores -> weights -> weighted value mix
  -> multi-head concat + projection
  -> residual + pre-norm + FFN, ×N layers
  -> contextual representation