What actually happens at inference

Training and inference run the same forward pass. The difference is the loop around it.

The generation loop

1. 01Run the prompt through the model.
2. 02Take the final hidden state, project it to the vocabulary → logits (one raw score per token).
3. 03Softmax the logits → a probability distribution.
4. 04Sample one token from that distribution.
5. 05Append it to the sequence and repeat.

The model does not "decide what to say" and then pick words. The reasoning-like behavior is entirely in the hidden states and the resulting distribution over next tokens. There is no separate planning step.

Decoding parameters do not change the model

Temperature, top-k, and top-p operate after the model has produced logits and before sampling. They shape how you draw from the distribution — they do not help the model predict.

• Temperature scales the logits: low → sharper and more deterministic, high → flatter and more random.
• Top-k keeps only the k highest-probability tokens.
• Top-p keeps the smallest set of tokens whose probability mass reaches p.

transformer -> logits
  temperature   scales logits
  top-k / top-p filters candidates
  softmax       -> final probabilities (renormalized)
  sample        -> next token

The KV cache

Every layer computes Q, K, V for every token. Without a cache, each new generated token would force recomputation of K and V for the entire sequence so far. The KV cache stores the previous tokens' K and V tensors so only the new token's projections are computed.

Prefill:  prompt tokens -> compute K/V -> store in cache
Decode:   new token -> compute Q/K/V
          Q_new compares against cached K
          attention pulls from cached V
          new K/V appended to the cache

It caches K and V, not Q, because the current token's Q searches the cached keys. It holds numerical tensors for the current sequence — system instructions, any conversation history sent in the request, the user message, and tokens generated so far — not raw text, and it is temporary to one request. A new API call usually means resending the history.

GQA, MQA, and cache size

Architecture decides how many KV heads exist, which decides how large the cache gets. With N query heads and M key/value heads:

• MHA — M = N. Every query head has its own K/V head. Largest cache.
• GQA — M < N. Groups of query heads share a K/V head. Smaller cache, minor quality cost.
• MQA — M = 1. All query heads share one K/V head. Smallest cache.

KV cache grows with:
  tokens × layers × KV heads × head dimension × concurrent requests

Quantization

Lower precision stores approximate values (0.734829 → 0.735), and several nearby values can collapse to the same stored number. It shrinks the model and the cache and speeds inference; it costs some accuracy, and the impact depends on what gets quantized (weights, activations, or KV cache) and how aggressively.

Context window

The context window is the maximum number of tokens the model can attend to for one prediction — a working-memory window, not permanent memory and not a limit on learned knowledge. It is bounded by the model's maximum length and by real resources: KV-cache memory, prompt-processing compute, per-token decode compute, and memory for other simultaneous requests.

KV caching removes repeated recomputation. It does not remove the memory or attention cost of a long context. When the conversation exceeds the window, older content has to be dropped, summarized, or retrieved back in.

One model, many requests

A served model shares its weights, embedding table, layers, and loaded GPU memory across all requests. What is per-request: the input tokens, the prefill pass, the KV cache, the decode loop, and the output. The server never mixes conversation state between users.

request A -> prefill A -> KV cache A -> decode A -> output A
request B -> prefill B -> KV cache B -> decode B -> output B
(weights shared, sequence state separate, GPU steps often batched)

This works because the weights are read-only during inference — every request reuses the same parameters, and the private state lives in each request's own cache and token stream. Batching groups requests into one GPU step for hardware efficiency while keeping each sequence logically separate.

• Throughput — total work the system completes
• Latency — how long one request takes
• TTFT — time to first token
• Tokens/sec — generation speed once streaming