大模型推理基础LLM inference fundamentals

Lesson 03 showed how the loop turns, round after round. This lesson drills one level deeper: what a transformer block actually computes, why autoregression works at all, and - most importantly - why the KV cache is exact rather than a "good enough" approximation. Understand this layer and you see why that loop can be so cheap.

decoder-only: what a block computes

llama-family models (like most LLMs today) are decoder-only. Unpacked, the data flows bottom-up in a straight chain: token ids are looked up into vectors, pushed through many identical blocks, then projected back to the vocabulary to score every candidate word. Inside one block there are two sub-layers - self-attention and the feed-forward network (FFN):

A word on why embeddings matter. They are not a plain "dictionary lookup into an integer"; each token maps to a point in a high-dimensional space where semantically similar words land close together - the starting point of the model "understanding" language. A raw embedding only carries "roughly what this word means", with no context; what makes it "read the whole sentence" is the dozens of blocks that follow. Think of the stacked blocks as a refinement pipeline: shallow layers capture local collocations and syntax (what modifies what, phrase boundaries), deeper layers compose more abstract semantics and long-range relations (coreference, logic, topic). Each layer injects more context into a token's vector; by the top, the last position's vector has distilled every clue about "what to say next", which the projection turns into logits.

Walk this "retrieval" through one concrete example: the current token scores 3 history tokens; a higher score means a higher weight after softmax, and the Values get summed by those weights into one vector.

An easily missed point: attention itself does not know word order - it only does similarity-weighted summing, so shuffling the same words and feeding them in gives the same result from pure attention. But language clearly cares about order ("dog bites man" vs "man bites dog"). Position information is added for exactly this: llama-family models commonly use RoPE (rotary position embedding), which does not simply add an "index vector" per position but rotates Query/Key by position, so two tokens' attention score naturally carries "how far apart they are". That is why the self-attention sub-layer above is tagged "with RoPE".

And the final output projection: it multiplies the top-layer token vector by a big "vocab-size x hidden-dim" matrix (lm_head) to get one score per vocabulary token - which is why the logits length always equals the vocabulary size. Many models also tie this with the bottom embedding matrix (weight tying), saving parameters and keeping "how inputs are encoded" consistent with "how outputs are scored".

Written as pseudocode, one block's forward pass is "two residual branches":

# one block's forward: two residual branches
def layer(x):                 # x: [n_tokens, n_embd]
    a = attn(rms_norm(x))     # tokens talk to each other here
    x = x + a                 # residual
    f = ffn(rms_norm(x))      # per-token non-linear mix
    return x + f              # residual

The causal mask: you may only look back

Since attention lets tokens reference one another, can token 5 peek at tokens 6 and 7 while writing? Never - during generation they do not exist yet. So the decoder adds a causal mask to attention: token i may attend only to positions <= i, masking out every "future" position. Drawn as a grid, it is a lower triangle:

Each row is "who one token may look at": a lit cell means attendable (no later than itself), a dim cell means masked (after itself). The implementation is direct: after computing attention scores, set every "future" score to -inf, then softmax turns those weights into 0 - as if unseen. This is what "autoregression" looks like mathematically - each position's output depends only on itself and what came before, never leaking the future.

Because training and inference share the same forward pass, llama.cpp has no "two codebases": whether prefill feeds in hundreds of tokens at once or decode feeds just 1 new token, both run the same compute graph, differing only in "how many tokens in this batch, and which positions to output". With that in mind, revisit lesson 03's seven-step flow and you see why a single llama_decode can carry both rhythms.

In llama.cpp this is assembled into the compute graph by build_attn / build_attn_mha in src/llama-graph.cpp, with the mask applied during softmax via ggml_soft_max_ext. Simplified:

// attention scores + causal mask (cf. build_attn / build_attn_mha)
kq  = ggml_mul_mat(ctx, k, q);            // scores [n_kv, n_q]
kq  = ggml_soft_max_ext(ctx, kq, mask,    // mask: causal -inf on j>i
                        scale, max_bias);
kqv = ggml_mul_mat(ctx, v, kq);           // weighted sum of values

(Per this repo's source as of 2026-06-15; the real implementation lives in llama-graph.cpp; only the trunk is shown, omitting scaling and multi-head details.)

Autoregression + why the KV cache is exact

Draw "predict the next from what we already have" as a loop and you get the autoregressive cycle:

Note the feed-back arrow at the end - the newly sampled token is appended to the sequence and becomes the next round's input. This raises the lesson's core question: to write each next word, must we recompute attention over all previous words? No - and the reason is hidden in the causal mask.

In attention each token computes three things: Query (the vector that "asks"), Key (the "tag" being matched), and Value (the "content" carried). To add token n+1, it takes its Query, compares against every historical token's Key, and sums their Values by the resulting weights. The crucial point: a historical token's K and V depend only on itself and its position, and the causal mask guarantees it cannot see any later token - so no matter how many new tokens are appended afterwards, those earlier K/V never change by a single number. Since they never change, they can be computed once, stored, and reused forever. That is the KV cache:

Look at the two rows - step n+1 differs from step n by just one highlighted new cell on the right; everything on the left is untouched old values. So the KV cache is not an "accuracy-for-speed" approximation; it is a mathematically exact reuse: the cached values equal what "recompute every step from scratch" would produce, bit for bit. What it saves is repeated work, not precision. And since predicting the next word needs only the last position's row of output, we read logits at the end:

logits = llama_get_logits_ith(ctx, -1);  // last position only, returns n_vocab scores

Of course, the KV cache is not free: it stores one K and one V for every layer and every historical token, so its footprint grows linearly with context length. At tens of thousands of tokens it eats a sizeable chunk of memory - one reason "context windows" always have a ceiling and long contexts are so hardware-hungry (allocation, writes, and reuse all live in src/llama-kv-cache.cpp). So engineering must both save compute with it and shrink its size - GQA/MQA below do exactly that - a tension to keep balancing.

Going deeper (optional)

Three questions below; open them if you want depth, skip them if you only want the main line.