核心算子Core operators

A compute graph is built from operators. This lesson picks the core few in a transformer - matmul mul_mat, normalization rms_norm, position encoding rope, masked soft_max_ext - to see what each computes, how input and output shapes line up, and where an operator "actually lands and computes" on the CPU. This lesson strings L04's attention math and L09/L10's graph and execution together with concrete operators.

The last three lessons were all about "containers and flow" - how memory is placed (L08), how the graph is built (L09), how the graph runs (L10) - but never touched "what each operator actually computes". This lesson fills that in. To be clear: we will not pore over the loop of some matmul line by line (that is Part 6's kernel lesson); instead, from the height of "being able to read and build networks", we nail down four things - what these operators do, how shapes line up, how attention is assembled from them, and where they actually land and compute. After this lesson, the graph-building code in llama.cpp becomes mostly readable - you can tell what each line is assembling.

The number-one operator: matmul mul_mat

In a neural network, the vast majority of compute goes into matrix multiplication - attention, feed-forward, output projection are all essentially matmuls. So ggml_mul_mat is the undisputed number-one operator. Its shape rule is the one thing this lesson is most worth memorizing:

See the rule in action with a small concrete example first: each output number is one row of A times one column of B - multiplied position by position, then summed - so the inner dim k vanishes and only each side's other dim is left.

Look at the two highlighted k: a and b must be equal on ne[0] (the inner dim), this equal dim is "eliminated" in the multiply, and the result's shape is formed from each one's "other dim". In source (ggml_mul_mat / ggml_can_mul_mat in ggml/src/ggml.c):

// assert: a's inner dim == b's inner dim (ne[0] equal)
GGML_ASSERT(a->ne[0] == b->ne[0]);          // k must match, else graph-build errors
// result shape: take a's "other dim", b's "other dim", high dims from b
ne = { a->ne[1], b->ne[1], b->ne[2], b->ne[3] };  // result type is always F32

Why is matmul so important it deserves its own section? Because it is both heavy and frequent. A 7B model does hundreds of matmuls per generated token, each a thousands-by-thousands matrix multiply - the vast majority of the model's parameters (those several GB of weights) exist as "the weight matrix in a matmul". So almost everything you have learned ultimately serves matmul: quantization (L06) to make weight matrices smaller and faster to move, backends (L07) to compute matmuls faster, memory reuse (L10) to make room for matmul intermediates. Understand mul_mat and you understand the main battlefield of inference.

To wrap up the matmul-shape thread: in ggml you will repeatedly see code like cur = ggml_mul_mat(ctx, model.layers[i].wq, cur) - multiplying this layer's Q weight matrix by the current hidden state to get the Query. The whole transformer's graph build is, skeletally, a string of such mul_mats interleaved with normalization, rope, softmax. So as long as you can derive each mul_mat's output shape from the weight shapes, you can "walk" the entire model's data flow through the code. This is the concrete meaning of "reading and building networks" from the lesson's opening - and at its core is this one mul_mat shape rule.

Three regulars: rms_norm / rope / soft_max_ext

Besides matmul, three operators recur in the attention layer. Stringing L04's attention with ggml operators gives roughly this pipeline:

Note that mul_mat appears several times in the pipeline above - projecting Q/K/V is three mul_mats, and the final weight-sum of Values is another. This confirms the earlier "matmul is the main battlefield": in one attention layer, almost all the real compute is these mul_mats, while rms_norm, rope, softmax are more like "seasoning" steps interspersed - each light on its own, yet attention is wrong without any of them. Compare this pipeline with L04's attention math and you find "the formula" and "the ggml operator sequence" map almost one-to-one - which is why understanding operators means understanding how a model lands as code.

These three regulars have plain signatures (ggml/include/ggml.h):

ggml_rms_norm(ctx, a, eps);                       // RMS-normalize over the last dim, eps avoids divide-by-zero
ggml_rope_ext(ctx, a, pos, ff, n_dims, mode, ...); // rotate by position pos, injecting relative position
ggml_soft_max_ext(ctx, a, mask, scale, max_bias); // fused: softmax(a*scale + mask)

One line each: rms_norm scales a row vector by its root-mean-square into a stable range, cheaper than LayerNorm (no mean to compute); rope does not add an "index vector" per position but rotates Q, K by position, so attention scores carry "how far apart two tokens are"; soft_max_ext is a fused operator merging "multiply scale + add mask + softmax" into one, saving memory and bandwidth. Note scale is usually 1/sqrt(d) (to keep scores from getting too large), and mask holds the causal mask (future positions at -inf).

A clarification in passing: ggml_rope and ggml_rope_ext are the same family, the latter with a string of extra parameters (freq_base, freq_scale, etc.) supporting long-context extension techniques like YaRN - in short, by adjusting the rotation "frequency", a model originally trained at 4K context can work at tens of thousands of tokens. You need not study these parameters now; just knowing "position encoding is tunable, and tuning it buys longer context" is enough.

One operator, two pieces of code

Finally, dispel a common confusion: an operator in ggml actually has two pieces of code, with a clear division. One does "graph building" (define the shape, fill op/src, L09), the other does "actually compute" (run the numbers on some backend):

These two meet through enum ggml_op, the "operator number": graph-build records the number in tensor->op, and at execution the backend uses one big switch(op) to dispatch each node to the matching ggml_compute_forward_* (CPU side in ggml/src/ggml-cpu/). So "many operators" is not scary - they share one graph-build and dispatch framework, and adding a new operator is mainly adding an enum value + writing a forward implementation. This "declaration separated from implementation" design is the very reason the same graph can run efficiently on CPU, CUDA, and Metal each.

This "two pieces of code" division, in hindsight, explains much of the earlier lessons' design. L09 said operator functions "only fill op/src, no compute" - that is because they are only the build side; the compute-side code is simply not there. L10 said the backend "computes node by node" - that compute is the compute side dispatching by op and calling each forward. So the build side and compute side correspond exactly to L09's "build" and L10's "execute" phases; one operator spans both phases, showing its face on the build side (define shape) and going all-out on the compute side (actually compute). Straighten this thread and ggml's whole execution flow strings into one complete chain in your head.

This is also why ggml can fully separate model logic from hardware acceleration: writing a new model, you only assemble ready-made operators on the build side, touching no CPU/GPU compute code at all; optimizing some operator's speed on some hardware, you only change that one forward on the compute side, affecting no model. This division - "model authors and kernel authors each mind their own patch, without disturbing each other" - is the organizational basis for why engines like ggml can be widely reused and continuously optimized.

Going deeper (optional)

Three questions below; open them if you want depth, skip them if you only want the main line.