构建计算图Building the compute graph

With the loaded weights (L14) and the architecture hyperparameters (L15), we can finally wire them into a forward compute graph that actually computes. This lesson covers llama-graph: it provides "standard parts" like build_attn/build_ffn/build_norm, each architecture decides "in what order to assemble them" in src/models/<arch>.cpp, and the result is a ggml_cgraph from Part 3 (L09).

This lesson is the pivot connecting both sides: above (L14/L15) the model became a set of tensors "with names, shapes, and sizes"; below (L09/L10/L11) is how ggml builds graphs, executes, and computes each operator. This lesson is the bridge joining the two - it translates "a concrete model" into "a ggml compute graph". Read it and you see how llama.cpp turns a pile of weights into something "runnable".

Who builds the graph: starting from build_graph

The main entry for graph-building is llama_model::build_graph. It is itself thin, doing mainly one thing: dispatching the work to the current architecture. Because different architectures (llama, qwen2...) have different forward flows, the real graph-building logic lives in each architecture's own file src/models/<arch>.cpp.

In source (simplified from src/llama-model.cpp):

// simplified from src/llama-model.cpp
ggml_cgraph * llama_model::build_graph(const llm_graph_params & p) const {
    auto llm = build_arch_graph(p);   // virtual: dispatch to src/models/<arch>.cpp
    // ... build_pooling / build_dense_out ...
    return llm->res->get_gf();        // ggml_cgraph(L09)
}

Two layers of abstraction to separate here: build_graph is the stable main entry (whatever the architecture, outside code calls it); build_arch_graph is a virtual function each architecture overrides. This continues L15's table-driven idea - "which architecture" decides which graph code runs. And every architecture's graph-building is based on a shared base class llm_graph_context, which carries the reusable block methods build_attn/build_ffn/build_norm.

Why split graph-building into files per architecture rather than one giant switch? Because each architecture's forward differs somewhat; writing them separately keeps each clean and non-interfering, while the commonality (how attention/FFN/norm are built) is lifted into reusable base-class blocks. This "differences per file, commonality in the base" organization means adding a new architecture is basically writing one new src/models/<arch>.cpp without touching others.

When is build_graph called? The answer is once per inference step - llama_decode (next lesson) builds this step's compute graph as its first act. Sounds costly: rebuild a graph for every generated token? Not really - the graph's "structure" is light (just a chain of operator declarations, no computation), so it builds fast; and for steps of the same shape the graph can be reused, no need to start from scratch each time.

The llm_graph_params passed to build_graph carries the context needed to build this step's graph: which tokens this step processes (from L18's batch), the KV cache's current state, which backend, and so on. In other words, build_graph does not build out of thin air but builds just enough graph for "what this step computes". This is why prefill (computing a whole prompt at once) and decode (one token at a time), though sharing the same graph code, build graphs of different sizes.

That build_pooling / build_dense_out comment in the code is worth a mention: beyond the main N blocks, graph-building also appends some "wrap-up" steps as needed - pooling for embedding tasks, extra output layers for certain models. These are optional tails, usually unused in plain text generation, but they share the same graph framework, attached to the same graph by architecture and task.

How one transformer layer is built

The model's body is N identical transformer blocks stacked. See clearly how one layer is built and you understand the whole forward. A layer's skeleton is fixed: norm -> attention -> residual -> norm -> feed-forward -> residual.

Written as pseudocode, this skeleton is nearly the loop body in src/models/llama.cpp:

# pseudocode: one layer (mirrors the loop body in src/models/llama.cpp)
cur  = build_norm(inpL, attn_norm_w)            # RMSNorm(L11)
cur  = build_attn(inp_attn, cur, wq,wk,wv,wo)    # Q/K/V + rope + KV + softmax(L11/L04/L19)
inpL = cur + inpL                              # residual
cur  = build_norm(inpL, ffn_norm_w)
cur  = build_ffn(cur, w_gate, w_up, w_down)      # feed-forward(L11 mul_mat)
inpL = cur + inpL                              # residual -> next layer

Loop 0 to n_layer() (L15's accessor method), assembling this set per layer, fetching weights by L15's naming convention (blk.il.attn_q.weight etc.). As you can see, "building the graph" at this level is very mechanical - feeding fixed blocks in a fixed order with each layer's own weights, chaining a long run of operators.

This nicely closes the last three lessons: L14 prepared tensors by name, L15 said which weights each layer takes and how many layers, and here L16 actually assembles them by the transformer structure. The three together answer "how a model goes from a pile of weights to a runnable graph".

Those two residual adds (cur + inpL) look trivial but are one key to deep transformers training and working: they let each layer learn only an "increment" on top of the "original input", so information and gradients flow straight down this shortcut without decaying away across dozens of layers. At graph time it is just an ordinary ggml add operator, but its meaning is far more than one addition.

Worth noting the relation between prefill and decode at graph time: both use the same graph code, differing only in the batch fed in (L18) - prefill feeds a whole prompt's many tokens at once, decode feeds one new token at a time. The graph's "shape" varies with token count, but its "structure" (how each layer is assembled) is identical. This is the benefit of unified graph-building: one logic handles both "passing the prompt through once" and "generating word by word afterward".

One more thing: no concrete numbers appear in the graph code. build_attn, build_ffn operate entirely on "not-yet-computed tensors" - they merely say "matmul this weight with that input, call the result cur". The actual floats are filled in only when L10 executes. So reading graph code, what you read is the shape of the data flow, not the data itself - the most intuitive feel of L09's lazy build.

Step back and this whole forward graph is really a directed acyclic graph (DAG): token vectors flow in from input leaves, transform through layer after layer of block operators, and finally flow to logits. Each operator is a node, arrows mean "who feeds whom". L09 covered the essence of such graphs; this lesson just lets you see that a real large model's forward, landed on a graph, is exactly such a clearly-structured, layer-stacked DAG.

Reusable blocks and graph inputs

Underpinning this assembly is a set of reusable blocks and graph inputs on llm_graph_context. The blocks are methods like build_attn/build_ffn/build_norm; the graph inputs are the entry points that wire "external data" into the graph.

Graph inputs (llm_graph_input_*) are an easily-overlooked but crucial concept. A compute graph needs more than "operators" - it needs "entry points": where token vectors enter, where each token's position comes from, where the KV cache attaches. These are the input nodes built by build_inp_embd/build_inp_pos and friends. They are the graph's "leaves" (the leafs from L09); each inference step fills new data into them, and the graph computes a new result.

And everything the blocks produce are the ggml tensors from L11 (outputs of operators like mul_mat, soft_max_ext). One build_attn call is internally a small subgraph of a dozen-odd ggml operators chained by the attention math (L04). Connect many such subgraphs end to end and you grow the model's entire forward graph - exactly L09's "operators chained into a graph", only here from the llama layer, organized by the transformer structure.

More on graph inputs and "leaves". L09 covered two kinds of nodes: computed nodes and non-computed, only-read leaves. Weights are leaves (prepared at load, L14), and graph inputs (token vectors, positions) are leaves too - except their data is filled fresh each step. Graph-building reserves these leaves' positions, execution fills in this step's data, and the same graph computes different results.

You will notice build_attn has several overloads. Why? Because attention has many variants: with or without KV cache (some prefill paths skip it, decode always uses it), standard multi-head or GQA, with or without a sliding window... Rather than write a full attention for each, "a common skeleton + optional differences" is made into a few overloads each architecture picks from. Another instance of "fold differences into options, distill commonality into blocks".

Making graph inputs a family of classes (the llm_graph_input_* all derive from a common interface) is deliberate too: different inputs have different "fill methods" - token vectors look up by token id, positions are generated by current progress, the KV mask is computed by the causal rule. Wrapping each input's "how to fill" into its own class, called uniformly before execution, readies the graph. This makes "which external inputs the graph needs" extensible and composable.

Build and execute, sharply separated

Finally, this lesson's most important point: build_graph only builds, never computes. It fills in the operators' op and src (L09's lazy build), then get_gf() hands out a ggml_cgraph; the actual node-by-node execution is the L10 backend's job.

This "build is build, execute is execute" separation pays off enormously: the same graph runs unchanged on wildly different hardware - CPU, CUDA, Metal (L10's backend scheduling) - with the upper model logic written once. And precisely because graph-building touches no concrete computation, switching backends or adding new hardware needs no change to graph-building code. L16 "assembles the correct graph", L10 "computes the graph fast on some hardware" - each to its job, together making complete inference.

Precisely because graph-building yields only "structure", not data, this graph can in many cases be cached and reused: in consecutive decode steps each is "one new token", the graph's structure is identical, so the engine can reuse the previous graph's skeleton and only swap the inputs, saving the cost of rebuilding. Another dividend of "lazy build + structure-data separation".

So what to truly take from this lesson is a mental model: model inference = assemble weights into a compute graph by architecture (L16) + execute that graph on some backend (L10). The assembly logic, written once, runs across all hardware; that is the root of llama.cpp being both lightweight and universal. Hold onto this, and the rest of Part 4 really unfolds along its thread.