架构与超参Architecture & hyperparameters

Last lesson the loader read the weights into a "name-indexed tensor list", but how do those tensors know which architecture they belong to (llama? qwen2?), and which parts each layer has? This lesson covers two things: llama-arch (architecture identity + key-name convention + tensor-name convention - three tables) and llama-hparams (how many layers, how wide, how many heads per layer) - together the blueprint that turns "a pile of tensors" into "a concrete, graph-able model".

Why a whole lesson on these two? Because they are the secret behind llama.cpp reading dozens of models with one codebase. Both are transformers, yet the difference between llama and qwen2 essentially hides in "which conventions, how many layers/wide, how tensors are named". Get this lesson and you understand how a new model "plugs into" the engine.

Architecture identity: llm_arch

It all starts with one question: given a model, how do you know it is llama or qwen2? The answer is in that GGUF metadata key from L13, general.architecture. It is a string (e.g. "llama"); the loader reads it, looks it up in a name table, and gets the matching architecture enum LLM_ARCH_LLAMA. Once that enum is fixed, which key names, which tensor names, and which graph builder (L16) to use are all fixed.

In source (simplified from src/llama-arch.{h,cpp}):

// simplified from src/llama-arch.h / src/llama-arch.cpp
enum llm_arch { LLM_ARCH_LLAMA, LLM_ARCH_QWEN2, /* ... dozens ... */ };
// LLM_ARCH_NAMES: enum <-> string
{ LLM_ARCH_LLAMA, "llama" }, { LLM_ARCH_QWEN2, "qwen2" }, /* ... */

This LLM_ARCH_NAMES table is a two-way bridge: on write (L02's conversion script) it turns the arch enum into a string written into GGUF; on read it turns the string back into an enum. A tiny string is the master switch for everything downstream - change it and the keys the loader seeks and the function used to build the graph all change with it. So you can read general.architecture as the model telling the engine: "please treat me by the llama rules."

Why an enum rather than string comparisons everywhere? Because an enum is fast, hard to mistype, and can index a switch / a table. The many "branch by architecture" decisions in llama.cpp - which hyperparameters to read, how tensors are named, how the graph is assembled - all route off this enum. Collapsing "who am I" into one enum value is what lets all the later "treat each architecture differently" code stay tidy.

Beyond general.architecture, the GGUF header carries a set of generic general.* metadata (mentioned in L13): model name, overall quantization level, quantization version, and so on. Together with the architecture enum, they form a complete self-introduction of "what this model even is". What the loader reads first is exactly this generic info plus the architecture-specific hyperparameters - the former recognizes "who", the latter measures "how big".

This "string <-> enum" bridge also explains one thing L02's conversion script does: when converting a HuggingFace model to GGUF, the script must decide which architecture it is and write the matching string (e.g. "llama") into general.architecture. Writer and reader share the same LLM_ARCH_NAMES, so write and read mesh perfectly. This is also why a brand-new architecture sometimes needs support added to llama.cpp first, before the conversion script recognizes and can convert it.

Hyperparameters: llama_hparams

Knowing the architecture, you still need the concrete dimensions: is this llama 7B or 70B? How many layers, how wide, how many heads per layer? These numbers are the hyperparameters, held in llama_hparams, all read from the GGUF KVs by last lesson's get_key.

// simplified from src/llama-hparams.h
struct llama_hparams {
    uint32_t n_embd;  uint32_t n_ctx_train;  float rope_freq_base_train;  float f_norm_rms_eps;
    std::array<uint32_t, LLAMA_MAX_LAYERS> n_head_arr, n_head_kv_arr, n_ff_arr; // stored per layer
    uint32_t n_layer() const;  uint32_t n_head(uint32_t il = 0) const;        // accessors, not fields!
};

Once read, these hyperparameters become the inference's "size baseline": graph-building (L16) stacks layers by n_layer(), splits heads by n_head(il), sets matrix shapes by n_embd; the KV cache (L19) sizes itself by layer count and KV head count. In short, hparams are the set of numbers that turn "an abstract transformer" into "this particular model".

While we are at it, untangle "parameter count" from hyperparameters. The 7B, 70B we casually say refers to the total count of floats in the model's weights (7 billion, 70 billion); and that total is computed from the hyperparameters - roughly n_layer x the sum of each layer's weight-matrix sizes, where each matrix's size is set by n_embd, n_ff, etc. So hyperparameters are not isolated numbers; together they determine how big the model is. Read the hyperparameters and you can estimate a model's VRAM appetite from just a few numbers.

One more word on that default il = 0: it makes the common "same for every layer" model effortless - pass no index, and it defaults to layer 0's (i.e. every layer's) value. So although the interface is "fetch per layer", it is not verbose for honest homogeneous models. A considerate design: it can express the complex case without cluttering the simple one.

Worth specially remembering is what n_head_kv means: it directly decides how big the KV cache is. GQA's idea is to let several Q heads share one set of K/V, so KV heads are far fewer than Q heads, and the KV cache shrinks several-fold (L19 covers this). So reading a model's hyperparameters, the ratio of n_head to n_head_kv nearly tells you "how friendly this model is to long context". A seemingly minor hyperparameter, with a big VRAM-and-speed account behind it.

Tensor naming convention

The last piece: how do the loader's tensors line up by name with the model structure? Via a naming convention. Every tensor in the file has a name, and these names are not arbitrary - they follow the templates defined in LLM_TENSOR_NAMES.

These names reveal the pattern: token_embd is the token-embedding table (the first layer), blk.0.attn_q is layer 0's attention Q projection, blk.0.ffn_gate is layer 0's FFN gate, output_norm is the final output norm. The %d in blk.%d is filled with the layer index - exactly where n_layer() earns its keep: loop 0 to n_layer(), and per layer build that layer's tensor names from the templates.

Names are built by a small constructor called LLM_TN (its tn(...) call assembles "part + suffix + layer index" into a full tensor name). At graph time (L16), each time a weight is needed, this constructor builds the name and looks it up in last lesson's weights_map - which strings together the "loading" and "graph" lessons via names: the loader stores by name, the graph fetches by name, aligned through the shared LLM_TENSOR_NAMES convention.

List a transformer layer's tensor names in full and the pattern is clearer still: the attention part has four projections attn_q/attn_k/attn_v/attn_output plus an attn_norm; the FFN part has three matrices ffn_gate/ffn_up/ffn_down plus an ffn_norm. Every layer replicates this template, prefixed with blk.<index>.. Understand this "what weights a layer has" list and you understand all the learnable parameters of one transformer block.

Why dot-separated hierarchical names (blk.0.attn_q.weight)? Because it naturally forms a hierarchy tree: under blk are the layers, under a layer the parts, under a part weight/bias. Such naming is both clear and convenient for prefix matching in bulk - to find all of layer 0's weights, match the blk.0. prefix. A good naming convention is not just "giving a name"; it encodes structural information into the name itself.

This naming is also cross-architecture: whether llama or qwen2, layer 0's attention Q projection is named blk.0.attn_q. Because everyone shares one naming set, generic tensor tools (quantization, conversion, visualization) can process any model without caring about architecture.

By the way: some tensors are optional - many modern models' linear layers have no bias, so the .bias tensor simply does not exist in the file. At graph time, the architecture knows "which tensors this layer should have", and missing optional ones are skipped. The naming convention plus knowledge of "which are required, which optional" together fully describe an architecture's tensor makeup.

How self-description cashes out at the architecture layer

Connect the three things and L13's "self-description" cashes out fully at the architecture layer: general.architecture selects the arch; %s keys (like llama.block_count) fill the template with the arch name and are read by get_key into hyperparameters; tensors line up by their LLM_TENSOR_NAMES names. Once the three conventions mesh, the loader's "pile of tensors" becomes "a concrete model with names, shapes, and sizes", ready to hand to L16 for graph-building.

What is worth savoring is the table-driven flavor of this design: architecture is a name table, keys are a key-name table, tensors are a tensor-name table. The engine's trunk hard-codes no single model; it "acts by the tables". So supporting a new model is mostly not editing the engine, but adding a few rows to these tables + writing one graph builder (L16).

Imagine if it were not table-driven: every new model would mean a pile of if (arch == "llama") ... else if (arch == "qwen2") ... branches in the engine trunk - reading hyperparameters, naming tensors, building graphs, all needing edits everywhere. With enough models, those branches weave into a web no one dares touch. Table-driven design gathers these differences from "ifs scattered through the code" into "a few centralized tables + one graph file", so the trunk code always reads as "act by the tables" - clean and clear.

These three steps - "recognize the architecture -> read hyperparameters -> fetch tensors by name" - are really the inescapable skeleton of any "general model loader": first figure out what it is, then measure how big, finally slot the parts into place. llama.cpp does these three exceptionally cleanly, the engineering basis for it catching up to one new model after another so quickly. Next lesson, carrying this "blueprint + spec sheet + parts list", we actually assemble them into a runnable forward compute graph.

Connecting this lesson with the last reveals a clear through-line: L14's loader turns bytes into "named tensors", and L15's arch/hparams give those tensors "a blueprint and spec". By here, the model has gone from "a file on disk" to "an in-memory object with names, knowing how many layers and how wide it is" - one step short: actually assembling these parts by the blueprint into a runnable network. That is exactly the next lesson.