模型加载Model loading

Part 3 took apart the GGUF format (L13) and the ggml engine (L08-L12). This lesson moves up to the llama layer - seeing how llama_model_loader turns a .gguf into an in-memory model: reading metadata, organizing the weights into a name-indexed tensor list, using mmap to bring data into place zero-copy, and handling "one model split across several files".

Why a whole lesson on loading? Because it is the first gate from disk bytes to a usable model: L13 taught us what GGUF looks like, but between "understanding the format" and "turning it into a graph-able, inferable llama_model" sits this loader layer. Read it and you connect the two ends - "file" and "model".

Loading overview: what the loader does

Opening a .gguf, the loader roughly takes four steps: read the header, build the list, map the data, point the pointers. It computes nothing - it just organizes disk bytes into named, ready-to-fetch tensors in memory.

Landing that figure on the struct (simplified from src/llama-model-loader.h):

// simplified from src/llama-model-loader.h
struct llama_tensor_weight { uint16_t idx; size_t offs; ggml_tensor * tensor; }; // source file idx + offset + tensor
struct llama_model_loader {
    gguf_context_ptr metadata_ptr;                       // GGUF header(KV + tensor infos)
    std::map<std::string, llama_tensor_weight> weights_map; // list indexed by tensor name
    bool use_mmap;  llama_mmaps mappings;                // zero-copy data
};

Field by field: metadata_ptr holds the whole GGUF header, with all hyperparameters, vocab, and template inside (L13's self-description); weights_map is this lesson's star - a name-indexed tensor list whose key is the tensor name (e.g. blk.0.attn_q.weight) and whose value records which file the tensor came from (idx, for splits), its offset in the file (offs), and the matching ggml_tensor.

Note a key design: the list is a map (by name), not a vector (by order). Why? Because next (L15) these tensors must be matched by name to specific positions in the model architecture; and with splits, one logical model's tensors are scattered across several files, where look-up by name is far steadier than by index. Right here the loader collapses "scattered bytes" into "a list you can call by name".

This "organize into a list" step looks mundane but is the key to making everything later simple. On disk, tensors are just runs of contiguous bytes with no "who am I" information; the loader gives them names, records positions, and builds an index, so that whether building a graph by architecture (L16) or hot-swapping a LoRA (L24), the engine can point precisely at one tensor by name. Turning chaos into order first is what lets everything afterward proceed calmly - a common but underappreciated step in engineering.

When you run llama.cpp, that wall of startup logs - "loaded meta data with N key-value pairs", lines of tensor names and types - is mostly the loader reporting its work: how many KVs it read, which architecture it recognized, how big each tensor is, whether mmap was used. Next time you see those logs, you will know the process behind the screen is exactly what this lesson describes.

Reading hyperparameters and tensors

How does the loader know how many layers, how wide? Not by guessing - it reads it all from the GGUF metadata KVs, via a templated get_key that maps a "key" to a specific hyperparameter field. The keys are self-describing (L13); the next lesson (L15) covers how they are templated per architecture.

# pseudocode: typical loader usage
ml = llama_model_loader(path)              # internally gguf_init_from_file reads the header
ml.get_key(LLM_KV_BLOCK_COUNT, n_layer)   # read a hyperparameter from the KVs(L15)
for name, w in ml.weights_map:             # walk the tensor list
    t = create_tensor(name, w.tensor->ne)   # build metadata in ggml_context(L08)
    ml.load_data_for(t)                    # use_mmap: data points into the map; else read in

This snippet ties the earlier lessons together: the hyperparameters get_key reads will be packed into llama_hparams next lesson (L15); create_tensor builds only metadata (shape/type) in the ggml_context (L08, with the no_alloc idea); and load_data_for brings the real weight data into place - which, under mmap, just means pointing the data pointer into the file mapping, copying not a byte (L13's zero-copy).

So every tensor the loader reads lines up with that tensor info from L13 (name / dims / type / offset): the name becomes a key in weights_map, dims/type build the ggml tensor's metadata, and the offset tells load_data_for where in the file to fetch from. L13 covered "how it is stored in the file", this lesson covers "how the loader reads it back into memory" - the two meet end to end.

Here is a clever touch echoing L13's "light head, heavy tail": the loader first reads that small header (metadata + tensor info, usually tens of KB) and thereby learns the model's entire structure - how many layers, how wide, which tensors, where each sits in the file; only then does it touch those GB of weight data on demand. Because "description" and "data" are separated in the file, the loader can build the whole list and skeleton at tiny cost and leave the real bulk to mmap's lazy, page-by-page loading. The lightness of this step directly decides whether a large model can "open instantly".

A bit more on the offs field: it records this tensor's data byte offset in the file (matching L13's tensor info offset). With it, load_data_for can directly compute where this tensor starts in the mmap mapping - mapping base + data-section start + offs, located in one step, no sequential scan. This is exactly L13's "addressing tensors by offset" realized in loading code.

While reading tensors the loader also does some consistency checks: a tensor's shape must match what the architecture expects, its type must be supported. Doing these at load time is most economical - once they pass, graph-building and inference can assume "the tensors are all correct" without re-guarding at every step. Centralizing validation at the gate is what lets the later code stay clean.

Splits: one model across several files

Very large models (tens to hundreds of GB) are often split into multiple files for easier download and distribution. The loader reads these shards as one logical model: opening them by number and merging each shard's tensors into the same weights_map.

Splits line up via a few things: the filename format "%s-%05d-of-%05d.gguf" (built by llama_split_path, src/llama.cpp) tells you "which shard, of how many"; the metadata key split.count (LLM_KV_SPLIT_COUNT) records the total shard count; and each tensor's idx field in weights_map records "which shard I came from".

This also explains why weights_map is keyed by name: splits scatter one model's tensors across files, and only by name can they be looked up uniformly across files and stitched into a complete whole. To the caller, split-or-not is nearly invisible - you just hand the entry point a path, and the loader stitches the pieces behind the scenes.

A natural question: after splitting, does the earlier mmap zero-copy still hold? It does - the loader makes one mapping per shard (mappings is a set, not a single map), each tensor's idx records which shard it belongs to, and load_data_for fetches from that shard's mapping. So "splitting" and "zero-copy" are orthogonal: splitting solves "the file is too big to move around", mmap solves "the data is too big to copy"; together, a huge model is both easy to distribute and instant to load.

Clearing up a misconception: splitting is not a special mode. A single-file model is just the degenerate "one shard" case - split.count is 1 (or the key is simply absent). The loader's code is written uniformly as "there may be multiple shards", and a single file is just the one-shard special case. Treating the special case as one instance of the general case keeps the code simpler and less error-prone.

Tooling (including L02's gguf-py) splits a large model automatically by a target shard size when exporting, writing the split.* keys into every shard - which the loader reads to stitch them back seamlessly. Writer and reader sharing one split convention is GGUF's self-describing spirit (L13) extended into the "multiple files" dimension.

Entry points and the hand-off

From the outside, loading is one function call, with a straightforward chain:

The public entry llama_model_load_from_file (and the split version llama_model_load_from_splits) both funnel into ..._impl, then call the static llama_model_load, which creates the loader, reads metadata and tensors, and returns a llama_model - whose weight tensors already have their data pointers pointing into the mmap mapping, ready to use.

Loading ends here. The next lesson (L15) asks what comes next: this pile of "named tensors" - how does it know which architecture it belongs to (llama? qwen2?), which parts each layer has, how it is organized by the hyperparameters - that is, by what blueprint the loader's output is assembled into a model.

A bit more on what the llama_model you get after loading actually is: a read-only object holding all weight tensors (data pointers into mmap), plus the hyperparameters read from metadata (L15 packs them into llama_hparams) and the vocab (L20). It contains no "session state" - no KV cache, no notion of where computation currently is. That foreshadows why the next layer (L17) separates llama_model from llama_context: knowledge (weights) is read-only and shareable, state (KV/progress) is per-session. What this loading lesson delivers is exactly that "read-only knowledge".

To close this lesson: the loader stands between "format" and "model" - downward it only cares how GGUF's bytes are laid out (L13), upward it only delivers a clean, name-addressable set of tensors with data in place. It knows nothing of attention, nothing of how llama differs from qwen - those belong to the lessons above. This very "each does its job, with clear boundaries" layering is what lets llama.cpp keep supporting more format details on one side and more architectures on the other, with the two rarely entangling. Understand loading and you hold the first link from disk to model.

Finally, a look at why the "entry -> _impl -> _load" chain has so many layers. The outermost llama_model_load_from_file is the stable public C API, meant to stay unchanged for the long run and be called by bindings in many languages; the inner _impl and static llama_model_load are implementation that can be refactored anytime. Separating "the outward promise" from "the inward implementation" is library-design basics - you call a name that holds for years, and need not care how it evolves underneath.