从 HF 转换模型Converting HF models

You download a model from HuggingFace and get a pile of .safetensors weight shards, plus a config.json and a tokenizer. But llama.cpp speaks exactly one file format end to end: .gguf. That middle step - "convert an HF model to GGUF" - is the star of this lesson, handled by convert_hf_to_gguf.py at the repo root. Many treat it as a black box: run one command, wait a bit, out comes a .gguf. This lesson pries the box open to see the three things it actually does.

Those three things are plain: (1) recognize the architecture - read architectures from config.json and dispatch to the matching converter class (Llama goes to LlamaModel, Qwen to Qwen2Model); (2) rename and align every tensor - HF calls the attention Q projection model.layers.0.self_attn.q_proj.weight, GGUF wants blk.0.attn_q.weight, and the hyper-parameters (layer count, dims, RoPE settings) must be written as GGUF metadata; (3) serialize in GGUF's byte format - write the header, then the metadata key-values, then the tensor-info table, then the actual weights after alignment. Understand these three steps and you can not just "use" the script but, when it does not support your model, know where to go fix it.

One structural change is worth flagging first: convert_hf_to_gguf.py used to be a multi-thousand-line monolith; it has been refactored into a thin command-line entry (about 300 lines), and the real work moved into a standalone conversion/ package - one module per architecture. So this lesson is about "how a package collaborates", not "what is inside one big file". Roadmap: first the dispatch flow behind one command (with a trace), then how the registry wires an "architecture name" to a "converter class", then tensor renaming and hyper-parameters, and finally the byte layout of the GGUF file itself.

Behind one command: the thin CLI dispatches the work

Follow one command through. You type python convert_hf_to_gguf.py /path/to/hf-model, and the first thing this thin entry script does is not convert but identify: it opens config.json in the model directory and reads the architectures field - say "LlamaForCausalLM". That is the model's ID card. With the identity in hand it asks a registry one question: "who converts this architecture?" The answer is a Python class (LlamaModel); it instantiates that class and hands everything else to its write() method. The whole main() is surprisingly short:

# simplified from main() in convert_hf_to_gguf.py
hparams = ModelBase.load_hparams(dir_model)            # read config.json
arch    = get_model_architecture(hparams, model_type)  # take architectures[0]
model_class = get_model_class(arch)                   # look up the subclass in the registry
model = model_class(dir_model, output_type, fname_out, ...)
model.write()                                          # the real conversion + serialization live here

The key to this code is that the entry does almost nothing itself - load_hparams reads the config, get_model_architecture pulls out the architecture name, get_model_class looks up the class in the registry, then instantiate and call write(). The real conversion logic (how to read tensors, how to rename, how to write hyper-params) all lives in the looked-up class. This "the entry only dispatches, the details go to plugins" structure is exactly why it could slim from thousands of lines to 300: each architecture's special handling moved into its own module under the conversion/ package, out of each other's way. So what happens inside write()? It runs prepare_tensors (rename each tensor, pick its quant type, hand it to the writer) then prepare_metadata (gather hyper-params and vocab into metadata), then calls GGUFWriter to write the header, the KV, and the tensor data, and finally close - the second half of that trace above is almost entirely compressed into this one write() call.

The dispatch flow is clearest frozen into one diagram: in goes an HF directory, through "identify arch -> look up class -> instantiate -> convert and serialize", and out comes a .gguf. As you read the diagram, hold one contrast: what goes in on the left is "a pile of files scattered to PyTorch habits", what comes out on the right is "one file gathered to the GGUF spec", and every station in between does the same one thing - translate plus gather.

The registry: how an architecture name reaches a converter class

That line "look up the registry for who is responsible" is the hinge of the whole conversion package. The registry itself is just an ordinary dictionary: architecture name (a string) -> converter class. The clever part is how it gets filled - through a classmethod called register used as a decorator. Each architecture module, when it defines its class, hangs one line above it: @ModelBase.register("LlamaForCausalLM", ...). Python runs that line when it loads the module, registering "these architecture names -> this class" into the dictionary. One class can claim several HF architecture names (Llama / Mistral / Mixtral share one conversion path), so a single register line often lists several names:

# conversion/base.py: the registry + decorator factory
class ModelBase:
    _model_classes = {ModelType.TEXT: {}, ModelType.MMPROJ: {}}   # arch name -> class

    @classmethod
    def register(cls, *names):          # takes some HF architecture names
        def func(modelcls):
            for name in names:
                cls._model_classes[model_type][name] = modelcls   # register it
            return modelcls
        return func

# conversion/llama.py: one architecture = one registered subclass
@ModelBase.register("LlamaForCausalLM", "MistralForCausalLM", "MixtralForCausalLM")
class LlamaModel(TextModel):
    model_arch = gguf.MODEL_ARCH.LLAMA
    def set_gguf_parameters(self): ...   # write layer count / dims / RoPE
    def modify_tensors(self, data, name, bid): ...  # rename / permute

Read this and you hold the entry point for "adding a new model to llama.cpp": without touching any trunk code, you just create a module under conversion/, write a subclass decorated with @ModelBase.register("YourArchName"), implement set_gguf_parameters (write hyper-params) and modify_tensors (adjust tensors as needed), and it is automatically claimed by the dispatch system. The official docs/development/HOWTO-add-model.md walks exactly this path. That is the power of "registry plus plugins": dozens of architectures are dozens of independent little files under conversion/, and anyone can copy an existing one into the next - which is how llama.cpp keeps up with the steady stream of new community models.

Tensor renaming and hyper-parameters: from PyTorch names to GGUF canonical names

After dispatch to LlamaModel, the real conversion runs two lines in parallel: one writes hyper-parameters, the other moves tensors. Hyper-params are handled by set_gguf_parameters - it writes the layer count, hidden dim, attention head count, RoPE settings and so on from config.json as GGUF metadata key-values (such as llama.block_count, llama.embedding_length). The tensor line runs through modify_tensors: it walks each weight, calls map_tensor_name to translate the HF name into the GGUF canonical name, and reshapes or re-lays-out the tensor itself when needed. The core is just a few lines:

# conversion/base.py: renaming (base-class default)
def map_tensor_name(self, name):
    new_name = self.tensor_map.get_name(key=name, try_suffixes=(".weight", ".bias"))
    if new_name is None:
        raise ValueError(f"Can not map tensor {name!r}")
    return new_name

def modify_tensors(self, data_torch, name, bid):
    return [(self.map_tensor_name(name), data_torch)]   # default: rename only, data as-is

Renaming relies on a mapping table, TensorNameMap: it converges the wildly varied naming across HF models onto llama.cpp's canonical names. For example the attention projections that HF calls self_attn.{q,k,v}_proj all become blk.N.attn_{q,k,v} in GGUF, with the layer number expanded per layer. Llama also has one unavoidable wrinkle: its Q/K weights must first be permuted - the row order HF stores Q/K in differs from what llama.cpp's RoPE implementation assumes, and without the re-permute the positional encoding computes wrong. It is exactly this "one or two special cases per model" handling that makes each architecture need its own subclass. It also explains a common error "Can not map tensor ...": usually it is not that the model is broken, but that this architecture's TensorNameMap has not yet recorded this tensor name, and you need to add a mapping or a small special case in the matching subclass. The table below is the most direct picture of this renaming step:

What a GGUF file looks like: the self-describing byte layout

Once the tensors are renamed and the hyper-params written as KV, the last step is to serialize all of it in GGUF's byte format - the job of GGUFWriter in gguf-py. It writes in four sections, strictly in order: first the header (so you can see at a glance how much is inside), then the metadata KV section (the model's manual), then the tensor-info table (a directory of every weight), and finally the tensor-data section (the actual weight bytes). Stack those sections vertically and you have a .gguf file end to end:

The header is the simplest yet most crucial section - just four fixed-length fields, written in a few lines by write_header_to_file. Its reader (llama.cpp's loader) likewise reads these four numbers first to know how much follows:

# gguf-py/gguf/gguf_writer.py: write the header (four fields)
fout.write(self._pack("<I", GGUF_MAGIC))    # 0x46554747 = "GGUF"
fout.write(self._pack("I",  GGUF_VERSION))  # 3
fout.write(self._pack("Q",  len(tensors)))  # tensor count (u64)
fout.write(self._pack("Q",  len(kv_data)))  # metadata entry count (u64)

Each record in the tensor-info table carries an offset stating which byte of the data area this tensor's data starts at. The offset is accumulated as the table is written: after each tensor, the next offset goes += ggml_pad(this tensor's byte size, 32) - that is, each tensor is rounded up to a 32-byte boundary. This is exactly where --outtype lands: choosing f16 versus q8_0 decides how many bytes each tensor occupies and which GGMLQuantizationType code is written into the table (the quantization algorithm itself was covered in L06/L12; here we only record the result per the format). By now you can see GGUF has no "magic" at all: it is a plain convention for laying "manual + directory + data" into one aligned file - and precisely because it is plain, it can be read back reliably from any language on any platform. One more easily-missed point: every value in the metadata KV section is self-typed - a leading GGUFValueType tag says whether it is a u8, i32, f32, string, or array, so the reader knows how many bytes to take and how to parse them. Because values carry their own type, GGUF can pack an integer block_count, a long array of token strings, and a few floating-point RoPE parameters all into the same section without confusion - exactly the underpinning that lets it be "self-describing".

Deeper: alignment and the vocabulary

Two accordions, filling in two details that conversion cannot avoid yet the script "auto-handles" so quietly you never see them.