quantize 工具The quantize tool

L06 and L12 already covered quantization's "principle" - why a few bits can approximate a float, and how each format lays out its bytes. This lesson takes a different angle: how to actually shrink a model with the tool. One llama-quantize command turns a dozen-GB fp16 model into a 3-5 GB Q4_K_M; add imatrix (the importance matrix) and the same bit width claws back a chunk of the lost quality.

In other words, L06/L12 is "understand the principle", this lesson is "operate the tool": knowing what each flag tunes, how different levels trade size against quality, and how to use imatrix, that "quality-restoring" key. Quantization is the crucial step that lets big models run on ordinary GPUs or even pure CPU, and this lesson teaches you to do it by hand.

The two stars here are tools/quantize (the compressor itself) and tools/imatrix (the companion that builds the importance matrix). We first see how quantize compresses in one command and which public API it calls underneath, then why imatrix can raise quality without adding any bits.

How the quantize tool is used

The most common use is one line: llama-quantize in.gguf out.gguf Q4_K_M - feed an fp16/fp32 GGUF, name a target level (here Q4_K_M), and it emits a compressed small GGUF. The entry is tools/quantize/main.cpp (thin), the body logic is in quantize.cpp, and the real work is the public API it calls, llama_model_quantize(in, out, &params) - note this is a function from L25's llama.h, so the quantization capability is public too, not only the command line; you can call it from your own program.

// the quantize tool's heart: the public API behind one command (simplified from tools/quantize/quantize.cpp)
llama_model_quantize_params params = llama_model_quantize_default_params();
params.ftype   = LLAMA_FTYPE_MOSTLY_Q4_K_M;  // target level
params.imatrix = imatrix_data;               // optional: feed in the importance matrix
params.dry_run = false;                       // true = only compute size, do not really compress
llama_model_quantize("in.gguf", "out.gguf", &params);

That params (llama_model_quantize_params) hides several practical knobs: ftype picks the target level; dry_run set to true only trial-computes how big the result would be without really compressing (very handy when picking a level); output_tensor_type / token_embedding_type can give a few key tensors their own higher precision; keep_split keeps the shard structure. In other words, quantization is not "one blunt cut", but can be tuned per tensor class, even per layer.

So what is a "level"? It is a llama_ftype enum value, mapping to a scheme of "how many bits per weight on average" (bpw). quantize.cpp has a table listing each level's name, bpw, and measured size/perplexity cost. Below are a few representative levels and where they stand between "size" and "quality":

The intuition for picking a level follows L06: the lower the bpw, the smaller and thriftier the model, but the greater the precision loss and the higher the perplexity (ppl, next lesson). Most people land on a "sweet spot" like Q4_K_M - small enough for consumer GPUs, yet barely any visible regression. Only when VRAM is very tight do you go toward ultra-low-bit IQ2, and there imatrix becomes the lifeline. So "picking a level" is never picking the smallest, but picking the smallest level whose quality still holds up under your VRAM budget.

The imatrix importance matrix

Here is a plain but crucial observation: not all weights matter equally. Some are almost always strongly activated and heavily affect the output; others mostly "sit around". Quantizing them all to the same precision is wasteful - too little precision on important weights clearly hurts quality, while high precision on unimportant ones wastes bits. imatrix (importance matrix) exists to solve this "how to split the bit budget" problem. By analogy, it is like a timed exam: rather than spend equal time on every question, spend more on the high-mark big questions and breeze through the small ones - the total score is naturally higher. imatrix does exactly this "allocate precision by marks" job for weights: first figure out which weights are the "big questions", then pour the precious bit budget mainly into them. Without this "mark sheet", quantization can only blindly treat all alike, inevitably wasting precision on places that hardly matter.

How do we know which weights are important? Directly: take a batch of calibration text (a few hundred representative passages) and actually run the model, and during the forward pass an eval-callback hook collect_imatrix accumulates each column's activation magnitude for every weight tensor (stored in the source as Stats values / counts). The more strongly a column is activated, the more "important" it is. When done, these stats are saved into an imatrix.gguf file, to be fed back at quantize time.

# step 1: build the importance matrix from calibration text (tools/imatrix)
llama-imatrix -m model.gguf -f calib.txt -o imatrix.gguf
# inside: during the forward pass collect_imatrix(t, ...) accumulates each weight tensor's per-column activation

# step 2: feed it at quantize time, precision goes first to important columns
llama-quantize --imatrix imatrix.gguf in.gguf out.gguf IQ2_XS

With this importance list, quantization can teach to each according to its aptitude: under the same bit budget, give important weight columns a more accurate quantization (smaller rounding error), and push the unavoidable error more onto the "irrelevant" columns. Below a minimal example shows how one row of weights is quantized under imatrix weighting:

Why this is better

The reason fits in a line: the same bits, spent where they count. Plain quantization spreads error evenly across all weights; imatrix quantization lets important weights lose almost no precision and dumps the error onto weights that hardly mattered anyway. The result: at exactly the same size (bit count), the model's overall perplexity (ppl) is lower and its behavior closer to the original fp16. Same bits, yet quality comes back a notch - that is imatrix's magic, and the real payoff of the plain effort of "measure first, then optimize". Better still, all of this is transparent to the user: you download an imatrix quant, change not a line of your load-and-infer code, yet quality is better out of nowhere - all the cleverness happens "at the moment of compression", and when you use it you simply enjoy the result.

Choosing a level for yourself

After all this talk of levels, which should you actually pick? A practical decision order is: look at VRAM first. Roughly estimate "model size" as "parameter count x bpw / 8", compare it with your GPU's VRAM - if it fits with room to spare, pick a higher level (better quality); only when it does not fit do you compress further down. For example, an 8B model is about 8GB at Q8_0, 4.5GB at Q4_K_M, 2.5GB at IQ2 - how big your card is roughly frames the range of choices.

Within what VRAM allows, then look at the use. For "a small slip is a real error" tasks like coding or reasoning, prioritize quality and try not to go below Q4_K_M; for high-tolerance scenes like casual chat or continuation, dropping a level or two is usually harmless. One often-overlooked point: at the same size, prefer a low level of a bigger model over a high level of a smaller one - a 13B Q4 is often smarter than a 7B Q8 even if they are about the same size. This is a rule of thumb the community has verified again and again.

Finally, whenever you go to ultra-low bits (IQ2, IQ3), always prefer the imatrix version; for mid-to-high levels like plain Q4/Q5 the difference with or without imatrix is smaller, though having it is usually only a gain. Remember these three steps - "VRAM frames the range, use sets the floor, ultra-low-bit demands imatrix" - and you can quickly lock onto the one best suited to you from a screen full of quant file names.

Deep dive: level naming and practical flags

Two final folds for two practical issues you will surely hit hands-on: how to read those odd level names, and what useful flags exist besides picking a level.