量化入门Quantization, intuitively

Quantization is the "compression trick" that lets llama.cpp fit a 7B - or even 70B - model into consumer GPU / RAM. Lesson 01 mentioned its 4/5/8-bit tiers; this lesson nails down three things: why it compresses, how it compresses (block quantization), and what Q4_0 / Q8_0 / K-quant are. The hardcore byte-level details wait for Part 3's "quantization format" lesson; here we build the intuition first.

Why quantize: memory and bandwidth

A trained model is essentially a huge pile of floating-point numbers (weights). Stored by default in 16-bit float (FP16 / BF16), each weight is 2 bytes - a 7-billion-parameter (7B) model needs about 14 GB for weights alone, which ordinary GPUs and RAM simply refuse. Quantization approximates these weights with fewer bits: 8-bit halves it, 4-bit halves it again, dropping memory needs several-fold.

Why can a big model be squeezed this hard and still work? Because its weights carry a lot of redundancy: each weight's exact value need not be so precise; what the model really relies on is the overall statistical pattern of the weights. Dropping each number from "very precise" to "roughly right" looks lossy one at a time, but across thousands of weights the errors largely cancel, leaving the output almost unaffected. That is why 4-bit quantization often costs only a sliver of quality for several-fold gains in size and speed.

So why is 4-bit the go-to rather than more aggressive 2-bit or safer 8-bit? It is a "sweet spot" question: 8-bit barely loses quality but saves too little; 2-bit and 3-bit save a lot but quality starts to slide noticeably. 4-bit (especially K-quant 4-bit) sits at the knee of the curve - about a quarter the size, only a sliver of quality lost - so it is the most-downloaded tier. Which exact tier still depends on your hardware, your quality tolerance, and the model's own size (bigger models usually withstand aggressive quantization better).

A bit of background: the FP32 / FP16 / BF16 that keep coming up are all floating-point formats, differing in how many bits they use and how they split them between "exponent" and "mantissa". FP32 is 4-byte full precision, common in training; FP16 and BF16 are both 2-byte half precision, with BF16 trading some mantissa precision for the same wide range as FP32, which large models like. Quantization goes a step further, replacing these floats with cheaper integer representations - think of it as "squeezing further down, on top of the half FP16 already saved".

Block quantization: one scale per block

So how exactly do we "approximate with fewer bits"? The naive idea: pick one global scale for the whole weight matrix and map the floats proportionally into a small integer range. The problem: weight magnitudes within one matrix can vary a lot, and a single global scale cannot serve both large and small values well, so the error is big.

Block quantization's answer: cut the weights into small blocks (32 weights per block in Q4_0) and give each block its own scale. A block's dynamic range is small, the scale fits tightly, and the approximation is naturally more accurate. This is "divide and conquer" applied to quantization:

Another way to see "why a small in-block range is more accurate": quantization approximates continuous values with a few fixed levels, and the narrower the range those levels must cover, the smaller the gap between levels and the finer the quantization. A whole matrix holding both 0.001 and 5.0 forced into 16 levels has coarse gaps; but cut into small blocks, the numbers in each block usually cluster at a similar magnitude, so the same 16 levels resolve them far more finely. That is the root reason block quantization is more accurate.

In code, one Q4_0 "block" is a compact struct (from ggml/src/ggml-common.h):

#define QK4_0 32       // 32 weights per block
typedef struct {
    ggml_half d;            // scale (fp16), one per block
    uint8_t   qs[QK4_0/2];  // 32 weights, each 4-bit, two packed per byte
} block_q4_0;              // 2 + 16 = 18 bytes -> 4.5 bit/weight on average

Do the math: a block of 32 weights uses 2 bytes for the scale (half-precision ggml_half) and 16 bytes for 32 4-bit quants (4 bits each, two packed into a byte), totaling 18 bytes. Per weight that is 18 x 8 / 32 = 4.5 bit - which is why "4-bit quantization" actually takes a bit more than 4 bits: the extra 0.5 bit is the per-block scale, amortized over the block.

To use it you "dequantize": restore the stored 4-bit integers to approximate floats. Q4_0's rule is simple (cf. dequantize_row_q4_0 in ggml/src/ggml-quants.c):

# each 4-bit value q in 0..15, restored to a signed weight:
for i in range(32):
    q    = nibble(qs, i)     # 0..15
    x[i] = (q - 8) * d       # subtract 8 to center near 0, then scale by the block's d

Conversely, how do you get those 4-bit integers from float weights? For symmetric quantization: find the largest-magnitude weight in the block and use it to set the scale; Q4_0's recipe is d = max / -8 (max is the block's signed extreme weight), anchoring that extreme to the end of the range (q=0) so the full -8..7 range is used; then divide each weight by d, round, and add the offset to pack into 0..15. So "quantize" and "dequantize" are inverse operations: quantize with q = round(x/d) + 8, use with x ~= (q-8)*d. The little remainder lost between the two roundings is exactly where quantization error comes from.

A side question: why is the block size often 32? Another trade-off - smaller blocks fit the local scale better and raise accuracy but need more scales, lowering the compression ratio; larger blocks do the reverse. 32 is a practice-tested compromise between accuracy and size, and it also matches common hardware parallel widths, so it computes nicely. K-quant's 256-weight super-block with sub-division then tries to get "both the high compression of big blocks and the high accuracy of small ones".

Push a few real weights through and back, and "lossy but barely" becomes visible:

Q8_0 / K-quant: trading accuracy against compression

Q4_0 is just one member of the quantization family. The same "block + scale" idea, with different parameters, gives different tiers:

Q8_0 stores each weight in 8 bits (block still 32, struct is d + 32 int8), ~8.5 bit/weight - large but closest to the original FP16, used where quality matters most. Q4_0 is lightest at 4.5 bit, fast and memory-saving, but with more visible accuracy loss.

And K-quant (the ones with K, like Q4_K, Q5_K) is a smarter tier: it uses a larger "super-block" (QK_K = 256 weights), splits it into several sub-blocks each with its own finer scale and min, and even mixes bit-widths across tensors. The result: at the same average bits, K-quant's perplexity (a measure of prediction quality, lower is better) is usually clearly lower than the matching Q4_0 / Q5_0. The GGUF files people download today are mostly K-quant tiers like Q4_K_M.

Read these names together and a pattern emerges: letter Q + bit count + optional K + optional tier suffix. Q4_0 is "4-bit, symmetric, basic block", Q4_K_M is "4-bit, K-quant, medium mixed precision", Q8_0 is "8-bit, symmetric, basic block". Next time you see a long list of Q3_K_S, Q5_K_M, Q6_K on a download page, you can read off roughly how big and how accurate each is.

typedef struct {
    ggml_half d;          // super-block overall scale
    ggml_half dmin;       // super-block overall min (asymmetric)
    uint8_t scales[...];  // finer per-sub-block scale/min (6-bit, quantized)
    uint8_t qs[...];      // quants
} block_q4_K;            // QK_K = 256, simplified from ggml-common.h

Two more things worth knowing. First, llama.cpp mainly quantizes "weights"; the "activations" flowing during inference are usually still computed in higher precision (fp16/fp32) - weights are static and dominate memory, so they are most worth compressing, while activations are dynamic and over-quantizing them hurts accuracy more easily. Second, not every tensor uses the same tier: high-impact tensors like the embedding and output projection are often deliberately kept at higher bit-widths - exactly the "mixed precision" the K-quant _M / _L tiers do.

How do you measure "how much quality quantization cost"? The most common metric is perplexity: take a standard text and see how well the model predicts the "next word" - lower is better. A common community exercise is to run perplexity across a model's quant tiers and tabulate them - you will see Q8_0 nearly matching fp16, Q4_K_M only a hair higher, and Q2_K clearly higher. The llama-perplexity tool does exactly this, and Part 5 (L30) covers using it to "grade" quantization. A rule of thumb: pick one tier higher whenever memory allows, for safer quality.

Finally, a boundary to clarify: quantization only compresses the weights' representation (how many bits each number uses); it does not change the model's structure - layer count, dimensions, parameter count all stay. That makes it a different path from pruning (removing some weights/neurons) and distillation (training a smaller new model to mimic a big one). Quantization's big advantage is being nearly free and plug-and-play: no retraining, one command shrinks and speeds up an existing model - which is why it is so ubiquitous in local inference.

Another practical detail: different hardware backends support and optimize quant formats to different degrees. The same Q4_K model may perform quite differently dequantizing via SIMD on CPU versus a dedicated kernel on CUDA. So "which tier" sometimes also depends on what hardware you will run on - Part 6 on kernels gets concrete about this. Overall, the mainstream Q4_K / Q8_0 are well supported on all backends, so you can pick them blind without much risk.

Going deeper (optional)

Three questions below; open them if you want depth, skip them if you only want the main line.