量化格式细节Quantization formats in detail

Lesson 06 built the intuition of quantization - "each small block of weights shares one scale, and stores only relative offsets inside the block". This lesson goes down to the byte level: we open ggml/src/ggml-common.h to see what every byte of blocks like block_q4_0 and block_q4_K actually holds, then how the dequantize functions restore those bytes back to floats, and finally how ggml_type_traits wires dozens of quantization types uniformly into the engine.

L06 answered "why we can compress, and how much it saves"; this lesson answers "what it looks like in memory afterward, and how the source reads it back" - the two are the "intuition side" and the "implementation side" of one thing. If L06's block-quantization intuition is still fuzzy, glance back first; this lesson lands directly on every struct field.

Open one block: q4_0 and q8_0

Start with the two most classic blocks. q4_0 packs 32 weights into a block: the first 2 bytes are a half (FP16) scale, followed by 16 bytes holding 32 4-bit quantized values (two nibbles per byte). That is 18 bytes a block. q8_0 also groups 32 weights, but stores each as a full byte of int8, unpacked - 34 bytes a block (2-byte scale + 32-byte values), more accurate and larger.

These blocks are laid out one right after another, contiguously in file and memory: a weight matrix is just a long run of blocks. Knowing "how many bytes per block" lets you compute "where block n is" - exactly the basis for random addressing and parallel processing during dequantization later. A regular layout brings benefits far beyond saving space - it makes "fetch one small block on demand and dequantize it" possible, the very precondition for the "dequantize on the fly" mentioned earlier.

In source, the struct matches the figure almost one-to-one (ggml/src/ggml-common.h):

// simplified from ggml/src/ggml-common.h
#define QK4_0 32
typedef struct { ggml_half d; uint8_t qs[QK4_0/2]; } block_q4_0;  // 2 + 16 = 18 B
typedef struct { ggml_half d; int8_t  qs[QK8_0];   } block_q8_0;  // 2 + 32 = 34 B

Field by field: d is the block's shared scale, and ggml_half is exactly a 2-byte half-precision float (L06's "baseline"); qs is the array of quantized values. q4_0's qs[QK4_0/2]=qs[16] - 32 values in only 16 bytes, because each value is just 4 bits, two squeezed into one byte (one high nibble, one low). q8_0's qs[QK8_0]=qs[32] - 32 values filling 32 bytes, one value per byte, no bit-splitting.

Why is q8_0 both more accurate and larger? Because 4 bits can represent only 16 levels while 8 bits represent 256 levels, so for the same weight 8-bit can sit much closer, with smaller quantization error. The cost is double the size: from 0.5 byte per weight to 1 byte. This is the byte-level truth behind L06's "VRAM table" - choosing Q4_0 vs Q8_0 on the command line is, in essence, picking between these two block layouts.

One often-missed detail: a q4_0 block is 18 bytes for 32 weights, averaging 4.5 bits per weight, not exactly 4. The extra 0.5 bit is the 2-byte scale amortized across 32 weights. The bigger the block, the thinner this "management overhead" spreads - which already foreshadows why the next section's K-quant uses a big 256-weight super-block.

Super-block: the two-level scale of K-quant

q4_0's one scale per 32 weights is already decent, but it can be more accurate. The idea of K-quant (the K formats, such as q4_K and q6_K) is to use a 256-weight "super-block", sliced into several sub-blocks, with a two-level scale - the super-block gives an "overall baseline", and each sub-block records its own "fine-tuning scale". This both thins the baseline overhead and preserves local precision.

Take q4_K: 256 weights split into 8 sub-blocks (32 weights each). The super-block level has two halves, d and dmin; the sub-block level has 8 groups of 6-bit scale/min packed into a 12-byte scales[]; plus 128 bytes holding 256 4-bit quantized values.

Where does the 12-byte scales[] come from? 8 sub-blocks, each needing a scale and a min, could use one byte each for 16 bytes total; but K-quant squeezes them to 6 bits, 8x(6+6)=96 bits=12 bytes, saving another 4 bytes. Even the "management" scales themselves are quantized - this squeezing of every last bit of redundancy is the daily business of quantization-format design.

The two-level scale is the soul of K-quant. The first level (the super-block's d/dmin) sets a rough range; the second level (the sub-block's 6-bit scale/min) fine-tunes locally within that range - wherever a small run of weights skews high or low, the sub-block scale follows immediately. This is the byte-level source of L06's spark "for the same bit count, K-quant is usually more accurate": precision comes not from spending more bits, but from allocating scale more cleverly.

// simplified from ggml/src/ggml-common.h (QK_K = 256, K_SCALE_SIZE = 12)
typedef struct {
    ggml_half d;                   // super-block overall scale
    ggml_half dmin;                // super-block overall min
    uint8_t   scales[K_SCALE_SIZE];// 8 sub-blocks' 6-bit scale/min (no qh)
    uint8_t   qs[QK_K/2];          // 256 4-bit quantized values
} block_q4_K;

Compute q4_K's byte count too: 2 (d) + 2 (dmin) + 12 (scales) + 128 (qs) = 144 bytes for 256 weights, averaging 144*8/256 = 4.5 bits per weight - the same bit width as q4_0, yet higher precision. That is the most direct payoff of the "two-level scale": not one extra bit spent, error pushed down purely by a smarter structure.

Worth meeting the whole K-quant family: from q2_K and q3_K up to q6_K, the number in the name is the rough bit width, while K means they all use this "super-block + two-level scale" structure. The lower the bit width (like q2_K) the harder the compression and the riskier the precision, so it is often used only on less sensitive layers; the higher (like q6_K) the closer to original precision. This is why, downloading real models, you see suffixed names like Q4_K_M and Q5_K_S - schemes that mix different K-quant levels across different layers, striking different balances between size and quality.

Why pick a super-block as big as 256? Because the bigger the super-block, the more weights the fixed "overall d/dmin" overhead spreads across, and the smaller the per-weight overhead; meanwhile local precision is backstopped by the finer sub-block scales, so a big block does not get blurry. The big block handles "high compression", the sub-blocks handle "high precision", and you get both - this is the design motive for K-quant being more accurate than q4_0 at the same bit width (see Dig deeper 1).

Dequantize: restoring bytes to floats

Weights sit in memory as quantized bytes, but to actually take part in computation (say, fed into L11's mul_mat) they must first be restored to floats. That step is dequantization, handled by each format's own dequantize_row_* function (in ggml/src/ggml-quants.c). For q4_0 the logic is surprisingly simple:

// pseudocode, mirrors dequantize_row_q4_0 in ggml/src/ggml-quants.c
for each block:
    d = half_to_float(block.d)          // fetch this block's scale
    for j in 0..15:                     // two nibbles per byte
        q0 = (block.qs[j] & 0x0F) - 8   // low 4 bits, minus 8 back to signed
        q1 = (block.qs[j] >> 4)   - 8   // high 4 bits, minus 8 back to signed
        y[j]      = q0 * d              // x = (q - 8) * d
        y[j + 16] = q1 * d

Run one real block through it and the chain above gets concrete:

The core is one line: x = (q - 8) * d. q is the 0..15 4-bit value; subtracting 8 shifts it to a signed -8..7, then multiplying by this block's scale d restores the approximate original float. q8_0 is even more direct: x = q * d, since int8 is already signed and needs no offset.

In reverse, quantization (compressing floats to bytes) is the inverse. In q4_0's reference implementation quantize_row_q4_0_ref, the scale is d = max / -8 - find the weight with the largest magnitude in the block, map it to the endpoint of the quantization range (-8), and scale-and-round the rest proportionally. Dividing by -8 rather than 8 is an easy-to-misread detail: it pairs with the "minus 8" at dequantize time, ensuring the round trip lines up numerically.

One more thing worth remembering: dequantization does not necessarily expand the whole weight tensor to floats up front (that would cost a lot of memory). In hotspots like matmul, ggml often dequantizes on the fly - restoring just the small block it needs in the inner loop, feeding it into the dot product immediately, and discarding it. So quantization saves not only VRAM; even the "decompressed floats" mostly never land, benefiting bandwidth and cache too (echoing L06's "saves VRAM and speeds up").

Wiring into the engine: ggml_type_traits

Here is the problem: q4_0, q8_0, q4_K, q6_K... dozens of quantization formats, each with a different byte layout - must every one be special-cased inside mul_mat and dequantize? Of course not. ggml uses a type-traits table (ggml_type_traits) to register each type as one row of "basic parameters + how to convert to float", and operators just look it up, never caring which quantization it actually is.

A tensor only records "I am Q4_K" in its type field (L05's type field finally earns its keep here). When an operator needs it, it looks that type up in the type_traits table to get "how many elements per block, how many bytes, which function converts to float" and just follows it - there is not a single if in the operator code branching on quantization type.

// simplified from ggml/include/ggml.h (one row per type)
struct ggml_type_traits {
    const char * type_name;     // "q4_K"
    int64_t       blck_size;     // elements per block, e.g. 256
    size_t        type_size;     // bytes per block, e.g. 144
    bool          is_quantized;
    ggml_to_float_t   to_float;       // dequantize function pointer
    ggml_from_float_t from_float_ref; // quantize function pointer (reference)
};

The key lies in those two function pointers: to_float points to this type's dequantize function, from_float_ref to its (reference) quantize function. An operator takes a tensor, calls to_float per its traits to dequantize, then computes; when saving, it quantizes via from_float_ref. Note the field name is from_float_ref (with _ref, the reference implementation), not from_float. So adding a new quantization format is mainly filling one traits row + writing these two functions, while mul_mat and the dequantize operators need almost no change - exactly L05's "structure fixed, type swappable" cashed out for quantization.

Stepping back, this lesson actually covers "quantization" at three levels: layout (what each byte in a block holds), restoration (how dequantize turns bytes back into floats), and wiring (how the traits table lets operators handle everything uniformly). These three map neatly onto the three situations you meet using llama.cpp: picking a quantization level (layout decides size and precision), running inference (dequantization happens quietly behind the scenes), and reading the source (traits is the single entry point for all quantization types). String the three together and you stop treating a name like Q4_K_M as a black box - you can say exactly what it is in memory and what happens when it computes. Breaking a quantization name down to the byte level makes your grasp of "how a model lands on disk and in memory" that much more solid.