评测与基准Evaluation & benchmarks

The model runs (L25-L28) and is shrunk (L29) - but how do you know a quant level or a config is actually "good"? Feel is not enough; you need two rulers: one for quality (perplexity) and one for speed (llama-bench). This lesson covers what each measures and how to read them, and rounds off all of Part 5.

These two rulers line up exactly with quantization's "trade": in L29 you spent a little quality for size and speed, and this lesson teaches you to measure both "the quality given up" and "the speed gained", so you can choose wisely. Without measurement, "optimization" is just guessing by feel; with it, you can argue with numbers.

Two rulers, two tools: tools/perplexity (computes perplexity, measures quality) and tools/llama-bench (measures throughput, measures speed). We first see how perplexity turns "is the model good" into a comparable number, then how llama-bench splits "is it fast" into two kinds of benchmark, and finally put both rulers together to see how they guide picking a model and a quant level.

perplexity measures quality

Perplexity (PPL) measures "how confident the model is about real text". The method: take a piece of real text, have the model predict each next word, and see how high a probability it gave the actual next word. The more confident and accurate the model, the lower the PPL. In a line: lower PPL is better, low meaning the model is "unsurprised" by this text.

The formula is compact: PPL = exp(mean negative log probability). For each step in the sequence, take the probability the model gave the real next word, take its log, and negate it (this is "surprise" - lower probability means more surprise, larger negative log); average the surprise over all steps, then take exp to restore, and you get PPL. The source is tools/perplexity/perplexity.cpp, with the core being log_softmax(n_vocab, logits, tok) taking the log probability of the real word tok, the result stored in results_perplexity's ppl_value.

# the essence of perplexity (condensed from tools/perplexity/perplexity.cpp)
total = 0
for i in range(len(tokens) - 1):
    logits = model.forward(tokens[:i+1])          # up to the i-th word
    logp   = log_softmax(logits, tokens[i+1])    # log prob of the real next word
    total += -logp                                # accumulate "surprise" (negative log)
ppl = exp(total / (len(tokens) - 1))            # exp(mean negative log) = perplexity

Below a minimal example walks the formula: take the three words "The cat sat", see how "surprised" the model is at each real next word, then compute the final PPL:

So PPL essentially compresses "the model's average surprise over a whole piece of real text" into one number. On the same text, compute PPL once for the fp16 original and once for its Q4 quant, and comparing the two numbers tells you exactly how much quantization "dumbed down" the model - the "perplexity cost" column in that quantize.cpp level table L29 pointed you to is measured exactly this way, text passage by text passage. Besides this language-modeling perplexity, the tool also supports task-level evals like HellaSwag and Winogrande (chunked by n_chunk, scoring multiple-choice accuracy), viewing quality from another angle. There is also an often-overlooked premise: computing PPL only makes sense on text the model did not see during training - compute it on training data and the model "memorized the answers", so PPL is falsely low and fails to measure real generalization.

llama-bench measures speed

Beyond quality, you also care about speed. tools/llama-bench is the tool for measuring throughput (tokens per second, t/s). It splits "fast" into two very different benchmarks, matching inference's two phases: pp (prompt processing, the prefill phase of reading the prompt) and tg (token generation, the decode phase of emitting word by word). In the source they are cmd_params's n_prompt (default 512) and n_gen (default 128).

Why measure them separately? Because pp and tg have wildly different performance characteristics: prefill is "compute-bound" (a big batch of tokens computed in parallel, saturating the GPU), decode is "memory-bound" (one token at a time, the bottleneck often being moving weights out of VRAM). A quant level may speed up decode a lot while barely changing prefill; only by reading the two numbers separately do you know which half the optimization moved. After a run llama-bench reports each benchmark's t/s with avg() +/- stddev (mean and standard deviation over repeats), and can compare quant levels, thread counts, and backends side by side in one table. Precisely because it compares side by side, bench's most useful trick is "control variables": fix the model and hardware, change only one quant level, run once, and you cleanly see how much speed that one change brought. Below is a sample bench output - the same model compared at two quant levels, where you can tell at a glance that Q4_K_M is much faster on tg (token generation) and barely different on pp (reading the prompt):

# sample llama-bench output (excerpt): same model, two quant levels
| model  | size    | test  |          t/s |
| Q8_0   | 7.2 GiB | pp512 | 2500 +/- 30  |
| Q8_0   | 7.2 GiB | tg128 |   45 +/- 1   |
| Q4_K_M | 4.1 GiB | pp512 | 2600 +/- 25  |
| Q4_K_M | 4.1 GiB | tg128 |   78 +/- 2   |

Reading both rulers together

Now use both rulers together and L29's trade becomes clear: quantization (say from fp16 to Q4_K_M) halves the model size and speeds up tg a fair bit (measured by llama-bench), while perplexity rises a touch (measured by perplexity). Only by reading ppl (quality) and bench (speed) together can you judge whether the trade is worth it - either alone misleads: speed only leads you to compress until the model is dumb; quality only and you never enjoy quantization's gains. A practical view: chase the fastest given acceptable quality, or the best quality given sufficient speed - which way you lean depends on whether your scenario fears slowness or fears errors more. Put the two numbers, ppl and bench, side by side in one table and the trade-off is obvious, the choice no longer a guess but a decision made looking at numbers.

And here Part 5 completes a full arc. Starting from the outward interface, we have watched how llama.cpp is used, served, compressed, and measured - the whole turn from "internal principle" to "outward application".

Common pitfalls in evaluation

Evaluation looks simple but has several easy traps. The most common: comparing PPL across datasets is meaningless. Perplexity depends heavily on which text you compute it on - the same model can give PPL several times apart on Wikipedia versus on code. So "model A's PPL is 5, model B's is 8, A is better" only holds when computed on the same text; always fix the dataset when comparing, or you are comparing apples and oranges.

The second trap is comparing bench numbers across hardware. llama-bench's t/s is the result on your machine, this backend, this thread set; swap a GPU or a build option and the numbers change. So do not directly compare someone else's posted bench numbers with yours; to compare, do it on the same machine, varying only the one variable you care about (say only the quant level), with everything else fixed - that is the right way to "control variables".

The third trap is subtler: low PPL does not fully equal "good to use". Perplexity measures "language-modeling" accuracy, sensitive to quantization loss and good for relative comparison; but a model with slightly higher PPL is not necessarily worse on a concrete task (coding, reasoning). So watching PPL while tuning quants is fine, but the final model choice should also draw on task-level evals, even your own real trial. Numbers are a guide, not gospel.

Deep dive: PPL intuition and task-level evals

Two final folds for two often-asked points: how to grasp that perplexity number "by feel", and what evals exist besides ppl and when to use which.