推理时模型一次只吐一个 token：把整个序列前向一遍、采样出下一个、再接上去前向一遍……一步接一步，纯串行。而上一部分刚说过（L30/L32），decode 阶段是访存密集的——大部分时间花在把几个 GB 的权重从显存搬进来，真正的乘加反而没占满算力。投机解码（speculative decoding）就是钻这个空子：让一个小模型先猜一串，大模型一次性批改，把"猜对的部分"白赚下来。
关键洞察只有一句：既然 decode 慢在带宽、算力有富余，那"验证 K 个 token"和"生成 1 个 token"几乎一样快——多验证的那几个 token 几乎是免费的。投机解码把这点免费算力，换成了实打实的加速：典型能快 2-3 倍，而且输出和直接采样完全等价，不掉一点质量。这听起来几乎像免费的午餐，而它确实接近免费——代价只是要额外备一个小草稿模型（或者连小模型都不用，走 n-gram）。这一课就把这件"又快又不掉质量"的事，从直觉一路讲到它为什么严格成立。
路线图：先讲清"为什么验证 K 个 ~= 生成 1 个"这点免费午餐，配一张追踪图看一轮投机怎么跑；再看草稿从哪来（小 draft model / n-gram）；最后看接受/拒绝怎么定、接受率怎么决定加速比，以及为什么它不改变输出质量。
先把"免费午餐"算清楚。一次 llama_decode 的开销，绝大部分是把模型那几个 GB 的权重从显存流过一遍（L30/L32 说的访存密集）；至于这一遍里顺便算 1 个 token 还是 10 个 token 的乘加，差别小到可以忽略。换句话说：前向一次的成本几乎是固定的，和这次算几个位置无关。这就留出了一个空子——如果能一次性把好几个候选 token 都喂进去并行验证，那它们摊到的"额外成本"趋近于零。举个量级感受：一次前向的开销，几乎全压在"把几个 GB 的权重在显存里流一遍"上（动辄几十毫秒）；而这一遍里多验几个 token 位置，只是多做一点点乘加，相比那笔固定的搬运开销只是个零头。所以"顺手多验几个 token"的边际成本，低到几乎白送。这也是为什么投机解码只在 decode（memory-bound）上有效、在 prefill（compute-bound）上意义不大：prefill 本来就在并行算一大批 token，没有"闲着的算力"可榨。
自回归解码偏偏不给这个机会：它一步只产 1 个 token，下一个 token 依赖上一个，逼着你串行地把那几个 GB 反复搬 N 遍。投机解码的破法，就是先用别的办法弄出一串候选（猜的），再用大模型一次前向并行验证这一串——把"串行 N 步"压缩成"猜 1 次 + 验 1 次"。这里有个容易混淆的点：投机解码并没有"少算"——大模型该过的前向一次都没省，被拒的草稿 token 也实打实算了一遍。它省的纯粹是"串行的轮次"：把原本要排队等 N 次的前向，合并成了少数几次。所以它换来的是延迟（latency）下降，而不是总算力下降；在算力本就吃紧（大 batch、多并发）的服务器上，投机的收益反而会被压缩——这又一次呼应了 L30 的 pp/tg、L28 的连续批处理。
N 个 token = N 次前向，每次搬一遍几 GB 权重，纯串行。慢在"等数据"，不在"算"。
小模型猜 K 个 -> 大模型 1 次前向并行验 K 个 -> 接受 m 个。猜得准时，1 次前向顶 m 步。
把一轮投机定格成一条流水看最清楚：
这一轮里，大模型只做了一次前向，却一口气敲定了 4 个 token（3 个接受 + 1 个 bonus）。要是草稿全中，5 个候选就能换来 6 个 token / 一次前向；要是第一个就错，那就退化成普通 decode（白跑一次草稿）。换个角度看整段生成：原本要 N 次串行前向才能出 N 个 token，现在每次前向平均落定 m+1 个，前向次数就降到约 N/(m+1)——省下的全是那些"排队等显存"的时间。所以投机解码的加速比，全看接受率——草稿猜得越准，省得越多。
候选不能瞎猜——猜得越接近大模型的真实输出，接受率越高、越省。ggml 的 common/speculative 给了两条主要路子：
(1) draft model（草稿模型）：一个和 target 同词表的小模型（比如 target 是 70B、draft 用同系列的 1B）。它就是个迷你版的自回归解码——贪心地一个个续写，直到够 K 个、或者自己都没把握了：
// 草稿模型贪心续写 K 个 (简化自 common/speculative.cpp) while (n_drafting < n_max) { llama_decode(ctx_dft, batch); // 小模型前向一步 auto * cur_p = get_probs(ctx_dft); // 取这一步的概率分布 if (cur_p->data[0].p < params.p_min) // 最可能的 token 都没把握? break; // 收手, 不硬猜 draft.push_back(cur_p->data[0].id); // 把它当下一个候选 // ... 把这个 token 接上, 继续猜下一个 ... }
注意那个 p_min 阈值：草稿模型一旦"自己都没把握"（最可能 token 的概率都低于阈值），就提前收手——与其硬猜一个大概率被拒的 token、白占一个验证位，不如少猜几个。这正是接受率和草稿长度之间的权衡：猜太短省不到，猜太长又容易在后段翻车。（这说的是草稿的"长度" K；另一个旋钮是草稿模型的"大小"——越小越快但越不准，得和接受率一起权衡，两个旋钮一起决定那个甜区落在哪。）
(2) n-gram：连小模型都不要，直接从已经生成的文本里查。原理很朴素：自然语言里很多片段会重复（人名、固定搭配、代码里的变量名），如果当前这几个 token 在前文出现过，那它后面跟过的那几个 token，很可能就是这次要续的。common/speculative 里有好几种 n-gram 变体（ngram-simple / ngram-cache 等），共同点是零额外模型、近乎零成本地凑出候选——特别适合"输出里有大量重复"的场景（改写、续写、结构化输出）。n-gram 这条路尤其适合 agent / 工具调用 / 长文档改写这类"输入和输出大量重叠"的活儿：要续的内容很多就照抄前文，命中率高得惊人，几乎白嫖一截速度，还省掉了加载和运行草稿模型的那份显存与算力。代价是 n-gram 只会照搬前文出现过的片段，遇到全新内容就没辙——所以实现里常把它和草稿模型串成一条链：先用近乎零成本的 n-gram 试一把，命中就直接用，没命中再退回让小草稿模型猜。前面源码事实里说的"多实现可链式"，就是这个意思。
| 草稿来源 | 额外成本 | 准确度（接受率） | 最适合 |
|---|---|---|---|
| draft model | 要加载并运行一个小模型 | 高（draft 与 target 同系列时） | 通用场景，draft 与 target 同源 |
| n-gram | 近乎零（只查前文） | 看输出的重复度 | 输入输出大量重叠：改写 / agent / 代码续写 |
草稿有了，轮到大模型当裁判。调用方把草稿的 K 个 token 一次性追加进 target 的 batch，llama_decode 一次算出 target 在这 K 个位置各自的预测，然后从头逐位比对：target 在第 i 位的预测和草稿的第 i 个一致就接受，碰到第一个不一致就停，接受前面那段前缀，并把 target 自己在该位置的预测白送进来（见下一节）。这里有个关键的并行点：这 K 个位置的验证是在一次 llama_decode 里同时算出来的——靠的是把草稿的 K 个 token 当作"已知输入"一起喂进去，让 target 在一趟前向里给出每个位置的预测。这正是前面那句"验证 K 个 ~= 生成 1 个"在代码层面的落地。最后告诉投机上下文这次接受了几个：
// 调用方的验证循环 (示意; 接受语义见 common/speculative.h) int n_accepted = 0; for (int i = 0; i < n_draft; i++) { if (target_pred[i] != draft[i]) // 第一个不匹配? break; // 接受到此为止 n_accepted++; // 这一位匹配, 接受 } common_speculative_accept(spec, seq_id, n_accepted); // 告知接受了几个
"接受率"就是平均每轮草稿被接受的比例。它直接决定加速比：设草稿长度 K、平均接受 m 个，那么大模型每做 1 次前向就推进了 m+1 个 token（含 bonus），理想加速约 m+1 倍——当然要减去跑草稿的开销。所以投机解码有个甜区：草稿模型既要够小（跑得快）、又要够准（接受率高），两者得平衡。draft 和 target 越像（同系列、同数据训练），接受率越高。实战里这对应 llama.cpp 的 --model-draft / -md（指定草稿模型）和 --spec-draft-n-max / --spec-draft-n-min（草稿长度上下限）等参数；草稿长度 K 也不是越大越好——K 越大，后段越容易翻车（一旦中途某个被拒，它后面猜的就全废了），所以通常取 4 到 8 这个量级最划算。总之，投机解码不是"开了就快"的开关，而是"配好草稿来源和长度才快"的调参题——配得好，免费的加速就到手了。
反过来，接受率低时投机解码会变慢：草稿老被拒，等于每轮都白跑一遍小模型、还多搬了一次草稿那几个 token 的 KV。所以它不是无脑开就赚——任务越"可预测"（代码、翻译、改写），越划算；越"发散"（创意写作、高温采样下的多样输出），越容易亏。一个实用的判断：如果你的场景里草稿接受率长期低于某条线（经验上大概 0.3 左右），那投机大概率在帮倒忙，不如关掉。好在 llama.cpp 跑投机时会打印接受率统计，看一眼就知道这笔买卖划不划算——这也呼应 L30 的态度：别猜，去量。
两个折叠，回答投机解码最容易让人犯嘀咕的两个问题。
因为 target 那一次前向，本来就把"每个位置的下一个 token 预测"都算出来了。验证草稿的第 i 个，用的就是 target 在第 i-1 位的预测；当草稿第 i 个被拒，意味着 target 在第 i-1 位给出的预测和草稿不同——而 target 这个预测，正是这一步本来就要采的那个 token。所以它不是"额外算的"，是"顺手已经算好的"，直接拿来用就行。于是每轮哪怕草稿一个都没中，至少也能靠这个 bonus 推进 1 个 token——保证投机解码最坏也不比普通 decode 少出 token（只是多花了草稿的开销）。顺带说，正因为有这个"保底 1 个"，投机解码的最坏情况是确定的：接受率为 0 时，它退化成"普通 decode + 每轮一次白跑的草稿"，慢一点但绝不会卡死或出错。这种"上不封顶、下有保底"的性质，正是它能放心默认开启的底气。
不会——这是投机解码最漂亮的地方：它在数学上等价于直接从 target 采样，输出分布一模一样。直觉是这样：被接受的 token，都是 target 自己也会给出的（验证通过）；被拒的位置，用的是 target 自己的预测（bonus）。草稿模型只在"提议"环节出现，它提议得好不好只影响速度（接受率），完全不参与"最终选哪个 token"的决定。换句话说，草稿是"加速器"不是"决策者"。贪心解码下这点最直观（逐位必须精确匹配）；带温度的随机采样下也一样：llama.cpp 在每个位置照常从 target 采一个 token，只有当草稿这一位和采到的完全相同才接受——发出去的永远是 target 自己采的那个，所以"开不开投机、采样分布完全一致"是天然成立的。（理论上还有一种更宽松的"修正拒绝采样"，能在保持分布不变的前提下提高接受率，但 llama.cpp 走的是更简单的"精确匹配才接受"这一路。）所以你可以放心：投机解码只换速度，不换质量。这一点之所以重要，是因为它把"要不要开投机"从一个质量问题降级成了纯粹的速度问题：你永远不必担心"开了投机会不会让模型变笨"，只需要关心"这个场景下它划不划算"——而后者，量一下接受率就有答案。
In inference the model emits one token at a time: run the whole sequence forward, sample the next token, append it, run forward again... step after step, purely serial. Yet as the last part showed (L30/L32), the decode phase is memory-bound - most of the time goes to hauling several GB of weights in from VRAM, while the actual multiply-adds leave compute underused. Speculative decoding exploits exactly this gap: let a small model guess a run of tokens first, have the big model grade them in one shot, and pocket the part it guessed right for free.
The key insight is one sentence: since decode is slow on bandwidth with compute to spare, "verifying K tokens" is about as fast as "generating 1 token" - those extra verified tokens are almost free. Speculative decoding turns that free compute into real speedup: typically 2-3x faster, and exactly equivalent to direct sampling, with no quality loss at all. It almost sounds like a free lunch, and it nearly is - the only cost is keeping a small draft model around (or no small model at all, going the n-gram route). This lesson takes this "faster with no quality loss" claim from intuition all the way to why it strictly holds.
Roadmap: first make clear why "verify K ~= generate 1" is a free lunch, with a trace of one speculative round; then where the draft comes from (a small draft model / n-gram); and finally how accept/reject is decided, how the acceptance rate sets the speedup, and why it does not change output quality.
First, do the "free lunch" math. The cost of one llama_decode is mostly streaming the model's several GB of weights through from VRAM (the memory-bound point of L30/L32); whether you also compute the multiply-adds for 1 token or 10 in that pass is a negligible difference. In other words: one forward pass costs almost a fixed amount, regardless of how many positions it computes. That leaves a gap - if you can feed several candidate tokens in at once and verify them in parallel, their "extra cost" approaches zero. A sense of scale: the cost of one forward pass sits almost entirely on "streaming the several GB of weights through VRAM" (often tens of milliseconds); verifying a few more token positions in that same pass only adds a bit more multiply-add, a mere fraction of that fixed hauling cost. So the marginal cost of "verifying a few more tokens in passing" is low enough to be nearly free. This is also why speculative decoding helps only in decode (memory-bound) and means little in prefill (compute-bound): prefill is already computing a big batch of tokens in parallel, with no "idle compute" to wring out.
Autoregressive decoding refuses this chance: it produces just 1 token per step, and the next token depends on the last, forcing you to serially haul those GB N times over. Speculative decoding's trick is to produce a run of candidates by some other means (a guess), then use the big model's one forward pass to verify the run in parallel - compressing "N serial steps" into "1 guess + 1 verify". A common confusion here: speculative decoding does not "compute less" - the big model's forward passes are not saved, and rejected draft tokens really were computed. What it saves is purely "serial rounds": forward passes that used to queue up N times are merged into a few. So what it buys is lower latency, not lower total compute; on a server where compute is already tight (large batch, high concurrency), speculation's gains shrink instead - echoing once more L30's pp/tg and L28's continuous batching.
N tokens = N forward passes, each hauling several GB of weights, purely serial. Slow on "waiting for data", not "computing".
small model guesses K -> big model verifies K in 1 pass -> accept m. When guesses are good, one pass does m steps' work.
Freezing one speculative round into a flow is clearest:
In this round the big model did one forward pass yet nailed down 4 tokens at once (3 accepted + 1 bonus). If the draft were all correct, 5 candidates would buy 6 tokens per forward pass; if the very first is wrong, it degrades to plain decode (a wasted draft run). Seen across a whole generation: where you used to need N serial forward passes for N tokens, each pass now lands m+1 on average, so the pass count drops to about N/(m+1) - and all the savings are that "queuing for VRAM" time. So speculative decoding's speedup hinges entirely on the acceptance rate - the better the draft guesses, the more you save.
The candidates cannot be wild guesses - the closer they are to the big model's real output, the higher the acceptance rate and the more you save. ggml's common/speculative offers two main routes:
(1) A draft model: a small model sharing the target's vocabulary (say target 70B, draft a 1B from the same family). It is just a mini autoregressive decode - greedily extending one token at a time until it has K, or until it is unsure:
// draft model greedily extends K tokens (simplified from common/speculative.cpp) while (n_drafting < n_max) { llama_decode(ctx_dft, batch); // small model, one forward step auto * cur_p = get_probs(ctx_dft); // the step's probability distribution if (cur_p->data[0].p < params.p_min) // not even confident in the top token? break; // stop, do not force a guess draft.push_back(cur_p->data[0].id); // take it as the next candidate // ... append this token, keep guessing the next ... }
Note that p_min threshold: once the draft model is "unsure even of itself" (the top token's probability falls below the threshold), it stops early - rather than force a token likely to be rejected and waste a verify slot, it guesses fewer. This is exactly the trade-off between acceptance rate and draft length: too short saves little, too long tends to derail in the later positions. (That is the draft's "length" K; the other knob is the draft model's "size" - smaller is faster but less accurate, to be weighed against the acceptance rate, and the two knobs together decide where the sweet spot falls.)
(2) n-gram: no small model at all - look it up directly in the already-generated text. The idea is plain: natural language repeats a lot (names, fixed phrases, variable names in code), so if the current few tokens appeared earlier, the tokens that followed them then are likely what comes next now. common/speculative has several n-gram variants (ngram-simple / ngram-cache, etc.); what they share is producing candidates at zero extra model and near-zero cost - great for outputs with heavy repetition (rewriting, continuation, structured output). The n-gram route especially suits agent / tool-calling / long-document-rewriting work where input and output overlap heavily: much of what comes next is copied from earlier text, the hit rate is astonishingly high, you grab a chunk of free speed, and you save the VRAM and compute of loading and running a draft model too. The cost is that n-gram only copies fragments that appeared earlier and is helpless against genuinely new content - so implementations often chain it with a draft model: try the near-zero-cost n-gram first, use it on a hit, and fall back to the small draft model's guess on a miss. The "multiple implementations can chain" from the source facts above means exactly this.
| Draft source | Extra cost | Accuracy (acceptance) | Best for |
|---|---|---|---|
| draft model | load and run a small model | high (when draft and target share a family) | general use, draft and target same origin |
| n-gram | near zero (just look up earlier text) | depends on output repetition | heavy input/output overlap: rewriting / agent / code continuation |
With a draft in hand, the big model becomes the judge. The caller appends the draft's K tokens into the target's batch at once, llama_decode computes the target's prediction at each of those K positions in one pass, then compares position by position from the start: if the target's prediction at position i matches the draft's i-th token, accept it; at the first mismatch, stop, accept the prefix so far, and add the target's own prediction at that position for free (see next section). A key parallelism point here: these K positions are all verified in one llama_decode - by feeding the draft's K tokens in as "known inputs" so the target gives a prediction at every position in a single forward pass. This is exactly the code-level realization of "verify K ~= generate 1" from earlier. Finally tell the speculative context how many were accepted:
// the caller's verify loop (illustrative; accept semantics in common/speculative.h) int n_accepted = 0; for (int i = 0; i < n_draft; i++) { if (target_pred[i] != draft[i]) // first mismatch? break; // accept up to here n_accepted++; // this position matches, accept } common_speculative_accept(spec, seq_id, n_accepted); // inform how many accepted
The "acceptance rate" is the average fraction of the draft accepted per round. It directly sets the speedup: with draft length K and m accepted on average, each big-model forward pass advances m+1 tokens (including the bonus), an ideal speedup around m+1x - minus the cost of running the draft, of course. So speculative decoding has a sweet spot: the draft model must be small enough (fast) yet accurate enough (high acceptance), and the two must balance. The more alike draft and target are (same family, same training data), the higher the acceptance. In practice this maps to llama.cpp's --model-draft / -md (pick the draft model) and --spec-draft-n-max / --spec-draft-n-min (draft-length bounds); and draft length K is not better when bigger - the larger K is, the more the later positions tend to derail (once one mid-draft token is rejected, everything it guessed after is wasted), so 4 to 8 is usually the sweet range. In short, speculative decoding is not a "turn it on and it's fast" switch but a "fast once you tune the draft source and length" knob-turning exercise - tune it well and the free speedup is yours.
Conversely, when the acceptance rate is low, speculative decoding gets slower: the draft keeps getting rejected, so every round wastes a small-model run and the extra KV for those draft tokens. It is not a free win you flip on blindly - the more "predictable" the task (code, translation, rewriting), the more it pays; the more "divergent" (creative writing, diverse output under high-temperature sampling), the easier it loses. A practical test: if your scenario's draft acceptance rate stays below some line (around 0.3 by rule of thumb), speculation is probably doing more harm than good and is better turned off. Helpfully, llama.cpp prints acceptance-rate stats when speculating, so one glance tells you whether the trade pays - echoing L30's stance: do not guess, measure.
Two folds answering the two things about speculative decoding that most make people uneasy.
Because that one target forward pass already computed "the next-token prediction at every position". Verifying the draft's i-th token uses the target's prediction at position i-1; when the draft's i-th is rejected, it means the target's prediction at position i-1 differs from the draft - and that target prediction is exactly the token this step was going to sample anyway. So it is not "extra computed" but "already computed in passing", free to take. Thus even if a round's draft is all wrong, the bonus still advances 1 token - guaranteeing speculative decoding never emits fewer tokens than plain decode in the worst case (it only spent the draft's cost). Worth adding: because of this "at least 1" floor, speculative decoding's worst case is well-defined - at acceptance rate 0 it degrades to "plain decode plus one wasted draft run per round", a bit slower but never stalling or erroring. This "no ceiling above, a floor below" property is exactly what makes it safe to leave on by default.
No - this is the prettiest part: it is mathematically equivalent to sampling directly from the target, the output distribution is identical. The intuition: accepted tokens are ones the target itself would give (they passed verification); rejected positions use the target's own prediction (the bonus). The draft model appears only in "proposing", and how well it proposes affects only speed (acceptance rate), never the decision of "which token is final". In other words, the draft is an accelerator, not a decision-maker. Under greedy decoding this is most obvious (each position must match exactly); under temperature sampling it is the same: llama.cpp samples a token from the target at each position as usual, and accepts the draft only when it exactly equals the sampled token - what is emitted is always the target's own sample, so "the sampling distribution is identical whether speculation is on or off" holds by construction. (There is also a looser "modified rejection sampling" in theory that raises the acceptance rate while preserving the distribution, but llama.cpp takes the simpler "accept only on exact match" route.) So rest assured: speculative decoding trades only speed, not quality. This matters because it downgrades "should I enable speculation" from a quality question to a pure speed question: you never have to worry "will turning on speculation make the model dumber", only "is it worth it in this scenario" - and the latter is answered by measuring the acceptance rate.