transformer 的注意力有个绕不开的代价：要记住整段历史，它得把每个 token 的 Key/Value 都存进 KV cache，于是显存随序列线性地涨（L19 讲过），每生成一个新 token 还得回头看全部历史（算力 O(n^2)）。状态空间模型（State-Space Model，SSM；代表是 Mamba 和 RWKV）走了另一条完全不同的路：它不存全部历史，而是把历史压进一个固定大小的"状态" h_t，每来一个新 token，就用上一个状态 h_(t-1) 和当前输入算出新状态——显存 O(1)、不随序列长度涨。
这是注意力之外的另一条技术路线。它不是要彻底取代 transformer，而是换一种"怎么记住历史"的办法：注意力把历史摊开、一条条存着随时回查；SSM 把历史卷进一个滚动更新的状态里，只留摘要。两条路各有所长，所以这一课的落点，不只是看懂 Mamba/RWKV 怎么算，更是看懂"固定状态 vs 全量 KV"这个贯穿始终的取舍。把这层取舍想透，你以后再看到任何"线性注意力""高效 transformer 替代品"，都能一眼问到点子上：它拿什么压缩了历史？又因此丢了什么？
路线图：先把"递推 vs 注意力"摆在一起对照（一个把历史摊开存、一个把历史卷进一个状态），配一张图追踪状态怎么沿时间扫描；再看 ggml 用哪两个算子（ggml_ssm_conv + ggml_ssm_scan）实现它、状态怎么从 recurrent cache 读出写回；接着把"选择性扫描"这个核心概念讲清楚（A/B/C/Δ 各是什么、"选择性"到底选什么）；最后两个折叠聊 SSM 的代价和当下流行的"混合架构"。
先把两条路并排放在一起。注意力（你在 L19 学的）记历史的方式是"全都留着"：每个 token 算出的 Key/Value 都进 KV cache，第 t 步要看前面所有 t 个位置——所以显存随序列线性增长，单步算力随历史线性增长（整段就是 O(n^2)）。SSM 反过来，把历史卷进一个固定大小的状态 h：第 t 步只拿上一步的状态 h_(t-1) 和当前输入 x_t，算出新状态 h_t——和 t 之前有多长完全无关。状态多大，是模型定死的超参（ssm_d_state），跟序列长度没关系。这一句听着平平无奇，却是 SSM 全部优势的源头：序列从 1k 拉到 100k，注意力的开销翻了一百倍，SSM 每步的开销却一个字节都没多。
每个 token 的 KV 都存下来，一路变长。第 t 步回看全部 t 个历史：显存 O(n)、算力 O(n^2)。长程精确，但越来越重。
只维护一个固定大小的状态 h，原地更新。第 t 步只看 h_(t-1) 和 x_t：显存 O(1)、算力 O(n)。轻、稳，但状态是有损摘要。
这个差别最直观的画面，就是"状态沿时间扫描"：输入 x_0, x_1, ... 一个个进来，状态 h 在原地一步步更新，每一步只依赖上一个状态和当前输入，旁边再顺手吐出一个输出 y。看下面这张图——注意 h 那一行始终是同样大的一个格子，不像 KV cache 那样越拖越长：
SSM 在 ggml 里主要落到两个算子上，加一套"递推状态"的读写。先是 ggml_ssm_conv——一个因果 1D 卷积：在做扫描之前，先让每个位置和它前面几个位置做一次局部混合（卷积宽度就是 ssm_d_conv，通常很小，比如 4）。它的作用类似给输入加一点"短期记忆"，让相邻 token 的信息先交融一下。然后是真正的核心 ggml_ssm_scan——选择性扫描，它吃一大把张量：状态 s、输入 x、时间步 dt(Δ)、状态矩阵 A/B/C、还有序列索引 ids，一趟把整段序列的递推都算出来。来看一个 Mamba 层里这两个算子的真实调用（出自 src/models/mamba-base.cpp）：
// Mamba 层核心 (简化自 src/models/mamba-base.cpp) x = ggml_ssm_conv(ctx0, conv_x, layer.ssm_conv1d); // 1) 因果卷积: 局部短期混合 x = ggml_silu(ctx0, x); // 激活 // 2) 从 x 投影出 dt, B, C —— 注意这几个量都依赖输入 x (这就是"选择性") dt = build_lora_mm(layer.ssm_dt, dt); // 每步的步长 Δ A = layer.ssm_a; // 状态转移矩阵 (学到的参数, 不随输入变) // 3) 选择性扫描: 按 A/B/C/dt 把状态沿时间递推 y = ggml_ssm_scan(ctx0, ssm, x, dt, A, B, C, ids);
注意第 2 步那个关键细节：dt、B、C 都是从当前输入 x 算出来的（输入相关），而 A 是学到的固定参数。这正是 Mamba 比老式 SSM 强的地方——下一节细讲。这里还有一个容易被忽略的问题：状态 h 存在哪？它不进 KV cache，而是进一套专门的"递推状态缓存"（llama-memory-recurrent），每个序列只占固定大小的一块。build_rs（rs = recurrent state）就负责把上一步的状态从这块缓存里读出来、喂给 scan、再把更新后的新状态写回去：
// 递推状态的读出 - 更新 - 写回 (示意; 见 src/llama-graph.cpp build_rs) // 1) 从 recurrent cache 读出这些序列上一步的状态 states = get_state_rows(state_cache, ids); // 2) 用 scan 在这些状态上往前推一段 y = ggml_ssm_scan(ctx0, states, x, dt, A, B, C, ids); // 3) 把最后的新状态写回 cache, 留给下一步用 ggml_cpy(ctx0, last_state, state_cache_view);
对照 L17/L19 你会发现一个漂亮的呼应：transformer 用 KV cache 存"历史的全部 KV"，SSM 用 recurrent cache 存"历史压成的那个状态"。两者都是"把上一步的东西留到下一步"的缓存，但一个随序列变大、一个永远固定大小。这也是为什么 llama.cpp 要专门给 SSM 做一套 llama-memory-recurrent，而不是塞进原来的 KV cache——它们的"记忆形状"根本不同。也正因如此，一个纯 SSM 模型跑起来时，你在显存里几乎看不到"KV cache 随对话变长"这件事——取而代之的，是每个序列一小块大小恒定的状态。
"扫描"（scan）就是上面那个递推：从头到尾走一遍序列，状态一步步往前推。难点在"选择性"（selective）这三个字。先看最朴素的递推，去掉所有花哨，它就两行：
# 选择性扫描的核心递推 (概念伪代码, 非真实 kernel) for t in 0..n: h = A * h + B * x[t] # 用上一个状态 h 和当前输入 x[t] 算新状态 y[t] = C * h # 从新状态读出这一步的输出
四个量各司其职：A 管"上一个状态要保留多少"（衰减 / 记忆），B 管"当前输入怎么写进状态"，C 管"从状态里读出什么当输出"，Δ(dt) 是"步长"，控制这一步更新得多猛（可以理解成离散化的时间间隔：Δ 大≈多看一眼当前输入、Δ 小≈更依赖旧状态）。如果 A/B/C/Δ 都是固定不变的常数，那就是经典的线性 SSM——快，但呆板：它对每个 token 一视同仁，没法"看人下菜碟"。这也是早期 SSM 一直打不过 transformer 的根本原因——它记东西一视同仁，可语言里偏偏有的词关键、有的词是废话。
Mamba 的关键创新就一句话：让 B、C、Δ 随输入变化（前一节代码里 dt/B/C 都是从 x 投影出来的，就是这个意思）。这叫"选择性"——模型可以根据当前读到的内容，动态决定"这个 token 重要、多写进状态一点"或"这个 token 没用、让状态忽略它"。打个比方：固定 SSM 像一台匀速传送带，什么都按同样节奏处理；选择性 SSM 像一个会自己调速的阅读者，遇到关键句就放慢、细记，遇到废话就快进、略过。正是这点"输入相关的门控"，让 Mamba 在语言这种"信息密度不均"的任务上，第一次追平了 transformer。A 之所以仍保持固定（是学到的参数、不随输入变），是因为它管的是状态自身的稳定衰减，让它随输入乱跳反而会让递推不稳定——这是一个精心保留的"锚"。换句话说，Mamba 把 SSM 里"该死板的"（A，保稳定）和"该灵活的"（B/C/Δ，随输入）拆开对待，这种"有所变、有所不变"的拿捏，正是它比前代 SSM 高明的地方。
真实的 kernel 当然比这两行复杂得多：要做离散化（把连续的 A/Δ 变成每步的乘子）、要在 GPU 上做并行前缀和（parallel scan，把看似串行的递推拆成可并行的部分，这就是 Mamba 论文里那个"associative scan"），还要处理多头、分组。但抓住"h = A*h + B*x; y = C*h，而且 B/C/Δ 随输入走"这条主线，你就抓住了 SSM 的灵魂——剩下的都是把它高效地搬上硬件的工程。
| 维度 | 注意力 (transformer) | SSM (Mamba/RWKV) |
|---|---|---|
| 记忆方式 | 存全部 token 的 KV | 一个固定大小的状态 h |
| 显存随序列 | 线性增长 O(n) | 不变 O(1) |
| 整段算力 | O(n^2) | O(n) |
| 最擅长 | 长程精确检索 / copy | 超长序列 / 流式 / 省显存 |
| llama.cpp 缓存 | KV cache (L19) | llama-memory-recurrent |
最后两个折叠，回答 SSM 落地时最值得想清楚的两个问题。
先说甜。注意力的 KV cache 随序列线性变大，第 t 步还要和前面全部 t 个位置算注意力——序列拉到几十万 token，显存和算力都会爆。SSM 的状态固定大小、每步只看上一个状态：显存 O(1)、整段算力 O(n)，序列再长，单步开销纹丝不动。这让它在超长上下文、流式输入、端侧低显存这些场景里格外香——你甚至可以近乎无限地往里喂 token，显存都不涨。再说苦。一个固定大小的状态，本质是把任意长的历史有损压缩进一个小向量。该记的太多、状态装不下时，它只能取舍着忘。于是 SSM 在"精确检索"类任务上天然吃亏：比如"把第 3000 个 token 原样复制出来"（copy 任务）、或者需要回头精确比对很久以前的某个细节——注意力能回头逐个查 KV，SSM 却只有一个被反复覆写的摘要。这不是实现不好，是"固定状态"这个选择的根本代价：你用"不随长度涨的开销"换走了"对任意历史的精确随机访问"。所以一个实用的直觉是：任务越偏"顺序处理、不怎么回头"（长文摘要、流式转写、音频），SSM 越占便宜；越偏"精确回查、大海捞针"（按 ID 检索、长文档里找某句原话），注意力越稳。
这正是当下最流行的做法——混合架构。既然注意力擅长精确检索、SSM 擅长高效处理长序列，那就让一部分层用 SSM、一部分层用注意力，各取所长。很多新模型（如 Jamba，以及各种 Mamba-Transformer 混合）就这么搭：大多数层用 SSM 扛长度、省显存，少数几层插注意力补上精确检索的能力。llama.cpp 为此专门做了 llama-memory-hybrid——一个能同时管理两种记忆的内存子系统：注意力层那部分走 KV cache、SSM 层那部分走 recurrent state cache，两套缓存在同一个模型里并存。这也呼应了前面几课反复出现的态度：架构不是非此即彼的信仰之争，而是工程上的权衡组合。你前六部分搭好的那套地基（计算图 L09/L10、内存管理 L17、KV cache L19），到了混合架构这里依然管用——只是现在同一个模型里，有的层记"全部 KV"、有的层记"一个状态"，调度器要把两者都照顾好。这恰好说明了一件事：你学透的那套 transformer 缓存与调度，并不会因为 SSM 出现而作废——反而正是看懂混合架构的前提。
第七部分到此收束。从投机解码、MoE、多模态到状态空间模型，我们把四个"标准 transformer 之外"的进阶机制逐一拆开看了一遍——它们要么换个角度榨瓶颈、要么换种结构换效率、要么干脆换掉一根支柱。下一站，第八部分把视线从"模型怎么算"转向"工程怎么落地"：怎么把 HuggingFace 的模型转成 GGUF、怎么编译调试和测试、怎么真正参与到 llama.cpp 的贡献里来。学完前七部分，你已经从"模型怎么跑"一路看到了"前沿架构怎么变"；接下来，该把这份理解落到手上了。
The attention in a transformer has an unavoidable cost: to remember the whole history, it must store every token's Key/Value in the KV cache, so memory grows linearly with the sequence (as L19 showed), and emitting each new token means looking back over the whole history (compute O(n^2)). State-space models (SSMs; the well-known ones are Mamba and RWKV) take a completely different road: they do not store the whole history but compress it into a fixed-size "state" h_t; each new token computes a new state from the previous state h_(t-1) and the current input - memory O(1), not growing with sequence length.
This is a technical road alongside attention, not a replacement for the transformer - just a different way to "remember the history": attention spreads the history out and stores it entry by entry to look up at will; an SSM rolls the history into a continuously updated state and keeps only the gist. Each road has its strengths, so this lesson lands not only on how Mamba/RWKV compute, but on the through-line tradeoff of "fixed state vs full KV". Think this tradeoff through and, whenever you later meet any "linear attention" or "efficient transformer alternative", you can cut straight to the point: what did it use to compress the history, and what did it lose for it?
Roadmap: first put "recurrence vs attention" side by side (one spreads the history out and stores it, one rolls the history into a single state), with a trace of how the state scans along time; then the two ggml ops that implement it (ggml_ssm_conv + ggml_ssm_scan) and how the state is read from and written back to a recurrent cache; then the core concept of "selective scan" (what A/B/C/Delta are, what "selective" actually selects); and finally two folds on SSM's cost and today's popular "hybrid architectures".
First put the two roads side by side. Attention (which you learned in L19) remembers by "keeping it all": every token's Key/Value goes into the KV cache, and step t looks at all t positions before it - so memory grows linearly with the sequence and per-step compute grows linearly with the history (O(n^2) over the whole sequence). An SSM does the reverse, rolling the history into a fixed-size state h: step t takes only the previous state h_(t-1) and the current input x_t to compute the new state h_t - completely independent of how long the history before t is. How big the state is, is a hyperparameter the model fixes (ssm_d_state), unrelated to sequence length. That sentence sounds unremarkable but is the source of all of SSM's advantage: stretch the sequence from 1k to 100k and attention's cost grows a hundredfold, while an SSM's per-step cost does not gain a single byte.
Every token's KV is stored, growing ever longer. Step t looks back over all t history: memory O(n), compute O(n^2). Long-range exact, but ever heavier.
Maintain only one fixed-size state h, updated in place. Step t sees only h_(t-1) and x_t: memory O(1), compute O(n). Light and steady, but the state is a lossy summary.
The most vivid picture of this difference is "the state scanning along time": inputs x_0, x_1, ... arrive one by one, the state h updates step by step in place, each step depending only on the previous state and the current input, emitting an output y on the side. In the figure below - note that the h row is always the same single cell, not dragging ever longer like the KV cache:
In ggml an SSM mainly comes down to two ops, plus a "recurrent state" read/write. First is ggml_ssm_conv - a causal 1D convolution: before the scan, it lets each position mix locally with the few positions before it (the convolution width is ssm_d_conv, usually small, e.g. 4). It acts like adding a bit of "short-term memory" to the input, letting neighboring tokens' information blend first. Then the real core, ggml_ssm_scan - the selective scan, which eats a whole armful of tensors: the state s, the input x, the timestep dt (Delta), the state matrices A/B/C, and the sequence indices ids, computing the whole sequence's recurrence in one pass. Here is the real call of these two ops in a Mamba layer (from src/models/mamba-base.cpp):
// Mamba layer core (simplified from src/models/mamba-base.cpp) x = ggml_ssm_conv(ctx0, conv_x, layer.ssm_conv1d); // 1) causal conv: local short-term mixing x = ggml_silu(ctx0, x); // activation // 2) project dt, B, C from x - note these all depend on input x (this is "selective") dt = build_lora_mm(layer.ssm_dt, dt); // the per-step step size Delta A = layer.ssm_a; // state-transition matrix (learned param, input-independent) // 3) selective scan: recur the state along time by A/B/C/dt y = ggml_ssm_scan(ctx0, ssm, x, dt, A, B, C, ids);
Note the key detail in step 2: dt, B, C are all computed from the current input x (input-dependent), while A is a learned fixed parameter. This is exactly where Mamba beats old-style SSMs - detailed next section. There is also an easily-missed question: where does the state h live? It does not enter the KV cache, but a dedicated "recurrent state cache" (llama-memory-recurrent), each sequence taking a fixed-size block. build_rs (rs = recurrent state) reads the previous step's state out of this cache, feeds it to the scan, and writes the updated new state back:
// recurrent state read - update - writeback (illustrative; see src/llama-graph.cpp build_rs) // 1) read these sequences' previous-step state out of the recurrent cache states = get_state_rows(state_cache, ids); // 2) use the scan to advance those states forward a stretch y = ggml_ssm_scan(ctx0, states, x, dt, A, B, C, ids); // 3) write the final new state back to cache for the next step ggml_cpy(ctx0, last_state, state_cache_view);
Against L17/L19 you find a neat echo: the transformer uses the KV cache to store "all the KV of the history", an SSM uses a recurrent cache to store "the one state the history compressed into". Both are caches that "keep the previous step's stuff for the next step", but one grows with the sequence and one is forever fixed-size. This is also why llama.cpp builds a dedicated llama-memory-recurrent for SSMs rather than stuffing them into the original KV cache - their "memory shapes" are fundamentally different. For the same reason, when a pure SSM model runs you will hardly see "the KV cache growing with the conversation" in VRAM - in its place is a small, constant-size block of state per sequence.
The "scan" is that recurrence above: walk the sequence start to end, pushing the state forward step by step. The hard part is the word "selective". First, the plainest recurrence, stripped of all frills, is just two lines:
# the core recurrence of the selective scan (conceptual pseudo-code, not the real kernel) for t in 0..n: h = A * h + B * x[t] # new state from previous state h and current input x[t] y[t] = C * h # read this step's output from the new state
The four quantities each have a job: A governs "how much of the previous state to keep" (decay / memory), B governs "how the current input is written into the state", C governs "what is read out of the state as output", and Delta (dt) is the "step size", controlling how hard this step updates (think of it as the discretized time interval: large Delta ~= take one more look at the current input, small Delta ~= lean more on the old state). If A/B/C/Delta are all fixed constants, that is the classic linear SSM - fast, but rigid: it treats every token alike, unable to "tailor to who it sees". This is also the root reason early SSMs kept losing to transformers - they remember everything alike, yet in language some words are crucial and some are filler.
Mamba's key innovation is one sentence: let B, C, Delta vary with the input (in the previous section's code dt/B/C were all projected from x - that is what this means). This is "selectivity" - the model can, based on what it is currently reading, dynamically decide "this token matters, write it into the state a bit more" or "this token is useless, let the state ignore it". By analogy: a fixed SSM is a constant-speed conveyor belt, processing everything at the same pace; a selective SSM is a reader who adjusts their own speed, slowing to note key sentences and fast-forwarding past filler. It is exactly this "input-dependent gating" that let Mamba, for the first time, match transformers on a task like language with its uneven information density. A stays fixed (a learned parameter, input-independent) because it governs the state's own stable decay, and letting it jump around with the input would make the recurrence unstable - a carefully kept "anchor". In other words, Mamba treats separately what "should be rigid" in an SSM (A, for stability) and what "should be flexible" (B/C/Delta, following the input), and this judgment of "some things vary, some stay fixed" is exactly where it outdoes earlier SSMs.
The real kernel is of course far more complex than these two lines: it must discretize (turn the continuous A/Delta into per-step multipliers), do a parallel prefix-sum on the GPU (a parallel scan, splitting the seemingly-serial recurrence into parallelizable parts - this is the "associative scan" from the Mamba paper), and handle multiple heads and groups. But hold the through-line "h = A*h + B*x; y = C*h, with B/C/Delta following the input" and you have the soul of an SSM - the rest is engineering to move it efficiently onto hardware.
| Dimension | Attention (transformer) | SSM (Mamba/RWKV) |
|---|---|---|
| How it remembers | stores every token's KV | one fixed-size state h |
| Memory vs sequence | grows linearly O(n) | constant O(1) |
| Whole-sequence compute | O(n^2) | O(n) |
| Best at | long-range exact retrieval / copy | very long sequences / streaming / low VRAM |
| llama.cpp cache | KV cache (L19) | llama-memory-recurrent |
Two last folds, answering the two questions most worth thinking through when deploying SSMs.
The sweet first. Attention's KV cache grows linearly with the sequence, and step t must compute attention against all t positions before it - push the sequence to hundreds of thousands of tokens and both memory and compute blow up. An SSM's state is fixed-size and each step sees only the previous state: memory O(1), whole-sequence compute O(n), and however long the sequence the per-step cost does not budge. This makes it especially sweet for very long context, streaming input, and low-VRAM on-device - you can even feed tokens in almost endlessly and memory does not grow. Now the bitter. A fixed-size state is in essence a lossy compression of arbitrarily long history into a small vector. When there is too much worth remembering and the state cannot hold it, it can only forget selectively. So SSMs are naturally handicapped on "exact retrieval" tasks: e.g. "copy out the 3000th token verbatim" (the copy task), or needing to look back and exactly compare some detail from long ago - attention can look back and check each KV, while an SSM has only one repeatedly-overwritten summary. This is not poor implementation, it is the fundamental cost of choosing "fixed state": you traded "exact random access to arbitrary history" for "cost that does not grow with length". So a practical intuition: the more a task leans "sequential, rarely looking back" (long-document summary, streaming transcription, audio), the more SSM wins; the more it leans "exact look-up, needle-in-a-haystack" (retrieval by ID, finding one verbatim sentence in a long doc), the steadier attention is.
That is exactly today's most popular approach - the hybrid architecture. Since attention excels at exact retrieval and SSMs at efficiently handling long sequences, let some layers use SSM and some use attention, each playing to its strength. Many new models (like Jamba, and various Mamba-Transformer hybrids) are built this way: most layers use SSM to carry the length and save memory, a few layers insert attention to restore exact-retrieval ability. llama.cpp built llama-memory-hybrid for this - a memory subsystem that manages both kinds at once: the attention layers' part goes through the KV cache, the SSM layers' part through the recurrent state cache, the two caches coexisting in one model. This echoes an attitude recurring across these lessons: architecture is not an either/or article of faith but an engineering combination of tradeoffs. The foundation you built in the first six parts (compute graph L09/L10, memory management L17, KV cache L19) still holds up here at the hybrid architecture - only now, within one model, some layers remember "all the KV" and some remember "one state", and the scheduler must serve both. This neatly shows one thing: mastering the transformer's caching and scheduling does not become obsolete when SSMs appear - it is precisely the prerequisite for understanding hybrid architectures.
Part 7 closes here. From speculative decoding, MoE, and multimodality to state-space models, we have taken apart four advanced mechanisms "outside the standard transformer" one by one - each either squeezing the bottleneck from a new angle, trading structure for efficiency, or outright replacing a pillar. Next stop, Part 8 turns from "how the model computes" to "how the engineering lands": converting a HuggingFace model to GGUF, compiling/debugging/testing, and how to genuinely contribute to llama.cpp. Having finished the first seven parts, you have followed the path from "how a model runs" to "how frontier architectures change"; next, it is time to put that understanding into your own hands.