前面每一层 FFN（前馈网络，L11），都是"每个 token 都老老实实从头到尾过一遍"。可现在最大的那些开源模型（Mixtral、DeepSeek、Qwen-MoE…）几乎都不这么干了——它们把一层 FFN 拆成几十甚至上百个"专家"（expert），每个 token 只挑其中两三个走。这就是 MoE（Mixture of Experts，专家混合）。这一课看 ggml 怎么实现它：一个 token 怎么被"路由"到几个专家、又怎么只算这几个而不浪费算力。这不是什么边角技巧——它已经是当下最强开源模型的标配架构，理解它，你才能读懂这一代模型为什么能又大又跑得动。
MoE 的魔力在一句话：参数容量像个大模型，单 token 的计算量却像个小模型。一个 8 专家、每 token 选 2 的 MoE 层，参数量约等于 8 个 FFN；可每个 token 只过其中 2 个——计算量只有"同样大的稠密模型"的四分之一。模型因此能用大参数"记"住多得多的东西，而每步推理的算力却省下一大截。
路线图：先看路由（router 怎么给每个 token 挑专家），配一张图追踪一个 token 的路由；再看 ggml 怎么用 ggml_mul_mat_id 只算被选中的专家；最后看这套"激活稀疏"的设计到底在用什么换什么、代价又在哪。
一切从一个小小的"打分"开始。MoE 层里有一个 router（也叫 gate，门控），本质就是一个小线性层：拿当前 token 的向量，对每个专家打一个分。分越高，说明这个 token 越该交给那个专家。你可以把 router 想成一个"调度员"：它读一眼 token 的"语义指纹"（就是那个向量），凭训练得来的经验判断"这活儿该派给哪几位"。ggml_argsort_top_k 从这些分里挑出最高的 k 个（比如 8 选 2），就是这个 token 这一步要走的专家。这里有个常被问到的点：为什么是 top-k（选好几个）而不是 top-1（只选最好的一个）？因为只选一个，路由一旦判断失误就没有退路、训练也更不稳；选 2 个则让两个专家的输出加权融合，既容错、又能表达"这个 token 介于两类之间"的细腻语义。来看 ggml 里真实的路由几行（出自 build_moe_ffn）：
// MoE 路由 (简化自 src/llama-graph.cpp build_moe_ffn) logits = build_lora_mm(gate_inp, cur); // router 线性层: 每个专家一个分 [n_expert, n_tokens] probs = ggml_soft_max(logits); // 转成概率 (有的模型用 sigmoid) selected = ggml_argsort_top_k(probs, n_expert_used); // 选 top-k 个专家 (如 8 选 2) weights = ggml_get_rows(probs, selected); // 取这 k 个专家的门控权重 weights = normalize(weights); // 归一化, 让 k 个权重加起来为 1
逐行看：build_lora_mm(gate_inp, cur) 是 router 线性层，给每个 token 算出对所有专家的打分（logits）；ggml_soft_max 把分变成概率；ggml_argsort_top_k 挑出最高的 n_expert_used 个（这就是"8 选 2"里的 2）；ggml_get_rows 把这 k 个专家对应的权重取出来、归一化——后面合并专家输出时，就按这组权重加权。整套路由极轻：相比专家本身的大矩阵乘，router 这点开销几乎可以忽略。值得强调的是，router 不是写死的规则，而是训练出来的：模型在海量数据上自己学会了"什么样的 token 该交给哪个专家"。所以你没法预先知道某个专家"专精"什么——它可能学成了"管标点的"、"管数字的"、"管某种语言的"，也可能是人类完全看不出规律的某种内部分工。这正是 MoE 既神奇又有点黑箱的地方：分工是涌现的，不是设计的。
把一个 token 的路由定格成一张图最直观：
选好了专家，接下来是 MoE 最关键、也最容易想当然的一步：怎么"只算被选中的专家"。一个偷懒的实现可能是"8 个专家全算一遍，再把没选中的 6 个扔掉"——那 MoE 就一点没省算力了。事实上，naive 的"全算再扔"在某些早期实现里真的存在过，效果就是"参数稀疏了、算力没省"，白白浪费了 MoE 的好处。所以"真稀疏"是 MoE 能不能落地的关键，也是 ggml 专门为它做一个 ggml_mul_mat_id 算子的原因。ggml 用这个算子来做到真正的稀疏：
// 稀疏专家矩阵乘 (src/llama-graph.cpp build_moe_ffn) // selected = 上一节选出的 ids: 每个 token 选中了哪几个专家 up = ggml_mul_mat_id(up_exps, cur, selected); // 只对选中专家做 up 投影 gate = ggml_mul_mat_id(gate_exps, cur, selected); // 只对选中专家做 gate 投影 act = silu(gate) * up; // 激活 (SwiGLU) out = ggml_mul_mat_id(down_exps, act, selected); // down 投影回原维度 // 最后按 router 的 weights 把 k 个专家的 out 加权求和
关键就在 ggml_mul_mat_id 里那个 id：它比普通矩阵乘多吃一个 ids 张量（就是上一节的 selected），告诉这次乘法"每个 token 该乘哪几个专家的权重"。于是它不把 token 和全部 8 个专家相乘，而是按 ids 间接寻址、只取出被选中的那 2 个专家来算。up_exps/gate_exps/down_exps 是把所有专家权重打包在一起的大张量，ids 就是从里面挑"该用哪几片"的索引。算力因此实打实降到 n_expert_used / n_expert（8 选 2 就是四分之一），而不是"算完再扔"。顺带说清一个常见误解：MoE 省的是 FFN 这一块的算力，不是整个模型的算力。注意力、归一化、embedding 这些每个 token 还是照常全算——只有 FFN 被稀疏化了。但在大模型里 FFN 恰恰是参数和算力的大头，所以把它稀疏掉，整体收益就很可观。（这也是为什么 MoE 几乎只动 FFN、不碰注意力——注意力本来就不是参数大头，拆它收益不大，还会破坏全局信息的流动。）
同样多的参数做成一个大网络，每个 token 全部过一遍：算力 = 8 份专家。
每个 token 只激活 2 个专家：算力 = 2 份，是等容量稠密的 1/4。
参数一样多（8 份专家全驻留显存）；差别只在"每步真正激活几份"。
把前面两节连起来，MoE 的取舍就清楚了：它赌的是"知识可以分而治之"——与其让每个 token 都过一个无所不包的大 FFN，不如让它只过几个最相关的专家。这样模型能把参数堆得极大（每个专家分管一摊"知识"），而单 token 的计算量只跟"选几个"有关、跟"总共有多少专家"无关。这就是为什么近两年的旗舰开源模型几乎清一色 MoE：在固定的推理算力预算下，MoE 能塞进比稠密模型多得多的参数，从而更"聪明"。换个比喻：稠密模型像请了一个什么都懂一点的全才，MoE 像请了一个专家团再配个分诊台——团队的总知识量大得多，但每次只惊动相关的那两位。MoE 的兴起背后是一条朴素的经济学：训练和推理的算力都很贵，而参数（显存）相对便宜。MoE 正好顺着这条线——花便宜的显存换贵的算力，在同样的算力预算下做出更强的模型。这就是为什么一旦有人证明 MoE 能 scale，整个开源社区几乎一夜之间都跟了上来。当然，MoE 也不是万灵药：它更难训练（负载均衡、稳定性都是坑）、对显存更挑剔、在小规模上未必比稠密划算。它是"大模型时代"的产物——当你想把参数堆到稠密架构吃不消的量级时，MoE 才真正显出威力。
但天下没有免费的午餐，MoE 的代价也很实在，而且正好踩在前几课讲过的痛点上。显存：虽然每步只算 2 个专家，但 8 个专家的权重全都得待在显存里——你不知道下一个 token 会路由到哪几个。所以 MoE 的显存占用是按"总参数"算的，往往大得吓人（这也是为什么 MoE 模型特别依赖 L33 的多卡 / offload）。好在量化（L06/L12/L29）在这里帮了大忙：把专家权重压到 4-bit，显存一下小四倍，很多原本装不下的 MoE 才挤得进消费级显卡。
访存不规整：相邻的 token 可能路由到完全不同的专家，ggml_mul_mat_id 的间接寻址让访存比稠密矩阵乘更跳跃、对缓存更不友好（呼应 L31/L32 的访存密集）。所以 MoE 是"省了算力、却更吃显存和带宽"的一笔交易——它把瓶颈从"算"挪向了"存和搬"。这也解释了一个实战现象：同样激活参数的 MoE 和稠密模型，MoE 跑起来不一定更快，但能在同样的显卡上装下聪明得多的模型。还有个微妙之处：prefill 时一批 token 一起算，不同 token 路由到不同专家，反而能把所有专家都用起来、并行度高；而 decode 时一次只有一个 token，往往只激活两个专家，硬件利用率偏低——这又一次呼应了 L18/L30 的 prefill vs decode。
最后两个折叠，补两个真正落地 MoE 时绕不开的问题。
设想 router 学偏了：它把绝大多数 token 都路由给同样那两个专家，剩下六个几乎没人光顾。那会发生两件坏事：被冷落的专家训练不充分、白占参数（容量浪费）；被挤爆的专家成了瓶颈，还可能超出它的"容量"（在某些训练和分布式推理实现里，一个专家一批能接的 token 有上限，超了就丢弃；但注意 llama.cpp 推理并不丢 token，永远把每个被选中的专家都算完）。所以训练 MoE 时通常会加一个负载均衡的辅助损失（auxiliary loss），鼓励 router 把 token 尽量均匀地分给所有专家；还会设"容量因子"给每个专家留出余量。这一步在推理时其实已经固化在权重里了（router 已经训练好），但理解它能帮你看懂 MoE 的很多设计——比如为什么有的模型要做 expert group、要做 token drop。推理侧 ggml 不需要再算 aux loss，但 router 给出的分布均不均衡，直接影响真实硬件上的专家利用率。还有个推理侧的现实问题，尤其在专家被分散到多张卡上时（L33 的专家并行）：一批 token（比如 server 同时处理的多个请求）如果恰好都挤到同一个专家，承载它的那张卡就成了串行瓶颈、别的卡却闲着——所以 MoE 在多卡高并发服务时，吞吐有时反而不如同等激活参数的稠密模型稳定。（在单卡上则相反：全挤到一个专家反而是一次规整的大矩阵乘，并不构成瓶颈。）（顺带说，正因为推理时 router 已经定型，换不同的输入、跑不同任务，专家的"忙闲"分布会跟着变——这也是为什么同一个 MoE 模型在不同任务上的实际速度会有波动。）
纯 MoE 有个隐患：有些"通用本事"（语法、常识）每个 token 都用得到，让它去挤那 k 个名额有点浪费。于是 DeepSeek 等模型加了 shared expert（共享专家）：一个所有 token 都过的常驻专家，专管通用部分，再让 router 在剩下的专家里挑 k 个管"专门"部分。build_moe_ffn 里也支持 expert group（专家分组）：先把专家分成几组、先选组再在组内选专家（源码里的 n_expert_groups / n_group_used），这在专家数特别多（上百个）时能让路由更高效、也更利于把一组专家放在同一张卡上。这些变体的共同点是：都在"路由怎么挑、挑出来怎么组合"上做文章，而底层那个 ggml_mul_mat_id 的稀疏算法，始终不变。这也是读 ggml MoE 源码的一个好心态：别被各种模型五花八门的变体绕晕，抓住"打分 -> 选 top-k -> ggml_mul_mat_id 稀疏算 -> 加权合并"这条主线，剩下的都是在这条线上加花样。举个具体的：DeepSeek-V3 有 256 个路由专家 + 1 个 shared expert、每 token 选 8 个；Mixtral 是 8 选 2；各家配置千差万别，但你拿这条主线去套，每一个都能对上号——这正是"理解机制远比记住配置更重要"的绝佳例子，也是这门课从头到尾想传达的态度。
| 维度 | Mixtral 8x7B | DeepSeek-V3 | 不变的主线 |
|---|---|---|---|
| 专家总数 n_expert | 8 | 256 (+1 shared) | 全部权重都要驻留显存 |
| 每 token 选 n_expert_used | 2 | 8 | 单步只算被选中的那几个 |
| shared expert | 无 | 有 1 个常驻 | 都在"怎么挑、怎么合"上做文章 |
| 稀疏算子 | ggml_mul_mat_id | ggml_mul_mat_id | 底层算法完全一样 |
Every FFN (feed-forward network, L11) so far has every token dutifully pass through the whole thing end to end. But the largest open models today (Mixtral, DeepSeek, Qwen-MoE...) almost all do otherwise - they split one FFN into dozens or even hundreds of "experts", and each token picks only two or three to go through. This is MoE (Mixture of Experts). This lesson looks at how ggml implements it: how one token is "routed" to a few experts, and how only those few are computed without wasting compute. This is no fringe trick - it is already the default architecture of today's strongest open models, and understanding it is what lets you grasp why this generation can be both huge and runnable.
MoE's magic is one sentence: the parameter capacity of a big model, but the per-token compute of a small one. An 8-expert, top-2 MoE layer has about the parameters of 8 FFNs; yet each token goes through only 2 of them - a quarter of the compute of an equally large dense model. The model can thus use its large parameters to "remember" far more, while the compute per inference step drops a great deal.
Roadmap: first routing (how the router picks experts for each token), with a trace of one token's routing; then how ggml uses ggml_mul_mat_id to compute only the selected experts; and finally what this "activation sparsity" design trades for what, and where the cost lies.
It all starts with a tiny "scoring". An MoE layer has a router (also called the gate), essentially a small linear layer: take the current token's vector and score every expert. The higher the score, the more this token should go to that expert. Think of the router as a "dispatcher": it reads the token's "semantic fingerprint" (that vector) and, on experience learned in training, judges "who should get this job". ggml_argsort_top_k picks the highest k of those scores (say 2 of 8) - the experts this token goes through this step. A frequently asked point here: why top-k (pick several) rather than top-1 (only the best one)? Because with only one, a single routing mistake has no fallback and training is less stable; picking 2 blends two experts' outputs by weight, which both tolerates errors and can express the subtler semantics of "this token is between two categories". Here are the real routing lines in ggml (from build_moe_ffn):
// MoE routing (simplified from src/llama-graph.cpp build_moe_ffn) logits = build_lora_mm(gate_inp, cur); // router linear: one score per expert [n_expert, n_tokens] probs = ggml_soft_max(logits); // turn into probabilities (some models use sigmoid) selected = ggml_argsort_top_k(probs, n_expert_used); // pick the top-k experts (e.g. 2 of 8) weights = ggml_get_rows(probs, selected); // take those k experts' gating weights weights = normalize(weights); // normalize so the k weights sum to 1
Line by line: build_lora_mm(gate_inp, cur) is the router linear layer, scoring every expert for each token (logits); ggml_soft_max turns scores into probabilities; ggml_argsort_top_k picks the highest n_expert_used (the "2" in "2 of 8"); ggml_get_rows pulls out those k experts' weights and normalizes them - later, when combining expert outputs, this is the weighting. The whole router is tiny: against the experts' own big matmuls, its cost is nearly negligible. Worth stressing: the router is not a hardcoded rule but learned - on vast data the model itself learns "what kind of token goes to which expert". So you cannot know in advance what an expert "specializes" in - it might have become "the punctuation one", "the numbers one", "the some-language one", or some internal division of labor with no pattern a human can see. This is what makes MoE both magical and a bit of a black box: the division of labor is emergent, not designed.
Freezing one token's routing into a picture is clearest:
With experts chosen, here comes MoE's most crucial and most easily-assumed step: how to "compute only the selected experts". A lazy implementation might "compute all 8 experts then throw away the 6 not chosen" - then MoE saves no compute at all. In fact the naive "compute all then discard" really existed in some early implementations, with the effect "parameters got sparse but compute did not", squandering MoE's benefit. So "true sparsity" is what makes or breaks MoE in practice, and the reason ggml built a dedicated ggml_mul_mat_id op just for it. ggml uses that op to be genuinely sparse:
// sparse expert matmul (src/llama-graph.cpp build_moe_ffn) // selected = the ids picked last section: which experts each token chose up = ggml_mul_mat_id(up_exps, cur, selected); // up projection on selected experts only gate = ggml_mul_mat_id(gate_exps, cur, selected); // gate projection on selected experts only act = silu(gate) * up; // activation (SwiGLU) out = ggml_mul_mat_id(down_exps, act, selected); // down projection back to the original dim // finally combine the k experts' out by the router's weights (weighted sum)
The crux is the id in ggml_mul_mat_id: beyond a normal matmul it takes an extra ids tensor (the selected from last section), telling the multiply "which experts' weights each token should multiply by". So it does not multiply the token by all 8 experts but indirectly addresses by ids, pulling out only the 2 chosen experts to compute. up_exps/gate_exps/down_exps are big tensors packing all experts' weights together, and ids is the index of "which slices to use". Compute thus really drops to n_expert_used / n_expert (a quarter for 2 of 8), rather than "compute then discard". Let me clear up a common misconception: MoE saves the FFN's compute, not the whole model's. Attention, normalization, embeddings are still computed in full for every token - only the FFN part is sparsified. But in large models the FFN is exactly where most parameters and compute sit, so sparsifying it yields a big overall gain. (This is also why MoE almost only touches the FFN and leaves attention alone - attention is not the parameter-heavy part anyway, so splitting it gains little and would disrupt the flow of global information.)
the same parameters made into one big network, every token passing through all of it: compute = 8 experts' worth.
each token activates only 2 experts: compute = 2 experts' worth, a quarter of the equal-capacity dense.
the same parameter count (all 8 experts resident in VRAM); the only difference is "how many are actually activated per step".
Connecting the last two sections, MoE's trade-off is clear: it bets that "knowledge can be divided and conquered" - rather than have every token pass through one all-encompassing big FFN, let it pass through only a few most-relevant experts. The model can then pile parameters up enormously (each expert owning a patch of "knowledge"), while a single token's compute depends only on "how many are picked", not "how many experts there are in total". This is why the flagship open models of the last couple of years are almost uniformly MoE: under a fixed inference-compute budget, MoE can pack in far more parameters than a dense model, and so be "smarter". Another metaphor: a dense model is like hiring one generalist who knows a bit of everything; MoE is like hiring a panel of specialists plus a triage desk - the team's total knowledge is far greater, but each query only disturbs the relevant two. Behind MoE's rise is a plain economics: training and inference compute are expensive, while parameters (VRAM) are relatively cheap. MoE rides exactly that line - spend cheap VRAM to save expensive compute, building a stronger model under the same compute budget. That is why, once someone showed MoE scales, the whole open-source community followed almost overnight. Of course, MoE is no panacea: it is harder to train (load balancing and stability are pitfalls), pickier about VRAM, and at small scale not necessarily a better deal than dense. It is a product of the "large-model era" - MoE truly shines only when you want to pile parameters up to a scale a dense architecture cannot bear.
But there is no free lunch, and MoE's cost is concrete, landing exactly on the sore points of earlier lessons. VRAM: though only 2 experts compute per step, all 8 experts' weights must stay in VRAM - you do not know which the next token will route to. So MoE's memory footprint is counted by "total parameters", often frighteningly large (which is why MoE models lean so heavily on L33's multi-GPU / offload). Helpfully, quantization (L06/L12/L29) does a lot here: squeeze expert weights to 4-bit and VRAM shrinks fourfold, which is what squeezes many otherwise-too-big MoEs onto consumer GPUs at all.
Irregular memory access: neighboring tokens may route to entirely different experts, and ggml_mul_mat_id's indirect addressing makes memory access jumpier than a dense matmul and less cache-friendly (echoing the memory-bound theme of L31/L32). So MoE is a "saves compute but costs more VRAM and bandwidth" trade - it moves the bottleneck from "computing" toward "storing and moving". This also explains a real-world observation: at equal active parameters, a MoE and a dense model do not necessarily run at the same speed, but MoE fits a far smarter model onto the same GPU. One subtlety more: in prefill a batch of tokens is computed together, and different tokens routing to different experts can actually exercise all experts with high parallelism; while in decode a single token at a time often activates only two experts, with low hardware utilization - echoing once more the prefill-vs-decode of L18/L30.
Two last folds for two issues you cannot avoid when actually deploying MoE.
Suppose the router learned badly: it routes the vast majority of tokens to the same two experts, while the other six are barely visited. Two bad things follow: the neglected experts are undertrained and waste their parameters (capacity wasted); the overloaded experts become a bottleneck and may exceed their "capacity" (in some training and distributed-inference implementations an expert can take at most so many tokens per batch, beyond which tokens are dropped; but note llama.cpp inference drops nothing - it always computes every selected expert). So training a MoE usually adds a load-balancing auxiliary loss, encouraging the router to spread tokens evenly across experts; a "capacity factor" leaves each expert some headroom. At inference this is already baked into the weights (the router is trained), but understanding it helps you read many MoE designs - why some models use expert groups or token dropping. ggml at inference does not compute the aux loss, but how balanced the router's distribution is directly affects expert utilization on real hardware. There is also an inference-side reality, especially when experts are sharded across several GPUs (L33's expert parallelism): if a batch of tokens (say, the multiple requests a server handles at once) happen to all pile onto the same expert, the GPU holding it becomes a serial bottleneck while others sit idle - so under multi-GPU high-concurrency serving, MoE throughput is sometimes less stable than a dense model with equal active parameters. (On a single GPU it is the opposite: all piling onto one expert is just one regular large matmul, no bottleneck.) (Incidentally, because the router is fixed at inference, different inputs and different tasks shift the experts' busy/idle distribution - which is why the same MoE model's real-world speed varies across tasks.)
Pure MoE has a pitfall: some "general skills" (grammar, common sense) are used by every token, and making them compete for the k slots is a bit wasteful. So models like DeepSeek add a shared expert: a resident expert every token passes through, handling the general part, leaving the router to pick k of the rest for the "specialized" part. build_moe_ffn also supports expert groups: split experts into groups, pick a group first then experts within it (n_expert_groups / n_group_used in the source), which keeps routing efficient when there are very many experts (hundreds) and helps put a group of experts on the same GPU. What these variants share: they all play with "how routing picks and how the picks are combined", while the underlying ggml_mul_mat_id sparse algorithm stays the same. This is also a good mindset for reading ggml's MoE source: do not get dizzy in the models' assorted variants - hold the through-line "score -> pick top-k -> ggml_mul_mat_id sparse compute -> weighted combine", and the rest are just flourishes on that line. Concretely: DeepSeek-V3 has 256 routed experts + 1 shared expert, picking 8 per token; Mixtral is 2 of 8; configs differ wildly, but lay this through-line over each of them and every one lines up - a fine example of "understanding the mechanism beats memorizing the configs", and the attitude this course keeps trying to convey.
| Dimension | Mixtral 8x7B | DeepSeek-V3 | The unchanging through-line |
|---|---|---|---|
| Total experts n_expert | 8 | 256 (+1 shared) | all weights must stay resident in VRAM |
| Picked per token n_expert_used | 2 | 8 | each step computes only the chosen few |
| shared expert | none | 1 resident | all play with "how to pick and combine" |
| sparse op | ggml_mul_mat_id | ggml_mul_mat_id | the underlying algorithm is identical |