上一课，我们把整张计算图搭了出来——但它此刻还只是一具"指针搭成的骨架"，里面一个数都没算。这一课，就让这张图真正 跑起来：先给它分配内存，再按拓扑序逐个算出每个节点；如果你的机器上既有 CPU 又有 GPU，还要在多个后端之间协调分工。 这一课接上 L07（后端是什么）和 L09（图是什么），把"先建图、后执行"的后半截补完整。
换句话说，前两课我们一直在"纸上谈兵"——L08 备好内存池、L09 画好计算图，但没有一个真实的数字被算出来。 这一课是"临门一脚"：把图变成结果。你会看到，这一脚踢得相当讲究：不是简单地从头到尾算一遍就完事，而是要精打细算地用内存、还要聪明地把活儿分给不同硬件。 正是这两件事，把"能算"变成了"又快又省地算"，也让 ggml 配得上"高性能推理引擎"这个名号。
把"让一张图算出结果"这件事拆开，正好是三步，顺次发生：
第一步建图是上一课的事。这一课的主角是中间两步：规划内存（决定每个张量的数据放在缓冲区的哪个位置）和逐节点执行 （按拓扑序把每个算子真正算出来）。这两步做完，图里那些原本空着的结果张量，data 就被填上了真实的数字——你终于能读到结果了。 听起来直白，但"规划内存"这一步藏着 ggml 一个相当漂亮的优化，值得细看。我们先讲"省料"（内存），再讲"分工"（后端与调度）。
为什么"规划内存"要单独成为一步、而不是边算边随手分配？这正是惰性建图的红利所在。如果边算边分配，你算到第 6 层时，根本不知道第 20 层会不会还要用第 6 层的输出， 只能保守地把它留着。而现在图是完整的，规划器可以在真正动手算之前，先把整张图从头到尾扫一遍，把"每块内存什么时候用、什么时候能让出来"算得明明白白， 再开始执行。"先规划、后执行"，就像出门前先把行李箱怎么装规划好，而不是边走边往里塞——前者总能塞得更紧。
先问个问题：一张有上千个节点的图，是不是要为每个节点的输出都单独留一块内存？如果真这么做，峰值内存会大得吓人。 但 ggml 不这么干——它发现，很多中间结果是"用完即弃"的：第 5 层的输出喂给第 6 层之后，就再也用不到了，那块内存完全可以让第 8 层的输出来复用。 这正是 ggml-alloc（图分配器 ggml_gallocr）干的事：
它凭什么能这么精准地复用？全靠 L09 那张完整的图。因为图把"谁依赖谁"说得一清二楚，分配器可以预先推算出每个张量的"生命周期" ——它从哪个节点开始被需要、到哪个节点之后就再没人用了。一旦某个张量"寿终正寝"，它占的内存立刻被归还，供后面的张量复用。把这套逻辑写成伪代码：
# 对应 ggml/src/ggml-alloc.c 的 gallocr 规划逻辑(简化) def plan(graph): for t in graph.nodes: # 按拓扑序 t.offset = free_blocks.best_fit(nbytes(t)) # 从空闲块里找一块复用 for s in t.src: if last_use(s) == t: # s 在这之后再没人用了 free_blocks.give_back(s) # 归还它的内存, 供后面复用
这里的关键词是 best_fit（从空闲块里挑一块大小最合适的）和 give_back（张量用完就归还内存）。 ggml-alloc 内部维护一组"空闲块"，分配时找最合适的复用，释放时把相邻的空闲块合并成更大的块。因为提前看到了整张图，它能把内存复用到极致—— 实际跑下来，峰值内存常常只有"每个张量各占一块"的几分之一。这就是 L09 强调"先建图"真正的回报之一：没有完整的图，就没法做这种全局的内存规划。
顺带说一个常见误区：内存复用不会影响计算结果的正确性。有人担心"A 的地盘给了 C，会不会把 A 的数据弄乱？"——不会，因为复用只发生在 A 确定不再被任何人需要之后。 分配器严格按生命周期办事：只要还有谁可能读 A，A 的内存就绝不会被征用。所以内存复用是一种完全无损的优化，省的是空间，动不了结果——这一点和前面讲过的"量化是有损（L06）、KV cache 是无损（L04）"那个区分，是同一种思维。
内存规划好了，终于到了"真正算"的一步。这一步由后端（L07 讲过的 CPU/CUDA/Metal 等）来执行。最简单的情形是只用一个后端：
// 简化的用法; 涉及 ggml-backend.h / ggml-cpu.h / ggml-alloc.h ggml_backend_t be = ggml_backend_cpu_init(); // 选一个后端(也可以是 cuda/metal...) ggml_gallocr_alloc_graph(galloc, graph); // 按规划真正分配内存 ggml_backend_graph_compute(be, graph); // 逐节点执行!
ggml_backend_graph_compute 做的事，就是按拓扑序遍历 graph.nodes，对每个节点调用该后端对应的算子实现 （matmul 调 matmul 核、softmax 调 softmax 核……这些核函数是 L11、第六部分的主题）。因为 L09 保证了拓扑序，每算到一个节点，它的输入必定已经算好， 所以从头到尾扫一遍就完事。算完最后一个节点，输出张量里就有结果了。叶子（权重、输入）则全程不计算，只是被算子读取。
这里要破除一个幻觉：ggml_backend_graph_compute 这个函数名听起来很"重"，好像它自己在做天大的事，其实它更像一个循环 + 派发器—— 真正的苦力活（一次矩阵乘里成千上万次乘加）是在每个算子的核函数里完成的，而核函数是各后端各自实现的（CPU 用 SIMD、CUDA 用 GPU kernel）。 所以"执行一张图"在 ggml 这一层很薄：按顺序遍历节点、对每个节点喊一声"该你了"；至于"怎么算得快"，是下一课（L11 算子）和第六部分（内核）的主题。 这种"调度归调度、计算归计算"的分层，正是 ggml 能把同一张图跑在天差地别的硬件上的原因。
再点破一个容易忽略的细节：执行不是从叶子开始算的，而是直接从第一个节点开始。叶子（权重、输入）在执行前就已经备好数据了——权重是从模型文件加载的、 输入是你喂进去的，它们不需要"算"。所以 ggml_backend_graph_compute 的循环只遍历 graph.nodes，对每个节点取出它的 src（输入已就绪）、调核函数算出结果、填进它的 data。 一圈下来，从第一个节点到最后一个节点，整张图就算完了。理解这一点，你就明白为什么 L09 要把叶子和节点分开存——正是为了让执行循环能干净利落地"只算节点"。
但现实往往更复杂：你可能想把模型的一部分层放 GPU、其余留 CPU（还记得 L07 的 -ngl 吗？）。这时一个后端不够用了， 需要一个"调度器"来协调多个后端，它就是 ggml_backend_sched：
一张图里，有的算子该在 GPU 上跑、有的在 CPU 上跑；数据在两种内存里，跨设备时要搬运。谁来统筹？
① 拆图：把图切成若干段，按设备归类；② 指派：每段算子分给合适的后端；③ 拷贝：在 CPU/GPU 边界自动插入数据搬运。
顺便厘清 ggml_backend_graph_compute 和 ggml_backend_sched 的关系：前者是"单后端"的执行——一张图、一个设备，直接从头算到尾； 后者是"多后端"的总指挥——它先把图拆成几段，每段再各自交给 ggml_backend_graph_compute 在对应设备上执行。所以 sched 是更上一层的协调者， 单后端执行是它手里的基本工具。只用 CPU 时你可能直接用前者；要混合 CPU/GPU，就得请出后者。理清这层包含关系，你看 ggml 的执行代码就不会绕晕。
为什么要分多个后端，而不是统统塞给 GPU？因为显存常常装不下整个模型。一个量化后还有几十 GB 的大模型，你的显卡可能只放得下一半的层； 剩下的层只能留在 CPU 内存里、用 CPU 算。这种"一半 GPU、一半 CPU"的混合执行，正是消费级硬件跑大模型的常态，也是 ggml_backend_sched 存在的根本理由。 它让你能按显存大小灵活地切：显存多就多放几层进 GPU、少就少放几层，剩下的 CPU 兜底，总能跑起来——只是放进 GPU 的层越多、整体越快。
下面三个问题，想深究的同学点开看；只想抓主线的可以先跳过。
核心还是那句话：靠 L09 的完整图。即时计算（边算边丢）时，你没法预知"这个中间结果以后还用不用"，只能保守地都留着，内存自然省不下来。
而有了完整的图，分配器能精确算出每个张量"最后一次被用"是在哪个节点。过了那个点，这块内存立刻可回收。再配合 best_fit 挑最合适的空闲块、 把相邻空闲块合并，整张图的峰值内存就被压到很低。可以说，惰性建图省内存，省的就是这一笔——这也是为什么训练/推理框架都爱用计算图。
顺便给个量级感：对一个几十层的大模型，"每个中间张量各占一块"和"复用"两种做法，峰值内存能差好几倍。在显存本就紧张的消费级显卡上，这"好几倍"往往就是 "跑得起来"和"爆显存跑不起来"的分界线。所以内存复用不是锦上添花的小优化，而是很多模型能在你机器上跑起来的前提条件之一。
大体看两点：一是张量当前在谁的内存里（数据已经在 GPU 上，自然倾向于在 GPU 算）；二是这个算子该后端支不支持。 如果某个算子 GPU 后端还没实现，sched 会让它回退到 CPU 算完（这就是 L07 提过的 fallback），保证整张图总能跑完。
每当一段的输出要喂给另一种设备上的下一段时，sched 就在边界处自动插入一个"拷贝"节点，把数据从一种内存搬到另一种。这些拷贝是有开销的， 所以"切在哪、放哪些层到 GPU"会影响性能——但这些细节 sched 都替你处理了，你通常只需给一个 -ngl 数字。
还有一种更细的切法叫"按张量切分"（tensor split）：把同一个大矩阵乘，按列拆成几块，分给多张 GPU 同时算，再把结果拼起来。 这适合多卡场景，能把一个单步算子的负载摊到几张卡上。无论是"按层切"还是"按张量切"，背后都是 sched 在统筹——你只需通过 --split-mode 等参数告诉它怎么切， 剩下的拆图、指派、拷贝、拼接，都由它默默完成。这种"把复杂的多设备协调藏在一个简单接口后面"的设计，正是 ggml 好用的地方。
这是 ggml 一个"配置一次、反复执行"的典型套路。reserve（预留）：先用图预演一遍，量出"这张图最多要占多少内存"， 然后一次性把这块缓冲开好。alloc（分配）：之后每次执行，都在这块已开好的缓冲里复用，把各张量摆到规划好的位置上。
为什么分两步？因为开大块内存（尤其是 GPU 显存）很贵，不能每次执行都重新开一遍。先 reserve 一次、量好峰值、开好缓冲，后面成千上万步推理就反复复用同一块缓冲， 省掉了反复大分配的开销。这和 L08 的 arena 思想一脉相承：大块内存一次拿好，内部反复腾挪。
把这条线串起来看，你会发现 ggml 从头到尾贯穿着同一个信念：大额的、昂贵的操作（找系统要内存）尽量只做一次，之后全靠内部复用。 L08 的 arena 如此、这里的 reserve/alloc 如此、L10 的内存复用也如此。理解了这个一以贯之的"一次拿好、反复复用"，你就抓住了 ggml 性能设计的灵魂。
Last lesson we built the whole compute graph - but it is still just a "skeleton of pointers", not a number computed in it. This lesson makes that graph actually run: first allocate its memory, then compute each node in topological order; and if your machine has both a CPU and a GPU, coordinate the division of labor across multiple backends. This connects L07 (what a backend is) and L09 (what the graph is), completing the second half of "build first, execute later".
In other words, the last two lessons were all "theory on paper" - L08 prepared the memory pool, L09 drew the compute graph, but not a single real number was computed. This lesson is the "final kick": turning the graph into results. And you will see this kick is rather refined: not simply computing front to back, but using memory frugally and cleverly splitting the work across hardware. These two things turn "can compute" into "compute fast and frugally", earning ggml the name "high-performance inference engine".
Break "make a graph produce a result" apart and it is exactly three steps, in sequence:
The first step, building, was last lesson. This lesson stars the middle two: planning memory (deciding where each tensor's data sits in the buffer) and computing nodes (actually computing each operator in topological order). After these two, the once-empty result tensors get their data filled with real numbers - you can finally read results. Sounds plain, but "planning memory" hides a rather beautiful ggml optimization worth a close look. We cover "saving materials" (memory) first, then "dividing labor" (backends and scheduling).
Why is "planning memory" its own step rather than allocating on the fly as you compute? This is exactly the dividend of lazy building. Allocating on the fly, by layer 6 you have no idea whether layer 20 will still need layer 6's output, so you can only conservatively keep it. But now the graph is complete, so the planner can - before actually computing - sweep the whole graph front to back and work out clearly "when each block is used and when it can be released", then start executing. "Plan first, execute later" is like planning how to pack a suitcase before leaving rather than stuffing it as you walk - the former always packs tighter.
First a question: for a graph with thousands of nodes, must each node's output get its own block of memory? Doing so would make peak memory frighteningly large. But ggml does not - it notices many intermediates are "use-once and discard": layer 5's output, once fed to layer 6, is never needed again, so its memory can perfectly well be reused by layer 8's output. That is exactly what ggml-alloc (the graph allocator ggml_gallocr) does:
How can it reuse so precisely? All thanks to L09's complete graph. Because the graph spells out "who depends on whom", the allocator can pre-compute each tensor's "lifetime" - from which node it starts being needed, to which node after which no one uses it. Once a tensor "dies", its memory is immediately returned for later tensors to reuse. As pseudocode:
# cf. the gallocr planning logic in ggml/src/ggml-alloc.c (simplified) def plan(graph): for t in graph.nodes: # topological order t.offset = free_blocks.best_fit(nbytes(t)) # find a reusable free block for s in t.src: if last_use(s) == t: # s is used by no one after this free_blocks.give_back(s) # return its memory for reuse
The keywords are best_fit (pick the most suitably-sized free block) and give_back (return a tensor's memory once done). ggml-alloc keeps a set of "free blocks", finds the best fit when allocating, and merges adjacent free blocks when releasing. Because it saw the whole graph in advance, it reuses memory to the hilt - in practice peak memory is often a fraction of "each tensor its own block". This is one of the real payoffs of L09's "build first": without the complete graph, this global memory planning would be impossible.
A common misconception in passing: memory reuse does not affect correctness. Some worry "giving A's plot to C - won't it corrupt A's data?" - no, because reuse only happens after A is certainly needed by no one. The allocator strictly follows lifetimes: as long as anyone might still read A, A's memory is never requisitioned. So memory reuse is a completely lossless optimization - it saves space without touching results, the same thinking as the earlier distinction "quantization is lossy (L06), the KV cache is lossless (L04)".
With memory planned, we finally reach "actually compute". This step is carried out by a backend (the CPU/CUDA/Metal from L07). The simplest case uses one backend:
// simplified usage; spans ggml-backend.h / ggml-cpu.h / ggml-alloc.h ggml_backend_t be = ggml_backend_cpu_init(); // pick a backend (could be cuda/metal...) ggml_gallocr_alloc_graph(galloc, graph); // actually allocate per the plan ggml_backend_graph_compute(be, graph); // compute node by node!
What ggml_backend_graph_compute does is walk graph.nodes in topological order, calling that backend's operator implementation for each node (matmul calls the matmul kernel, softmax the softmax kernel... these kernels are the topic of L11 and Part 6). Because L09 guarantees topological order, when each node is reached its inputs are certainly already computed, so one front-to-back pass does it. After the last node, the output tensor holds the result. Leafs (weights, inputs) are never computed, only read by operators.
Dispel one illusion here: the name ggml_backend_graph_compute sounds "heavy", as if it does something enormous itself, but it is really more of a loop + dispatcher - the real grunt work (the thousands of multiply-adds in one matmul) happens inside each operator's kernel, and kernels are implemented by each backend separately (CPU with SIMD, CUDA with GPU kernels). So "executing a graph" is thin at this ggml layer: walk the nodes in order and shout "your turn" at each; "how to compute fast" is the topic of the next lesson (L11 operators) and Part 6 (kernels). This "scheduling is scheduling, computing is computing" layering is exactly why ggml can run the same graph on wildly different hardware.
One more easily-missed detail: execution does not start from the leafs but straight from the first node. Leafs (weights, inputs) already have their data before execution - weights loaded from the model file, inputs fed by you; they need no "computing". So ggml_backend_graph_compute's loop only walks graph.nodes, taking each node's src (inputs ready), calling the kernel to compute the result, filling its data. One pass from the first node to the last and the whole graph is done. Grasp this and you see why L09 stores leafs and nodes separately - precisely so the execution loop can cleanly "compute only nodes".
But reality is often more complex: you may want some layers on GPU, the rest on CPU (remember L07's -ngl?). Now one backend is not enough; you need a "scheduler" to coordinate multiple backends - that is ggml_backend_sched:
In one graph, some operators should run on GPU, some on CPU; data lives in two memories, needing transfer across devices. Who coordinates?
1. Split the graph into segments, grouped by device; 2. Assign each segment's operators to a suitable backend; 3. Copy - auto-insert data transfers at CPU/GPU boundaries.
Let me clarify the relation between ggml_backend_graph_compute and ggml_backend_sched: the former is "single-backend" execution - one graph, one device, computed straight front to back; the latter is the "multi-backend" conductor - it first splits the graph into segments, then hands each to ggml_backend_graph_compute to execute on its device. So sched is the higher-level coordinator, and single-backend execution is the basic tool in its hand. On CPU only you might use the former directly; to mix CPU/GPU you call in the latter. Get this containment straight and ggml's execution code will not dizzy you.
Why multiple backends rather than just cramming everything onto the GPU? Because VRAM often cannot hold the whole model. A big model that is still tens of GB after quantization may only fit half its layers on your card; the rest must stay in CPU memory and compute on CPU. This "half GPU, half CPU" hybrid execution is the norm for running big models on consumer hardware, and the fundamental reason ggml_backend_sched exists. It lets you cut flexibly by VRAM size: more VRAM, put more layers on GPU; less, fewer, with the CPU as backstop, so it always runs - just faster the more layers go on GPU.
Three questions below; open them if you want depth, skip them if you only want the main line.
Still the same line: thanks to L09's complete graph. With eager computation (compute and discard as you go), you cannot foresee "will this intermediate be needed later", so you conservatively keep them all, and memory cannot be saved.
With the complete graph, the allocator can compute exactly at which node each tensor is "last used". Past that point, its memory is immediately reclaimable. Combined with best_fit picking the most suitable free block and merging adjacent free blocks, the whole graph's peak memory is crushed. You could say lazy building saves memory, and this is the saving - which is why training/inference frameworks love compute graphs.
For a sense of scale: for a many-layer big model, "each intermediate tensor its own block" vs "reuse" can differ in peak memory by several times. On a consumer GPU already tight on VRAM, that "several times" is often the line between "it runs" and "out of VRAM, won't run". So memory reuse is not a nice-to-have tweak but one of the preconditions for many models to run on your machine at all.
Roughly two things: one, whose memory the tensor currently lives in (data already on GPU naturally favors computing on GPU); two, whether that backend supports this operator. If some operator is not yet implemented in the GPU backend, sched lets it fall back to CPU (the fallback mentioned in L07), guaranteeing the whole graph always runs.
Whenever a segment's output must feed the next segment on a different device, sched auto-inserts a "copy" node at the boundary, moving data from one memory to another. These copies have a cost, so "where to cut, which layers on GPU" affects performance - but sched handles these details, and you usually just give a single -ngl number.
There is also a finer cut called "tensor split": split one big matmul by columns into several pieces, give them to multiple GPUs to compute at once, then stitch the results. This suits multi-card setups, spreading a single operator's load across several cards. Whether "split by layer" or "split by tensor", sched coordinates behind the scenes - you just tell it how to cut via parameters like --split-mode, and the rest - splitting, assigning, copying, stitching - it does quietly. This "hide complex multi-device coordination behind a simple interface" design is exactly what makes ggml pleasant to use.
This is a classic ggml "configure once, execute repeatedly" pattern. reserve: first rehearse with the graph to measure "how much memory this graph needs at most", then open that buffer once. alloc: on each later execution, reuse this already-opened buffer, placing tensors at their planned positions.
Why two steps? Because opening big blocks (especially GPU VRAM) is expensive; you cannot reopen it every execution. Reserve once, measure the peak, open the buffer, and the thousands of later inference steps reuse the same buffer over and over, sparing repeated big allocations. This is of a piece with L08's arena idea: grab a big block once, shuffle within it repeatedly.
String this thread together and you find one belief running through all of ggml: do the large, expensive operations (asking the system for memory) as few times as possible - ideally once - then reuse internally. L08's arena is like this, reserve/alloc here is like this, L10's memory reuse is like this. Grasp this consistent "grab once, reuse repeatedly" and you have the soul of ggml's performance design.