课 01 给了你一条最小主线（加载 -> 分词 -> 解码循环 -> 采样）。这一课把它放慢成慢镜头： 看清一个 token 是怎么从 prompt 一步步算出来的，又怎么被回灌进队尾、驱动下一步。 我们先走一遍七步数据流，再放大其中最重的"一次 decode"——看清它内部其实是"先建计算图、再交后端执行、最后吐出 logits"； 最后用 prefill / decode 两种节奏和 KV cache，讲清为什么这个循环能在本地便宜地一圈圈转下去。
把"一段 prompt 变出下一个 token"拆开，正好是这 7 步，从上到下顺次流过去：
把 prompt 文本切成一串 token id 序列——模型只认数字 id，不认字符；同一句话用不同分词器切出的 id 可能完全不同。
src/llama-vocab.cpp · llama_tokenize
把 token 序列包成一次输入（batch）；用 llama_batch_get_one 时，位置 pos 与序列 seq_id 由 llama_decode 自动补（位置顺序排、序列固定为 0），需要多序列 / 自定义位置时才用 llama_batch_init。
src/llama-batch.cpp · llama_batch_get_one
llama_decode 跑一次前向；内部先建计算图，再交给后端真正算在硬件上。这一步最重，下一节会专门把它放大看。
src/llama-context.cpp · llama_decode；建图 src/llama-graph.cpp（llm_graph_*）+ src/llama-model.cpp；执行 ggml-backend
从这次前向里拿到"下一个 token 的分数向量"——词表里每个 token 各有一个分。注意此刻还没有选定任何 token。
src/llama-context.cpp · llama_get_logits_ith
采样器链（sampler chain）按策略（贪心 / top-k / top-p……）从 logits 里选出一个 token；策略不同，同一份 logits 也会选出不同的字。
src/llama-sampler.cpp · llama_sampler_sample
先用 llama_vocab_is_eog 判断是不是结束符；不是，就用 llama_token_to_piece 把 token 还原成文字输出。
src/llama-vocab.cpp · llama_vocab_is_eog · llama_token_to_piece
把新 token 作为下一步输入再 decode；过去 token 的 K/V 已存在 KV cache 里，无需重算——于是循环每转一圈只多算一个 token。
src/llama-kv-cache.cpp
这 7 步里有几处容易一带而过、其实值得多看一眼。第 1 步"分词"切出的是 subword（子词），既不是按单字、也不是按整词：llama.cpp 的分词器走的是 BPE / SPM / WordPiece 这类子词算法（见 include/llama.h 的 LLAMA_VOCAB_TYPE_SPM / _BPE / _WPM），一个常见英文单词可能正好是一个 token，而生僻词或一个汉字往往被拆成好几个子词片。这也正好解释了第 6 步为什么要用 llama_token_to_piece 把 token 还原成"词片"——多个片拼起来才是一个完整的词或汉字，所以你看到的输出是"一个 token 一个 token 地往外蹦"，而不是规规矩矩一个字一个字地出。换句话说，token 并不等于"词"，它只是模型词表里的一个最小单位。这也带来一个很实际的后果：token 数和字符数往往对不上——同一段话，中文、英文、代码切出的 token 数可能差很多，而上下文窗口、计费、速度都是按 token 算的，不是按字数算的。
第 2 步"组批"里的 pos 与 seq_id 也值得点破。pos 是每个 token 在序列里的位置下标（第 0、1、2…个），模型靠它给注意力补上"谁先谁后"的位置信息；seq_id 标的则是这个 token属于哪一条序列。为什么需要后者？因为一次 batch 里可能同时塞进多条互不相干的对话（服务端并发、批量推理正是这么干的），seq_id 让它们各自的 K/V 互不串台、各算各的。用 llama_batch_get_one 时这两样会由 llama_decode 自动补好（位置顺序排、序列固定为 0）；只有要多序列或自定义位置时，才需要 llama_batch_init 手动填（字段定义见 include/llama.h 的 llama_batch）。
七步里最"重"的是第 3 步 decode。别把它当成一个黑盒——拆开看，一次前向内部正好是三小步串起来：
所以"一次 decode"= 建计算图 + 后端执行 -> logits：先由 src/llama-graph.cpp 的 llm_graph_*（经 src/llama-model.cpp 的 build_graph 拼出）把这一步运算描述成一张图，再交 ggml-backend 调度到硬件上真正算，算完用 llama_get_logits_ith 取出"下一个 token 的分数向量"。它的产出是 logits——不是文字，也不是已经选好的 token。
有人会问：既然每步 decode 都要"先建图"，那这张图是不是每次都从头搭、很费？其实不必担心——建图只是搭一张"算子骨架"（描述谁连谁、张量多大），并不真的搬运权重数据，开销远小于真正的矩阵乘；而且 decode 每步只处理一个新 token，这张图本身也很小。所以"建图 + 执行"里真正的重头始终是后端执行那一段，建图更像每圈开跑前快速摆好的赛道。
那"前向"内部到底在算什么？顺着 src/models/llama.cpp 的建图读，是一条很直的链：先把 token id 经 build_inp_embd 查成词向量（embedding），再让它穿过多层 transformer block（每层大致是"自注意力 + 前馈网络"，层数由 n_layer 决定），末了过一道输出归一化、由输出层（lm_head）投影回词表大小，得到每个候选 token 的分数。这里还藏着一个省算力的细节：decode 时其实只需要最后一个位置那一行结果——build_inp_out_ids（src/llama-graph.cpp）负责只挑出"要输出的位置"，所以第 4 步取 logits，取的正是"最后一个位置"对应的那一行分数（这也是 llama_get_logits_ith(ctx, -1) 里那个 -1 的含义）。
把这两种节奏摆到一条时间线上，差别一眼就清楚：prefill 是一段宽的并行块，decode 是一格一格往后接的小步。
为什么 prefill 能并行、decode 却必须串行？关键在"谁依赖谁"。prefill 面对的是已经完整给定的 prompt，里面每个 token 都是已知的，它们的 K/V 彼此不依赖，于是可以一口气并行算完、一次把缓存填满。decode 正相反：要算"下一个词"，前提是"上一个词已经写出来了"——第 n+1 步的输入，恰是第 n 步采样刚选中的那个 token。这种"后一步依赖前一步的产出"的链条天然无法并行，只能一步一个地往后挪。这也是单条对话生成速度上不去的根因之一：哪怕硬件还很闲，decode 也得乖乖排队、一格一格地走。
Prefill 把整段提示词一次并行算完；之后每步 decode 只算 1 个新 token，所以"接着往下写"很便宜。
"分块"其实分两层，别混了：一次 llama_decode 能接收的 token 数有个逻辑上限 n_batch（默认 2048），这是你一次最多能塞进去的量；真正落到硬件上算时，又会按物理块 n_ubatch（默认 512）进一步切开、一块块并行跑（两个默认值都在 common/common.h）。所以遇到很长的 prompt，prefill 既可能被拆成多次 decode（受 n_batch 约束），每次内部又再切成若干 n_ubatch 小块；但无论怎么切，同一块之内始终是并行的，这正是 prefill 远比逐个 decode 快的原因。
承上：decode 之所以每步只算一个新 token，靠的就是 KV cache。它把每个算过的 token 的 K/V 存下来，下一步直接复用，免去重算整段历史：
每一行只多出一个高亮新格（+K6、+K7），前面灰掉的部分都是"复用、不重算"。没有这层缓存，生成第 n 个 token 就得把前面 n-1 个全重算一遍；有了它，每步的新增计算基本是常数——这就是自回归循环能在本地便宜地一直转下去的原因。
那 K/V 到底是什么、又凭什么能缓存？注意力机制里，每个 token 都会被算出三样东西：Query（拿去"问"的向量）、Key（被查的"标签"）、Value（携带的"内容"）。算"当前 token 该关注谁"时，要拿它的 Query 去和所有历史 token 的 Key 逐一比对，再按比对出的权重，把对应的 Value 加权汇总起来。关键就在这里：历史 token 的 K 和 V 一旦算出便不再改变（它们只取决于那个 token 本身和它的位置），所以完全可以存下来反复复用——这正是 KV cache 缓存的东西。代价也很直接：缓存要为每一层、每个历史 token 各存一份 K 和 V，占用的内存/显存随上下文长度线性增长；上下文开到几万 token 时，KV cache 会吃掉相当可观的一块内存，这也是"上下文窗口"为什么总有上限、长上下文为什么格外吃硬件的原因之一（分配、写入与复用都在 src/llama-kv-cache.cpp）。
// 课 01 主线的"慢镜头"：每一步对应一个调用 tokens = llama_tokenize(vocab, prompt) // 1 分词 batch = llama_batch_get_one(tokens) // 2 组批 loop: llama_decode(ctx, batch) // 3 前向(内部建图 + 后端执行) logits = llama_get_logits_ith(ctx, -1) // 4 取 logits id = llama_sampler_sample(smpl, ctx, -1) // 5 采样 if llama_vocab_is_eog(vocab, id): break // 6 结束? print(llama_token_to_piece(vocab, id)) // 6 还原文字 batch = llama_batch_get_one([id]) // 7 回灌; KV cache 记住过去
第 4 步的 logits：真实代码里采样器会自己从 ctx 取，这里单独列出只为把"产出 logits"这一步看清楚。
循环体就是"自回归"引擎：每转一圈吐一个 token，直到 llama_vocab_is_eog 命中结束符才停。
"自回归"三个字落到这段代码上就很具体了：第 7 步把刚生成的 id 重新包成 batch 喂回去，下一圈的输入就含了上一圈的输出——模型一边写、一边把自己写出来的字当作新的上下文继续往下写。也正因为每圈只回灌一个新 token、过去的 K/V 又都在缓存里，每一圈的实际计算量几乎是常数，循环才能这样一圈圈稳稳地转下去，而不是越写越慢。
下面三个常见问题，想深究的同学点开看；只想抓主线的可以先跳过。
一句话：一次前向只算到"打分"为止。logits 是词表上每个 token 的"分数向量"——词表多大它就多长，每个 token 一个分，谁高谁低而已，还没"拍板"。
选哪个是另一步：从这串分数里挑出一个 token，是采样器的事（src/llama-sampler.cpp 的 llama_sampler_sample，按贪心 / top-k / top-p 等策略选）；把选中的 token 再还原成文字，是 llama_token_to_piece（src/llama-vocab.cpp）的事。
为什么这么分：把"打分 / 选词 / 还原文字"三件事拆开，采样策略就能随意替换而不动前向——同一份 logits，换个采样器就有不同风格的输出。
采样具体怎么挑：最简单的贪心（greedy）直接选分数最高的那个 token；但实际生成更常用温度（temperature）先把这串分数"摊平或拉尖"以调节随机性，再用 top-k（只在分数前 k 名里挑）、top-p（只在累计概率刚够 p 的那一小撮里挑）把候选范围收窄，最后按概率随机抽一个。所以"同一份 logits、换个采样器"才能给出从一板一眼到天马行空的不同风格——这部分后面会有专门一课展开，这里只需先记住"logits 负责打分、采样器负责拍板"。
再补一句"分数"的性质：logits 是未归一化的原始打分，可正可负、加起来也不等于 1，并不是现成的概率。要变成"每个 token 的概率"，还得再过一道 softmax（指数化后归一化）；温度其实正作用在这一步之前——把 logits 整体放大或缩小，softmax 出来的分布就更尖或更平。想通这层，就明白"贪心"为什么能跳过 softmax 直接取最大值：只比大小的话，归不归一化都不影响谁最大。
省的是"重算过去"：没有它，每生成一个新 token 都要把前面所有 token 重算一遍 -> 重算成本约 O(n^2)；有了它每步只算新 token 的 K/V 并追加进缓存 -> 重算降到 O(n)。
给个体感：假设上下文已经有 1000 个 token，现在要生成第 1001 个。没有缓存，这一步得把前面 1000 个 token 的 K/V 全部重算一遍；有了缓存，只需算第 1001 个这一个 token 的 K/V，再追加到缓存末尾，省下的几乎是整段历史的重复前向。把每一步都这么省下来，一整段生成累计的重算量，就从 O(n^2) 量级压到 O(n) 量级——这正是自回归生成能在本地"一直往下写而不越写越慢"的根本。
注意别夸大：注意力对历史的扫描仍是每步 O(n)（要看过去所有 token），省掉的是重复计算过去 token 的 K/V，不是把注意力也变成常数。
为什么长上下文这么"吃"硬件：KV cache 的占用大致正比于"层数 × 上下文长度 × 每层 K/V 的宽度"——层数和宽度由模型定死，唯一会涨的就是上下文长度。于是把上下文从几千开到几万，KV cache 就要成倍变大，往往成为权重之外最显眼的一块内存。这也解释了为什么本地跑长上下文时，光有"装得下权重"的内存还不够，得额外给 KV cache 留足空间；想省，就只能在更短的上下文、更省的缓存量化或共享 K/V 的注意力结构之间做权衡。
源码：缓存的分配、写入与复用在 src/llama-kv-cache.cpp；上下文越长，这块占用越大，也是本地推理要预留内存的地方。
一句话：ggml 先把这一步运算描述成一张图（节点是算子：matmul、rope、softmax……，边是数据流），再交后端按图执行。
谁来建：图由 src/llama-graph.cpp 的 llm_graph_* 搭骨架、由 src/llama-model.cpp 的 build_graph 按具体模型结构拼出；建好后交 ggml-backend 调度执行。
为什么分两步：把"描述运算"与"执行运算"分开，同一张图就能落到不同后端（CPU / CUDA / Metal……）上跑，也便于做内存复用、算子融合等优化——这是 ggml 能"一处描述、多端执行"的根。
这样分到底换来什么：因为图只是"描述"、并不绑定具体硬件，同一套模型结构不改一行就能落到 CPU、CUDA、Metal、Vulkan 等不同后端上跑；而且图一旦建好，调度器还能在真正开算之前统筹全局——做内存复用（算完即可丢弃的中间张量不必各占一块显存）、算子融合（把几个小算子并成一个、少几趟读写）、以及把彼此无依赖的分支并行起来等优化。这种"先把整张图看全、再决定怎么算"的余地，正是"边算边定"的即时执行模式很难拥有的。
Lesson 01 gave you the minimal main line (load -> tokenize -> decode loop -> sample). This lesson plays it in slow motion: how one token is produced from the prompt step by step, then fed back to the tail of the queue to drive the next step. We first walk the 7-step data flow, then zoom into the heaviest part - "one decode" - to see it is really "build a compute graph, run it on the backend, then emit logits"; finally we use the prefill / decode rhythms and the KV cache to explain why this loop can keep turning cheaply, round after round, on local hardware.
Break "turn a prompt into the next token" apart and it is exactly these 7 steps, flowing top to bottom:
Cut the prompt text into a sequence of token ids - the model understands numeric ids, not characters; the same sentence can split into completely different ids under a different tokenizer.
src/llama-vocab.cpp - llama_tokenize
Wrap the token sequence into one input (batch); with llama_batch_get_one, pos/seq_id are auto-assigned by llama_decode (sequential positions, sequence 0) - use llama_batch_init only when you need multiple sequences / custom positions.
src/llama-batch.cpp - llama_batch_get_one
llama_decode runs one forward pass; internally it first builds the compute graph, then hands it to the backend to run on hardware. This is the heaviest step - the next section zooms into it.
src/llama-context.cpp - llama_decode; graph build src/llama-graph.cpp (llm_graph_*) + src/llama-model.cpp; execution ggml-backend
Read out the "score vector for the next token" from this forward pass - one score per token in the vocabulary. At this point no token has been chosen yet.
src/llama-context.cpp - llama_get_logits_ith
The sampler chain picks one token from the logits by some strategy (greedy / top-k / top-p...); a different strategy can pick a different token from the very same logits.
src/llama-sampler.cpp - llama_sampler_sample
First use llama_vocab_is_eog to test for an end token; if not, llama_token_to_piece turns the token back into text.
src/llama-vocab.cpp - llama_vocab_is_eog - llama_token_to_piece
The new token becomes the next step's input and we decode again; past tokens' K/V already live in the KV cache, so nothing is recomputed - each turn of the loop adds just one token of work.
src/llama-kv-cache.cpp
A few spots in these 7 steps are easy to skim past but worth a second look. Step 1 "tokenize" cuts the text into subwords - not whole words, and not single characters: llama.cpp's tokenizers use subword algorithms like BPE / SPM / WordPiece (see LLAMA_VOCAB_TYPE_SPM / _BPE / _WPM in include/llama.h). A common English word may be exactly one token, while a rare word or a single CJK character is often split into several subword pieces. That is also why step 6 needs llama_token_to_piece to turn a token back into a "piece" - several pieces glue together into one full word or character, so output comes out "one token at a time" rather than one tidy character at a time. In other words, a token is not a "word"; it is just the smallest unit in the model's vocabulary. This has a very practical consequence: token counts and character counts rarely match - the same passage tokenizes into very different counts for Chinese, English, or code, and the context window, billing and speed are all measured in tokens, not characters.
The pos and seq_id in step 2 "batch" are also worth spelling out. pos is each token's position index in the sequence (0, 1, 2...), which the model uses to give attention its "who comes before whom" information; seq_id marks which sequence a token belongs to. Why need the latter? Because one batch may hold several unrelated conversations at once (exactly what server-side concurrency and batched inference do), and seq_id keeps their K/V from crossing wires, each computed on its own. With llama_batch_get_one these two are auto-filled by llama_decode (sequential positions, sequence 0); only for multiple sequences or custom positions do you fill them by hand with llama_batch_init (fields defined in llama_batch in include/llama.h).
The heaviest of the 7 steps is step 3, decode. Don't treat it as a black box - opened up, one forward pass is exactly three little steps chained together:
So "one decode" = build graph + run on backend -> logits: first llm_graph_* in src/llama-graph.cpp (assembled by build_graph in src/llama-model.cpp) describes this step's computation as a graph, then ggml-backend schedules it onto hardware to actually compute, and afterwards llama_get_logits_ith reads out the "score vector for the next token". Its output is logits - not text, and not an already-chosen token.
You might ask: if every decode step has to "build a graph" first, is rebuilding it from scratch each time expensive? No need to worry - building the graph only assembles an "operator skeleton" (describing what connects to what, and tensor sizes); it does not actually move weight data, so its cost is far below the real matrix multiplies. And since each decode step processes only one new token, the graph itself is small. So within "build + run", the real heavy part is always the backend execution; building the graph is more like quickly laying out the track before each lap.
So what does the "forward pass" actually compute inside? Reading the graph build in src/models/llama.cpp it is a very straight chain: first the token id is looked up into a word vector (embedding) via build_inp_embd, then it passes through several transformer blocks (each roughly "self-attention + feed-forward network", with the number of layers set by n_layer), and finally goes through an output norm and is projected by the output layer (lm_head) back to vocabulary size, giving a score for each candidate token. There is also a compute-saving detail hidden here: during decode you only need the result at the last position - build_inp_out_ids (src/llama-graph.cpp) selects just the "positions to output", so step 4 reads exactly the row of scores for the "last position" (which is what the -1 in llama_get_logits_ith(ctx, -1) means).
Put the two rhythms on one timeline and the difference is obvious at a glance: prefill is one wide parallel block, decode is cell-by-cell steps appended after it.
Why can prefill run in parallel while decode must be serial? It comes down to "what depends on what". Prefill faces an already fully given prompt where every token is known; their K/V do not depend on each other, so they can all be computed in parallel in one shot and fill the cache at once. Decode is the opposite: to compute the "next word", the precondition is that "the previous word has already been written" - the input to step n+1 is exactly the token that step n's sampling just chose. This "each step depends on the previous step's output" chain is inherently impossible to parallelize; it can only inch forward one step at a time. That is one root reason a single conversation's generation speed has a ceiling: even if the hardware is idle, decode still has to queue up and go cell by cell.
Prefill computes the whole prompt in parallel in one pass; afterwards each decode step computes just 1 new token, so "keep writing" is cheap.
"Chunking" actually happens at two levels, don't mix them up: one llama_decode call has a logical cap n_batch (default 2048) on how many tokens you can submit at once; when it actually runs on hardware, it is further split into physical chunks of n_ubatch (default 512) and run chunk by chunk in parallel (both defaults live in common/common.h). So for a very long prompt, prefill may be split into several decode calls (bounded by n_batch), each internally cut into several n_ubatch chunks; but however it is cut, within one chunk it is still parallel - which is exactly why prefill is far faster than decoding one token at a time.
Following on: the reason decode computes only one new token per step is the KV cache. It stores each computed token's K/V so the next step reuses them directly, sparing a recompute of the whole history:
Each row adds only one highlighted new cell (+K6, +K7); everything greyed out before it is "reused, not recomputed". Without this cache, generating the n-th token would recompute all n-1 before it; with it, the added work per step is essentially constant - that is why an autoregressive loop can keep running cheaply on local hardware.
So what exactly are K/V, and why can they be cached? In the attention mechanism, every token gets three things computed: a Query (the vector it uses to "ask"), a Key (the "label" it is matched against), and a Value (the "content" it carries). To compute "who should the current token attend to", you take its Query and compare it against the Keys of all past tokens, then weight-sum the corresponding Values by the resulting weights. Here is the key point: a past token's K and V never change once computed (they depend only on that token itself and its position), so they can simply be stored and reused - which is exactly what the KV cache holds. The cost is direct too: the cache must store one K and one V for every layer and every past token, so its memory footprint grows linearly with context length; push the context to tens of thousands of tokens and the KV cache eats a sizable chunk of memory - one reason a "context window" always has an upper bound and long context is especially hardware-hungry (allocation, writing and reuse all live in src/llama-kv-cache.cpp).
// "slow motion" of lesson 01's main line: one call per step tokens = llama_tokenize(vocab, prompt) // 1 tokenize batch = llama_batch_get_one(tokens) // 2 batch loop: llama_decode(ctx, batch) // 3 forward (build graph + run on backend) logits = llama_get_logits_ith(ctx, -1) // 4 get logits id = llama_sampler_sample(smpl, ctx, -1) // 5 sample if llama_vocab_is_eog(vocab, id): break // 6 end? print(llama_token_to_piece(vocab, id)) // 6 detokenize batch = llama_batch_get_one([id]) // 7 feed-back; KV cache remembers the past
In real code the sampler reads the logits from ctx itself; the explicit step-4 logits line is shown only to make the "produce logits" step visible.
The loop body is the "autoregressive" engine: each turn emits one token, until llama_vocab_is_eog hits an end token.
"Autoregressive" gets concrete on this very code: step 7 wraps the just-generated id back into a batch and feeds it in, so the next lap's input contains the previous lap's output - the model writes while treating what it just wrote as new context to keep writing. And precisely because each lap feeds back only one new token while the past K/V all sit in the cache, the actual work per lap is nearly constant, which is why the loop can keep turning steadily round after round instead of slowing down as it goes.
Three common questions below; open them if you want depth, skip them if you only want the main line.
In one line: a forward pass only goes as far as "scoring". logits is a score vector over the whole vocabulary - as long as the vocab is big, with one score per token; it just says who is higher or lower, nothing is "decided" yet.
Picking is a separate step: choosing one token out of those scores is the sampler's job (llama_sampler_sample in src/llama-sampler.cpp, by greedy / top-k / top-p...); turning the chosen token back into text is llama_token_to_piece's job (src/llama-vocab.cpp).
Why split it: separating "score / pick / detokenize" lets the sampling strategy be swapped freely without touching the forward pass - same logits, a different sampler, a different style of output.
How sampling actually picks: the simplest, greedy, just takes the highest-scoring token; but real generation more often uses temperature to first "flatten or sharpen" the scores to tune randomness, then top-k (pick only among the top k scores) and top-p (pick only within the smallest set whose cumulative probability just reaches p) to narrow the candidates, and finally samples one by probability. That is how "same logits, a different sampler" can range from buttoned-up to wildly creative - a dedicated lesson will unpack this later; here just remember "logits do the scoring, the sampler makes the call".
One more note on what "scores" are: logits are unnormalized raw scores - they can be positive or negative and do not sum to 1, so they are not ready-made probabilities. Turning them into "a probability per token" takes a softmax (exponentiate then normalize); temperature acts right before this step - scaling the logits up or down makes the softmax distribution sharper or flatter. Once you see this, it is clear why greedy can skip softmax and take the max directly: if you only compare magnitudes, normalizing or not does not change which one is largest.
It saves "recomputing the past": without it, every new token re-runs all previous tokens -> the recompute cost is ~O(n^2); with it each step only computes the new token's K/V and appends it -> recompute drops to O(n).
A feel for it: say the context already has 1000 tokens and you want to generate the 1001st. Without the cache, this step would recompute the K/V of all 1000 prior tokens; with it, you only compute the K/V of the 1001st token and append it to the end - saving nearly a whole history's worth of repeated forward work. Save that at every step and the total recompute for a full generation drops from the O(n^2) ballpark to the O(n) ballpark - the very reason autoregressive generation can "keep writing without getting slower and slower" locally.
Don't overstate it: attention's scan over history is still O(n) per step (it must look at all past tokens); what is saved is recomputing past tokens' K/V, not turning attention itself into a constant.
Why long context is so hardware-hungry: the KV cache footprint is roughly proportional to "layers x context length x the K/V width per layer" - layers and width are fixed by the model, so the only thing that grows is context length. Push the context from a few thousand to tens of thousands and the KV cache grows in proportion, often the biggest block of memory after the weights themselves. That is why running long context locally needs more than enough memory to "fit the weights" - you must reserve extra room for the KV cache; to save it, you can only trade among a shorter context, cheaper cache quantization, or attention that shares K/V.
Source: allocation, writing and reuse of the cache live in src/llama-kv-cache.cpp; the longer the context, the bigger this footprint - and the memory you must reserve for local inference.
In one line: ggml first describes this step's computation as a graph (nodes are operators: matmul, rope, softmax...; edges are data flow), then the backend executes the graph.
Who builds it: the graph skeleton comes from llm_graph_* in src/llama-graph.cpp, assembled per the concrete model structure by build_graph in src/llama-model.cpp; once built it is handed to ggml-backend to schedule and run.
Why two steps: separating "describe" from "execute" lets the same graph run on different backends (CPU / CUDA / Metal...), and makes optimizations like memory reuse and operator fusion possible - the root of ggml's "describe once, run on many backends".
What this split actually buys: because the graph is only a "description" and not bound to specific hardware, the same model structure runs on different backends (CPU, CUDA, Metal, Vulkan...) without changing a line; and once the graph is built, the scheduler can take a global view before any computation starts - doing memory reuse (intermediate tensors that can be freed right after use need not each hold their own block), operator fusion (merging several small ops into one, saving a few read/write passes), and parallelizing branches that have no dependency on each other. This room to "see the whole graph first, then decide how to compute" is exactly what eager, compute-as-you-go execution struggles to have.