上一课的 llama_decode 每次吃一个 llama_batch。这一课就拆开它：一个 batch 怎么同时装多个 token、每个 token 怎么带上"第几位、属于哪条序列、要不要输出"， 以及内部怎么被 llama_batch_allocr 切成小批（ubatch）喂给计算图。batch 是你和引擎之间那张"这一步要算什么"的订单。
为什么 batch 值得单讲？因为它是一个统一的接口：单条对话逐字 decode、多序列并行、prefill 整段 prompt——这些看着很不同的场景，到引擎眼里都只是"喂进来一个 batch"，区别全在 batch 里怎么填。搞懂 batch，你就懂了引擎"一次该算什么"是怎么被描述的。
先看这张订单的字段。一个 llama_batch 里，几个并行的数组共同描述"这一步要处理哪些 token、各自什么情况"。
| 字段 | 含义 |
|---|---|
| n_tokens | 这一批有多少个 token |
| token[] | 每个 token 的词 id |
| pos[] | 每个 token 的位置（喂给 rope 和 KV cache） |
| n_seq_id[] / seq_id[][] | 每个 token 属于哪条（或哪些）序列 |
| logits[] | 标志：这个 token 是否要算输出 logits |
// 简化自 include/llama.h struct llama_batch { int32_t n_tokens; llama_token * token; // 词 id llama_pos * pos; // 每个 token 的位置 int32_t * n_seq_id; llama_seq_id ** seq_id; // 属于哪条序列 int8_t * logits; // 标志: 是否输出 logits(源码注释: rename to "output") };
注意这是几个平行数组：token[i]、pos[i]、seq_id[i]、logits[i] 合起来，描述第 i 个 token 的全部信息。这种"结构数组"的布局，让一次塞进很多 token 变得简单——你只要把这几个数组按相同的下标填好，引擎就知道这一步要处理哪些 token、各自怎么对待。
最常用的构造是 llama_batch_get_one：把一串 token id 包成一个最简 batch（位置自动从 0 递增、单序列、只最后一个出 logits），适合"喂一段 prompt 或一个新 token"这种最常见的情形。需要更精细控制（多序列、自定义位置）时，再用 llama_batch_init 手动填那几个数组。
字段里还有个 embd（上面简化时略过了）：大多数时候你喂的是 token 的词 id（走 token），但有些场景（比如多模态、或外部已算好嵌入）想直接喂嵌入向量，就走 embd。两者二选一：要么给 id 让模型自己查嵌入，要么直接给嵌入。普通文本生成走 token 即可。
还要理解 batch 的定位：它是一个纯数据的输入容器，没有任何方法、不做任何计算。它只负责"把这一步要算的东西描述清楚"，真正的活全在 llama_decode 里。把"描述要算什么"和"真正去算"分成两个东西（batch 和 decode），是个清爽的接口设计——你填一张表，引擎照表干活。
具体感受一下：你在聊天框里发一句话，上层会先把它分词成一串 token id（L20），再把这些 id 填进一个 batch 的 token 数组、位置填进 pos、都归到同一个 seq_id，最后只在末位标 logits。这个 batch 一交给 llama_decode，模型就开始算"你这句话之后该接什么"。日常每一次对话，背后都是这样一张张 batch 在流转。
那个 logits 数组是这一课的一个关键。它是个开关数组：logits[i]=1 表示"我要第 i 个 token 的输出 logits"，=0 则表示"算它，但不用给我它的 logits"。
为什么要这个开关？因为算 logits 不便宜——它是一次"隐藏向量 -> 词表大小（n_vocab）"的大矩阵乘。而很多 token 的 logits 我们根本用不到：prefill 把整段 prompt 过一遍时，中间那些 token 只是为了把 K/V 填进 KV cache，只有最后一个的 logits 才用来预测下一个词。
所以这个标志直接省算力：标了几个位置，就只算几次输出投影，其余 token 算到隐藏向量为止、不做那次大矩阵乘。decode 阶段每步只新增一个 token、只它要 logits；prefill 整段只要末位。这套"按需算输出"，正是 L17 讲 logits 时说的"只在某些 token 上有"的来源。
顺带解释源码里那句 rename this to "output" 的注释：logits 这个名字其实有点窄——这个标志控制的是"要不要这个位置的输出"，而输出既可以是 logits（生成任务），也可以是嵌入向量（embedding 任务，L17 的 embd）。叫 output 更准确。知道这点，你看到 logits 这个字段名时就不会被它字面意思框住。
多序列场景下，这个标志更显灵活：同时给三个不同 prompt 各做 prefill，可以把它们的 token 全塞进一个 batch，只在每条序列各自的最后一个 token 上标 1。一次 decode，三条序列的下一个词 logits 都拿到了。这种"一批里多条序列、各取各的输出"的玩法，正是服务器批量处理多个请求的基础。
顺带一提，logits 标志和 seq_id 配合，能表达很精细的需求：比如一个 batch 里有三条序列，你可以只要其中两条的输出、第三条只填 KV 不要输出。这种"逐 token 级别的精确控制"，是把多个不同请求高效拼在一起算的前提——服务器正是靠这种精细，才能在一次 decode 里同时推进很多条对话。
把输出标志这件事和显存也连一下：n_outputs 越大，输出缓冲（每行 n_vocab 个 float）就越占内存。对词表几十万的大模型，多标几行输出，缓冲就多吃不少。所以"只标该标的"，省的不只是计算，还有那块输出缓冲的内存。又一个"精打细算"体现在一个小标志上的例子。
你提交的 batch（逻辑上"这一步要算这么多 token"）不一定能一次性塞进硬件算。引擎用 llama_batch_allocr 把它校验、补全、再切成物理可算的小批（ubatch），逐个送进计算图。
# 伪代码: batch 切成 ubatch(llama_decode 内部) alloc = llama_batch_allocr() alloc.init(batch) # 校验 pos/seq_id, 填默认 for ub in alloc.split_simple(n_ubatch): # 按物理批大小切成 ubatch decode_ubatch(ub) # 建图(L16)+执行(L10)
这里要分清两个"批大小"：n_batch 是逻辑批——你一次最多能提交多少 token；n_ubatch 是物理批——硬件一次真正高效处理多少 token。前者方便你"一次多交点活"，后者受限于硬件，allocr 就负责把大的逻辑批切成若干个物理批逐个算。
除了最简单的 split_simple，allocr 还有 split_equal、split_seq 等切法，应对多序列等更复杂的排布。但核心思想都一样：把"你想算的"翻译成"硬件能一口口吃下的"。这层切分，让上层只管描述意图、不用操心硬件一次能吃多少。
多说一句 init 这一步的"校验、补全"。它会检查你填的 pos、seq_id 合不合法（比如位置不能乱、序列号不能越界），并为你没填的字段补上合理默认（比如 pos 留空就按顺序自动编号）。这层把关，让上层调用方少踩坑——很多"喂错 batch"的错误，会在这里被当场拦下，而不是带着错继续算。
| 切法 | 怎么排 | 适用 |
|---|---|---|
| split_simple | 按顺序切 | 普通单序列（最常见） |
| split_equal | 多序列时各序列尽量均匀分布到每个 ubatch | 多序列并行 |
| split_seq | 把同一序列的 token 切到一起 | recurrent 类模型（L19 变体） |
为什么要分这么细？因为像 recurrent（L19 变体）这类模型对"同序列 token 要连续"有要求，不同切法是为了照顾不同模型的约束。普通模型用 split_simple 就够。
"逻辑批 vs 物理批"这层区分，其实是计算机里很常见的"提交和执行解耦"：你按方便提交一大批，系统按自己的节奏分批执行。数据库的批量插入、GPU 的 kernel 启动，都是类似套路。llama.cpp 把它用在 token 上：你按一句话、一段 prompt 的粒度提交，引擎按硬件能吃的粒度执行。
每个 ubatch 会触发一次完整的"建图 + 执行"：llama_decode 对每个 ubatch 调 build_graph（L16）搭出针对这批 token 的图、交后端执行（L10）。所以"一个大 batch"在内部可能变成"好几张图轮流跑"。理解了这点，你就明白为什么 batch 切分发生在 decode 内部、而不用你操心——它是连接"你的订单"和"实际计算"的那道自动工序。
把 batch 放回整条链路：是你（或上层框架）准备 batch -> llama_decode 吃 batch、切 ubatch、建图执行 -> logits 出来 -> 采样（L21）挑词 -> 把新词包成下一个 batch…… batch 就是这条循环里"每一圈的输入"。读懂它，你就握住了和引擎对话的那张"订单格式"。
一个常见的节奏：开头 prefill 时，把整段 prompt 的几十上百个 token 一次塞进一个大 batch（高效地一次填满 KV cache）；之后 decode 时，每步只喂一个新 token 的小 batch。同一个 llama_batch 结构，一会儿装很多、一会儿装一个——它的弹性，正好贴合 prefill/decode 这一快一慢的两段节奏（L03）。
顺带埋个伏笔：服务器为了榨干吞吐，会玩一种"连续批处理"（continuous batching）——把多个用户、不同进度的请求，按 seq_id 拼进同一个 batch 一起算，谁生成完了就把谁换出、把新请求换进。这套高级调度（第五部分会提）能成立，底层正是因为 batch 支持"一批里多条序列、各自独立"。你现在学的这个朴素的 batch 结构，撑起的是相当复杂的服务能力。
还有个实践细节：decode 循环里，每步那个"只装一个新 token"的小 batch，常常是复用同一块 batch 内存反复填的——不必每步都重新分配。配合上 L16 说的"图结构可复用"，连续 decode 其实相当轻量：同一张图、同一个 batch 壳子，每步只换里头那一个 token id 和位置。这就是逐字生成能那么快的工程细节之一。
再把 ubatch 这个词的来历点破：u 是 "micro"（微）的意思，ubatch 就是"微批"。它和 batch 的关系，就像"你下的一整单"和"厨房一锅锅做的小份"。这个命名本身就提示了它的角色——它是 batch 在物理执行层面被切细后的产物，是真正一次性送进硬件计算的最小单位。
pos 是每个 token 的位置。它有两个去处：一是喂给 rope（L16），让注意力知道两个 token 相距多远；二是写进 KV cache 的 cell（L19），标记"这个 K/V 是第几位的"。所以 pos 填错，位置编码和缓存都会乱。
seq_id 标明每个 token 属于哪条序列。一个 batch（和一个 context）可以同时装好几条不同的序列——比如同时给三个不同 prompt 生成回答。它们共享这份权重和这套调度，但各有各的 KV（按 seq_id 区分，L19），互不串味。
正是 pos + seq_id 这两样，让 batch 能精确表达"哪个 token、在哪条序列的第几位"。有了这个，多序列并行、同一序列里续写，都能在一个统一的 batch 接口里表达出来。
因为硬件一次能高效处理的 token 数是有限的（受显存、计算单元规模限制），这个上限就是 n_ubatch。如果你一次提交了很多 token（大的逻辑批），不切就可能塞不下、或者塞下了也不高效。
所以 allocr 把大的逻辑批切成若干个不超过 n_ubatch 的物理批，逐个算、把结果拼起来。对你来说，提交多少 token（n_batch）是"我想一次交多少活"的事；硬件一次算多少（n_ubatch）是"机器一口能吃多少"的事——两者解耦，互不打架。
这层切分还带来灵活：同样一段 prompt，在显存大的机器上可以用大 n_ubatch 一次多算、更快；显存小就用小 n_ubatch 多切几次、慢一点但跑得起来。把"逻辑意图"和"物理执行"分开，正是这种"同一份代码适配不同硬件"的底气。
关键在于"算 logits"是一次昂贵的操作：把最后的隐藏向量投影到词表大小（几万维），是一次大矩阵乘。如果每个 token 都做这一步，prefill 一段长 prompt 就要做几百上千次无用的大投影。
而我们真正需要 logits 的位置很少：decode 阶段每步只新增一个 token、只它要预测下一个；prefill 整段也只要最后一个。logits 标志就让引擎只在标了的位置做输出投影，其余 token 算到隐藏向量就停，省下大量大矩阵乘。
这和 L03 讲的 prefill/decode 节奏正好对应：prefill 是"把整段 prompt 一次过完、只取末位 logits"，decode 是"逐字生成、每步取新词的 logits"。两种节奏，都靠这个标志数组在 batch 层面精确表达"这一步谁要输出"。一个 int8 数组，省下的是实打实的算力。
Last lesson's llama_decode eats one llama_batch each time. This lesson takes it apart: how a batch holds many tokens at once, how each token carries "which position, which sequence, output or not", and how it is internally split by llama_batch_allocr into small batches (ubatch) fed to the compute graph. The batch is the "what to compute this step" order between you and the engine.
Why a whole lesson on the batch? Because it is one unified interface: word-by-word decode of a single conversation, multi-sequence parallelism, prefill of a whole prompt - these seemingly different scenarios are, to the engine, just "a batch fed in", with all the difference in how the batch is filled. Get the batch and you get how "what to compute at once" is described.
First, this order's fields. In a llama_batch, several parallel arrays together describe "which tokens this step processes, each in what situation".
| field | meaning |
|---|---|
| n_tokens | how many tokens in this batch |
| token[] | each token's word id |
| pos[] | each token's position (fed to rope and the KV cache) |
| n_seq_id[] / seq_id[][] | which sequence(s) each token belongs to |
| logits[] | flag: whether this token computes output logits |
// simplified from include/llama.h struct llama_batch { int32_t n_tokens; llama_token * token; // word id llama_pos * pos; // each token's position int32_t * n_seq_id; llama_seq_id ** seq_id; // which sequence int8_t * logits; // flag: output logits?(source comment: rename to "output") };
Note these are several parallel arrays: token[i], pos[i], seq_id[i], logits[i] together describe all of the i-th token's info. This "struct-of-arrays" layout makes stuffing many tokens at once simple - just fill these arrays by the same index and the engine knows which tokens this step processes and how to treat each.
The most common constructor is llama_batch_get_one: it wraps a run of token ids into a minimal batch (positions auto-increment from 0, single sequence, only the last emits logits), suited to the very common case of "feed a prompt or one new token". When you need finer control (multi-sequence, custom positions), use llama_batch_init to fill those arrays manually.
There is also an embd field (omitted in the simplification above): most of the time you feed token word ids (via token), but some scenarios (e.g. multimodal, or externally pre-computed embeddings) want to feed embedding vectors directly, via embd. The two are either-or: give ids and let the model look up embeddings, or give embeddings directly. Plain text generation uses token.
Understand the batch's role too: it is a pure-data input container, with no methods and no computation. It only "describes what to compute this step", with all the real work in llama_decode. Splitting "describe what to compute" from "actually compute" into two things (batch and decode) is a clean interface design - you fill a form, the engine works by the form.
Concretely: you send a sentence in a chat box, the upper layer first tokenizes it into a run of token ids (L20), fills those ids into a batch's token array, positions into pos, all under one seq_id, and finally flags logits only on the last. Hand this batch to llama_decode and the model starts computing "what follows your sentence". Every everyday conversation is, behind the scenes, such batches flowing.
That logits array is a key point of this lesson. It is a switch array: logits[i]=1 means "I want the i-th token's output logits", =0 means "compute it, but I do not need its logits".
Why this switch? Because computing logits is not cheap - it is a big "hidden vector -> vocab size (n_vocab)" matmul. And many tokens' logits we never use: when prefill passes a whole prompt through, those middle tokens are just there to fill K/V into the KV cache, and only the last one's logits are used to predict the next word.
So this flag directly saves compute: however many positions are flagged, that many output projections are done, while other tokens stop at the hidden vector without that big matmul. In decode each step adds one token and only it needs logits; prefill needs only the final position. This "compute output on demand" is the source of L17's "logits only on some tokens".
By the way, that source comment rename this to "output": the name logits is actually a bit narrow - this flag controls "do we want this position's output", and output can be logits (generation tasks) or embedding vectors (embedding tasks, L17's embd). "output" is more accurate. Knowing this, the field name logits will not box you in by its literal meaning.
In multi-sequence scenarios this flag shows more flexibility: prefilling three different prompts at once, you can stuff all their tokens into one batch and flag 1 only on each sequence's last token. One decode, and all three sequences' next-word logits are obtained. This "multiple sequences in one batch, each taking its own output" is the basis for a server batch-processing multiple requests.
By the way, the logits flag together with seq_id can express very fine needs: say a batch has three sequences, you can want output from only two and have the third just fill KV without output. This "per-token precise control" is the precondition for efficiently splicing multiple different requests to compute together - it is exactly this granularity that lets a server advance many conversations in one decode.
Tie the output flag to VRAM too: the larger n_outputs, the more the output buffer (n_vocab floats per row) takes. For a large model with a vocab of hundreds of thousands, flagging a few more output rows eats notably more buffer. So "flag only what should be flagged" saves not only compute but also that output-buffer memory. Another example of frugality embodied in one tiny flag.
The batch you submit (logically "this step computes this many tokens") may not fit into the hardware in one go. The engine uses llama_batch_allocr to validate, fill in, then split it into physically-computable small batches (ubatch), fed one by one into the compute graph.
# pseudocode: batch split into ubatch (inside llama_decode) alloc = llama_batch_allocr() alloc.init(batch) # validate pos/seq_id, fill defaults for ub in alloc.split_simple(n_ubatch): # split into ubatch by physical batch size decode_ubatch(ub) # build graph(L16)+execute(L10)
Distinguish two "batch sizes": n_batch is the logical batch - how many tokens you can submit at most at once; n_ubatch is the physical batch - how many tokens the hardware efficiently processes at once. The former lets you "hand over more work at once", the latter is hardware-limited, and the allocr splits a big logical batch into several physical batches computed one by one.
Beyond the simplest split_simple, the allocr has split_equal, split_seq, and more, for multi-sequence and other complex arrangements. But the core idea is the same: translate "what you want to compute" into "what the hardware can swallow bite by bite". This split lets the upper layer just describe intent, not worry about how much hardware eats at once.
A bit more on init's "validate, fill in". It checks whether the pos and seq_id you filled are legal (e.g. positions cannot be disordered, sequence ids cannot overflow), and fills sensible defaults for fields you left out (e.g. leaving pos empty auto-numbers in order). This gatekeeping spares callers pitfalls - many "wrong batch" errors are caught here on the spot rather than computing on with the error.
| Split | How it arranges | Used for |
|---|---|---|
| split_simple | cut in order | plain single sequence (most common) |
| split_equal | with multiple sequences, distribute each evenly across ubatches | multi-sequence parallel |
| split_seq | group one sequence's tokens together | recurrent-style models (L19 variant) |
Why so fine-grained? Because models like recurrent (an L19 variant) require "same-sequence tokens be contiguous", and different splits accommodate different model constraints. Plain models use split_simple.
This "logical vs physical batch" distinction is actually computing's common "submit and execute decoupling": you submit a big batch for convenience, the system executes in batches at its own pace. Database bulk inserts and GPU kernel launches are similar patterns. llama.cpp applies it to tokens: you submit at the granularity of a sentence or a prompt, the engine executes at the granularity hardware can eat.
Each ubatch triggers a full "build graph + execute": llama_decode calls build_graph (L16) per ubatch to assemble the graph for that batch of tokens and hands it to the backend (L10). So "one big batch" may internally become "several graphs run in turn". Understand this and you see why batch splitting happens inside decode, with no worry on your part - it is the automatic step connecting "your order" and "actual computation".
Put the batch back into the whole pipeline: you (or an upper framework) prepare a batch -> llama_decode eats the batch, splits ubatch, builds and executes -> logits come out -> sampling (L21) picks a word -> the new word is wrapped into the next batch... The batch is "each round's input" in this loop. Read it and you hold the "order format" for conversing with the engine.
A common rhythm: at the start, prefill stuffs a whole prompt's tens-to-hundreds of tokens into one big batch (efficiently filling the KV cache at once); then decode feeds a small batch of one new token each step. The same llama_batch structure, sometimes holding many, sometimes one - its elasticity fits exactly the fast/slow two-phase rhythm of prefill/decode (L03).
A foreshadow: to squeeze throughput, servers play a "continuous batching" - splicing multiple users' requests at different progress into one batch by seq_id to compute together, swapping out whoever finishes and swapping in new requests. That advanced scheduling (mentioned in Part 5) works precisely because the batch supports "multiple independent sequences in one batch". The plain batch structure you are learning now upholds quite complex serving capability.
Another practical detail: in the decode loop, that small batch "holding just one new token" each step is often refilled into the same batch memory - no need to reallocate every step. Together with L16's "reusable graph structure", consecutive decode is quite lightweight: the same graph, the same batch shell, each step swapping only that one token id and position. This is one of the engineering details behind word-by-word generation being so fast.
Unpack the word ubatch: the u means "micro", so ubatch is "micro-batch". Its relation to batch is like "the whole order you placed" versus "the small portions the kitchen cooks pot by pot". The name itself hints at its role - it is the product of a batch split fine at the physical-execution level, the smallest unit actually sent to hardware to compute at once.
pos is each token's position. It has two destinations: one, fed to rope (L16) so attention knows how far apart two tokens are; two, written into the KV cache cell (L19), marking "which position this K/V is". So a wrong pos messes up both position encoding and the cache.
seq_id marks which sequence each token belongs to. One batch (and one context) can hold several different sequences at once - say, generating answers for three different prompts simultaneously. They share these weights and this scheduling but each has its own KV (distinguished by seq_id, L19), none mixing flavors.
It is exactly pos + seq_id that let a batch precisely express "which token, at which position of which sequence". With this, multi-sequence parallelism and continuing within one sequence are both expressible through one unified batch interface.
Because the number of tokens hardware can efficiently process at once is limited (by VRAM and compute-unit scale), and that limit is n_ubatch. If you submit many tokens at once (a big logical batch), without splitting it might not fit, or fit but inefficiently.
So the allocr splits a big logical batch into several physical batches no larger than n_ubatch, computing each and stitching results. To you, how many tokens to submit (n_batch) is "how much work I want to hand over"; how many hardware computes at once (n_ubatch) is "how much the machine eats in one bite" - the two decoupled, not clashing.
This split also brings flexibility: the same prompt, on a high-VRAM machine, can use a large n_ubatch to compute more at once, faster; with low VRAM, a small n_ubatch splits more times, slower but runnable. Separating "logical intent" from "physical execution" is exactly the confidence behind "the same code fitting different hardware".
The key is that "computing logits" is an expensive operation: projecting the final hidden vector to vocab size (tens of thousands of dims) is a big matmul. If every token did this, prefilling a long prompt would do hundreds or thousands of useless big projections.
And the positions where we truly need logits are few: in decode each step adds one token and only it predicts the next; prefill of a whole segment needs only the last. The logits flag makes the engine do the output projection only at flagged positions, stopping other tokens at the hidden vector, saving a lot of big matmuls.
This corresponds exactly to L03's prefill/decode rhythm: prefill is "pass a whole prompt once, take only the last position's logits", decode is "generate word by word, take the new word's logits each step". Both rhythms express "who outputs this step" precisely at the batch level via this flag array. An int8 array, saving real compute.