张量是什么What is a tensor

ggml is llama.cpp's compute engine, and in its eyes all data is "tensors" - weights, activations, the KV cache, all tensors. This lesson makes the ggml_tensor struct tangible: what shape is, what stride is, why ggml is "row-major", and why "transposing a tensor" costs almost nothing. Understand this layer and you can read every compute-graph and operator diagram that follows; tensors are, you could say, the first building block for reading ggml source.

A tensor = shape + type + one contiguous block of memory

A tensor is really "one contiguous block of memory + a manual for how to read it". The manual has: the data type (type, e.g. F32, F16, Q4_0), how many elements per dimension (ne[], up to 4 dims in ggml), and a pointer data to that memory. First, what "shape" looks like:

A quick clarification of "how many dims": 0-D is a scalar (one number), 1-D a vector (e.g. one embedding), 2-D a matrix (e.g. a layer's weights), and 3-D / 4-D show up in "batched, multi-head" intermediates. ggml infers a tensor's rank from how many of ne[] are greater than 1; unused higher dims are filled with 1. So ne=[4,3,1,1] is really just a 4x3 two-dimensional tensor.

Note ggml's convention: ne[0] is the innermost, fastest-changing dimension (laid out contiguously in memory), ne[1] is "the next row", and so on. This is the opposite of the "rows x cols" ordering many people are used to - a dedicated deep-dive below covers this trap. Here are the core fields of ggml_tensor:

struct ggml_tensor {
    enum ggml_type   type;           // F32 / F16 / Q4_0 ...
    int64_t          ne[4];          // #elements per dim (ne[0] = innermost)
    size_t           nb[4];          // byte strides
    enum ggml_op     op;             // how it was produced (graph node)
    struct ggml_tensor * src[...];    // inputs (back-pointers)
    struct ggml_tensor * view_src;     // set if this tensor is a view
    void *           data;           // the actual bytes
    char             name[...];
};  // simplified from ggml/include/ggml.h, GGML_MAX_DIMS = 4

Two groups of fields are worth remembering. ne / nb / type / data describe "what this data looks like and where it is"; while op / src describe "how it was computed" - op is the operator that produced it (e.g. matmul), src are back-pointers to input tensors. That second group is the key to ggml's "build the graph first, then execute": every tensor remembers its origin, and stringing them together via src is a compute graph (mentioned in lesson 03, expanded in Part 3). Don't overlook type either: the same 1000 elements take 4000 bytes as F32 but only ~500 as Q4_0 - the type alone decides how big this memory is, which is the root of how quantization saves memory.

About type: ggml supports a long list of data types - full-precision F32, half-precision F16 / BF16, and a whole family of quantized types (Q4_0, Q8_0, Q4_K, etc. - next lesson). The same tensor struct and the same ne/nb logic hold all of them, because one field, type, uniformly describes "how many bytes per element (or per block), and how to interpret them". This "same structure, swappable type" design lets quantization plug seamlessly into the existing tensor and operator machinery without rewriting it per precision. Because of this, switching quantization levels (say Q4_0 to Q5_K) is nearly transparent to higher-level code - only the type and per-block byte count change; the ne/nb logic is untouched.

You might ask: to hold a multi-dim array, why not just use C++'s std::vector or a raw array - why a dedicated ggml_tensor? Because ggml needs far more than "store some numbers": it must describe computation as a graph (via op/src), let data live in different backends' memory (via buffer), uniformly hold many types including quantized ones (via type and per-block nb), and reshape with zero copies (via ne/nb and view_src). All these needs together produced this seemingly simple but carefully designed struct - the engine's "atom".

Row-major and stride: how nb[] is computed

Memory is actually one-dimensional (a long strip of bytes), but tensors are multi-dimensional; what flattens many dims into one is stride. ggml uses nb[i] to record "how many bytes to skip in memory when dim i increases by 1". ggml is row-major: the ne[0] dimension is laid out contiguously with the smallest stride. Here is how an ne=[3,2] (3 per row, 2 rows) F32 tensor lies in memory:

This byte stream makes it click: adjacent elements in a row are 4 bytes apart (one F32), so nb[0]=4; jumping from row 0 to row 1 (the highlighted a10) skips a whole row of 3 elements = 12 bytes, so nb[1]=12. As a formula - straight from the comments in ggml.h: nb[0]=ggml_type_size(type) (bytes per element), nb[1]=nb[0]*(ne[0]/ggml_blck_size(type)) (bytes to cross one row; for plain types just "bytes-per-element x elements-per-row"), and above that nb[i]=nb[i-1]*ne[i-1] (strictly there is also an alignment "padding" term, which is 0 for contiguous unpadded tensors and is omitted here). So given any multi-dim index, dot it with nb to get the byte offset:

# multi-dim index (i0, i1, i2, i3) -> byte offset in memory
offset = i0*nb[0] + i1*nb[1] + i2*nb[2] + i3*nb[3]
ptr    = (char*)tensor->data + offset   # the address of this element

Why 4 dims (GGML_MAX_DIMS = 4)? Because inference tensors rarely exceed 4 dims - a typical shape is "batch x sequence x heads x per-head-dim", and 4 is enough; storing ne/nb in a small fixed-size array is simple and fast, with no dynamic allocation. One more easily-missed point: the memory data points to is not necessarily on the CPU - it may live in GPU memory, and the tensor has a separate buffer field recording "which backend owns this data". That foreshadows the multi-backend story of lesson 07 (the same tensor struct, with data living on different hardware).

Walking the formula through a concrete example makes it stick. Using the same ne=[3,2] F32 tensor from the memory diagram above: each element is 4 bytes, so nb[0]=4; crossing a whole row skips 3 elements, so nb[1]=4x3=12. To fetch row 1, element 2 (index i1=1, i0=2, zero-based - a12 in the diagram), the byte offset is 1*nb[1] + 2*nb[0] = 12 + 8 = 20 - exactly the 20 marked under a12. The whole tensor is 3x2x4 = 24 bytes. So with ne, nb, and data, any element's address is one step away - that is the full power of stride; other shapes like ne=[4,3] are left for this lesson's closing exercise.

Conversely, knowing the byte layout explains why "row-major traversal" is usually faster than "column-major": walking along ne[0] (within a row) touches contiguous, adjacent bytes, friendliest to the CPU cache; jumping along higher dims takes a big stride each time and misses the cache more often. Many operator implementations deliberately loop along the contiguous dimension for exactly this reason - layout decides performance, an intuition we will reuse when reading kernel implementations later.

Why view / transpose copy no data

Now a very "thrifty" ggml design. Since "shape (ne/nb)" and "data" are stored separately, many "reshape" operations need not touch the data at all - just change a few numbers. The classic is transpose: turning a [rows, cols] matrix into [cols, rows], ggml merely swaps ne[0] with ne[1] and nb[0] with nb[1], moving data by not a single byte:

One common operation in passing: broadcast. When two not-quite-same-shape tensors do an element-wise op (e.g. adding the same bias vector to every row), ggml lets some dimension use "one element as many" - again via a stride trick: set that dim's nb to 0, so no matter how the index changes the address does not, and the same value is re-read, looking "copied" while taking not one extra byte. Another "change nb, don't move data" classic, of a piece with views and transpose.

Take transpose itself, laid out in full: the same row of memory, read across as the original and down as the transpose - none of the 6 values moved.

A very common reshape: take activations of shape [n_embd, n_tokens] and "fan them out" for multi-head attention into [head_dim, n_head, n_tokens] - the element count is unchanged (n_embd = head_dim x n_head) and no data moves; only ne/nb are reinterpreted, turning "one big vector" into "several heads' small vectors". The multi-head attention from lesson 04 is full of these "same data, different shape" operations, all done with views and almost no extra copies.

To spell out what "contiguous" means: a tensor is contiguous when its elements sit adjacent in memory, in ne[0], ne[1], ... order without gaps (i.e. nb follows the formula above exactly). Transpose and some slices break this: same elements, but the "walk order" no longer matches the memory layout, so ggml_is_contiguous returns false. Most operators handle contiguous tensors directly; where contiguity is required, ggml_cont copies the data into a fresh tidy block in the current shape. Remember this and you will dodge many shape-related debugging traps.

The struct also has a humble but handy field, name: each tensor can carry a name. This helps when debugging (spotting "which weight is this" when printing the graph), and GGUF stores each weight tensor with a name (like blk.0.attn_q.weight), matched up by name at load time. So a "name" is not just a comment - it is the index between model weights and code. The tensor names you see in GGUF tools or debug logs come straight from this field.

Going deeper (optional)

Three questions below; open them if you want depth, skip them if you only want the main line.