ggml 核心对象ggml core objects

Part 2 gave you the intuition for tensors, quantization, and backends; from Part 3 on, we take the ggml engine apart. First stop: memory - how does ggml manage the memory of thousands of tensors? The answer is surprisingly plain: a memory pool called ggml_context, plus a "never scatter-malloc" allocation philosophy. Get this layer and the compute graph, execution, and operators all have ground to stand on.

First, the problem at hand: one inference conjures up thousands of tensors out of thin air - weights, intermediate activations, the KV cache, all tensors. If each one went to the operating system for memory, the overhead of "request, bookkeep, return" alone would drag the whole engine down. ggml's answer is: don't buy retail, buy wholesale. It fences off one big block at once and carves it carefully inside - that block is ggml_context. This lesson is about how it fences, how it carves, and why it is designed this way.

While we are at it, let me name the technique: this way of "grab one big block, carve internally, free wholesale" has a common name in systems programming - an arena (or region) allocator, also called a linear allocator. It is not ggml's invention but a well-worn trick in high-performance code; ggml just applies it to the perfectly-suited scenario of "managing all the tensors of one compute graph". When we say arena below, we mean that big block held by ggml_context.

The three foundation pieces

At ggml's lowest layer there are really just three things buckled together: a config (how big a pool you want), the pool itself, and the objects lined up one after another inside it (each wrapping a tensor or a compute graph). First, how they relate:

The keyword here is "pool": ggml does not scatter-request memory per tensor; it grabs one big block once and carves inside it. The next three sections walk through these three pieces.

Let me plant a contrast that runs through the whole lesson: "metadata" and "data" are two different things. Metadata is "what this tensor looks like" - its shape ne, strides nb, type, and op/src (the fields from L05) - adding up to mere tens or hundreds of bytes; data is "the big slab of actual floats", where one weight matrix might be tens of MB. The ggml_context arena very often holds only metadata, leaving the heavy data to the backend. Read on with this "light metadata / heavy data" split in mind and many design choices fall into place.

ggml_context: a memory pool

It all starts with ggml_init(params). You tell ggml three things in ggml_init_params and it hands back a ggml_context - a ready memory pool:

// simplified from ggml/include/ggml.h
struct ggml_init_params {
    size_t mem_size;     // total arena size
    void * mem_buffer;   // NULL = ggml allocates this block internally
    bool   no_alloc;     // true = build tensor "metadata" only, no data buffer
};

ctx = ggml_init(params);   // grab the whole arena at once
// ... build many tensors in ctx ...
ggml_free(ctx);             // free it all at once

ggml_init is straightforward: it allocates a little space for the ggml_context management struct itself, then prepares that big arena - if you passed a mem_buffer it uses your memory; pass NULL and it does ggml_aligned_malloc(mem_size) itself (see ggml/src/ggml.c). ggml_free returns the whole arena (only freeing the block if ggml allocated it itself, i.e. mem_buffer_owned). That "you can pass in mem_buffer" matters: ggml can work on memory someone else gave it, handy for embedding into various environments or reusing one buffer to build graphs repeatedly.

So strictly, the difference between ggml_aligned_malloc and plain malloc is just "alignment": plain malloc only guarantees big-enough, not that the address falls on a boundary; ggml's memory must start at a multiple of some alignment (16 or 32 bytes), so the backend can confidently use aligned SIMD loads to move a batch at once. This small thing reflects ggml's bias of "always making way for backend compute".

This mechanism is everywhere in llama.cpp. When loading a model, the loader first estimates how big a metadata arena it needs from the tensor count and sizes in the GGUF header, ggml_inits a (usually no_alloc=true) context, and registers every weight tensor in it; the weights' real data is taken up by backend buffers (or even mmap'd straight from the file, see L13). Each inference step uses another context to temporarily build that step's compute graph, discarded once done. So you can picture ggml_context as ggml's most basic "workbench": to work, set up a bench; when done, clear bench and materials together. Understand it and you understand "where" all of ggml's data structures live.

no-malloc: bump allocation

Once it has the arena, every tensor or graph it builds does not go back to the system for memory; it just "pushes forward one slot" inside the arena. That is bump (pointer-bump) allocation: keep a "how far used" cursor, and for each object move the cursor right and place it in situ:

Each object is prefixed with a tiny ggml_object header (recording its offset, size, and a pointer to the next object); they form a linked list, and the cursor always rests at the end of the last object. A new object is carved from the cursor forward. As pseudocode it is clear at a glance:

A word more on what that ggml_object header holds: its offset offs in the arena, the size size it occupies, a pointer next to the next object, plus a type field marking "tensor or graph". ggml_context itself tracks the list head/tail (objects_begin / objects_end) and the object count n_objects. With a tail pointer, "append at the end" is O(1) - another face of why bump is fast.

# cf. ggml_new_object / ggml_new_tensor_impl in ggml/src/ggml.c
def new_object(ctx, size):
    cur = ctx.objects_end.offs + ctx.objects_end.size   # current cursor
    if cur + size > ctx.mem_size:                       # pool is out of room
        abort("arena out of space")                     # no growth, just fail!
    obj = place_at(ctx.mem_buffer + cur)                # place in situ
    link_into(ctx.objects, obj)                         # append to the list
    return obj

For a concrete feel: just how light is "metadata"? A ggml_tensor struct plus object header, as measured by ggml_tensor_overhead(), is only a few hundred bytes. Even a graph with thousands of tensors totals just a few hundred KB of metadata - next to the multiple GB of weight data, practically nothing. This again confirms the "light metadata / heavy data" split: under no_alloc=true, the ggml_context arena need only be a few MB to comfortably hold the whole graph's skeleton; the truly memory-hungry data lives elsewhere.

Put the two allocation styles side by side and the arena's benefits are obvious:

One easily-missed detail: bump allocation naturally yields determinism. Because objects line up strictly in creation order, the same graph-building code produces the exact same memory layout every run - useful for debugging, reproduction, and for the backend reading/writing at fixed offsets. By contrast, malloc returns unpredictable addresses that may differ each run. ggml's "predictable contiguous layout" is the invisible premise that lets it move a whole graph to other memory (e.g. plan on CPU first, then map to a GPU buffer).

no_alloc: metadata only, no data

Back to that no_alloc in ggml_init_params. When it is true, ggml reserves space in the arena only for a tensor's "metadata" (type, ne, nb, op, src - from L05), not for the tensor's "data" (that big slab of floats).

Why? Because often all we want is to "build the compute graph first" - that step only needs each tensor's shape and dependencies, not real data memory yet. Once the graph is built and the whole picture is clear, we hand it to the backend to allocate the real data buffers in one pass (exactly the premise of L09's lazy build and L10's memory reuse). So no_alloc=true is the switch for the "describe first, allocate later" approach, which you will see again and again as llama.cpp loads models and builds graphs.

To wrap up this lesson's view of memory: a full usage is one line - ggml_init fences the land (gets the arena) -> build tensors and the compute graph in the ctx (bump-carve metadata) -> hand to the backend to allocate real data and execute (L09, L10) -> ggml_free clears it all. You will notice ggml_context always plays the role of a lightweight scaffold: it makes the building process cost almost no memory, no fragmentation, no leaks, leaving the truly heavy lifting (the multi-GB weight data) for after the whole picture is clear, carried by a more hardware-aware backend. The foundation only "holds the frame steady" - exactly as it should.

So what you should really take away is not a few function names but a mental model: every tensor and every graph in ggml lives in some ggml_context arena; they line up via bump, no individual malloc; metadata is light in the ctx, data is heavy in the backend buffer; clear the whole block when done. With this model, the next lesson goes to see how the tensors built on this arena are strung via op/src into a compute graph.

Going deeper (optional)

Three questions below; open them if you want depth, skip them if you only want the main line.