C API 总览The C API

Across the first four parts we took apart the whole internal machine - how a .gguf file is loaded, how the graph is built, how it infers, samples, and is constrained (L14-L24). But how does the outside world actually drive that machine? The answer is the set of C functions in include/llama.h - llama.cpp's main switch: llama-cli, llama-server, and the Python / Go / Rust / Node bindings all load the model, feed in tokens, and read out results through this one stable C interface.

From this lesson on we switch angle: the first four parts were about "how it turns inside"; from now on we re-thread the whole project along "how you use it from outside". This lesson meets the three cornerstones of that C API - opaque handles (you only get a pointer, never the internals), the typical call sequence (the fixed arc from init to free), and the C++ RAII wrappers (which free your handles automatically). These three look scattered but interlock: the handles are what you operate on, the call sequence is the order you operate in, and the RAII wrappers clean up after you. Connect them and you hold a key to reading any higher-level code.

Handles and ownership

This C API does not spread the model's internal structs out for you; it only hands you a few opaque handles: what you get is a pointer whose contents the library owns internally - you cannot see, and should not touch, its fields. Why "just a pointer, implementation hidden" is good comes later; first meet the handles you deal with most. This is not to make life hard: hiding the fields lets the library reshuffle its internal layout without disturbing callers, and keeps you from corrupting state by poking a field you were never meant to touch. All you do is treat a handle like a claim ticket - carry it to the matching function, and never mind how the shelves behind the counter are arranged.

Here llama_model holds read-only knowledge: model weights never change once loaded (echoing L14's loading and L17's read-only sharing), so many sessions can safely share one copy - load once, infer in many places, only one copy in memory. This pays off especially on the server side: when dozens of requests arrive on one machine, they can share a single llama_model and each open only a lightweight llama_context, instead of duplicating the multi-GB weights dozens of times. Splitting the "heavy but constant" weights from the "light but changing" session state is the key to this API saving memory and scaling concurrently.

And llama_context is per-session state: it holds the KV cache, compute resources, and logits buffers (echoing L17's context and L19's KV cache), one per session, each isolated from the rest. The vocab llama_vocab is pulled from the model via llama_model_get_vocab (L20), and the sampler llama_sampler is L21's sampler chain. Note the llama_vocab handle: it belongs to the model (the tokenizer rules ship with the model, after all), so you neither load nor free it separately - you just take a pointer and use it. This "one handle derived from another" relationship is common in the C API, and being clear on who owns whom is what keeps your frees correct.

The handles are yours, so freeing them is your job too. In plain C this means calling the matching _free by hand: llama_free releases the context, llama_model_free the model, llama_backend_free tears down the global backend - and the order matters (free the dependent first, then what it depended on): a context is built from a model and depends on it, so you must free the context first and the model second; do it the other way and the context becomes a dangling pointer. Miss a free and you leak memory, get the order wrong and you crash - in plain C, all of this is on you to watch.

// C: you own every handle, and must free it yourself (include/llama.h)
llama_model   * model = llama_model_load_from_file(path, mparams);
llama_context * ctx   = llama_init_from_model(model, cparams);
// ... use ...
llama_free(ctx);              // free the context first
llama_model_free(model);      // then the model

// C++: include/llama-cpp.h wraps a handle in unique_ptr, freed at scope exit
llama_model_ptr   model(llama_model_load_from_file(path, mparams));
llama_context_ptr ctx(llama_init_from_model(model.get(), cparams));
// ... use ... scope end auto-calls llama_free / llama_model_free

C++ users get a tidier option. include/llama-cpp.h, a tiny header of about 30 lines, defines a std::unique_ptr alias for each handle - llama_model_ptr, llama_context_ptr, llama_sampler_ptr, llama_adapter_lora_ptr - each with a deleter that calls the matching _free. A handle is released the moment it leaves scope, so you never miss one or get the free order wrong. Declare the handles in stack order, built before destroyed, and destruction runs in reverse order automatically - exactly the "free the context, then the model" rule from earlier. You hand the error-prone manual bookkeeping to the compiler to enforce, which is precisely what C++ RAII is good at.

The typical call sequence

With the handles known, see how they string together. Almost every program using llama.cpp shares the same skeleton pipeline: init the backend, load the model, build a context, get the vocab, tokenize the text, feed it in to decode, take the logits, sample one token, turn it back into text, loop, and finally free things one by one. Whether it is a tiny example of a few dozen lines or a mature service like llama-server, the skeleton never escapes this line; the only difference is that the server splits "load" and "loop" across threads and adds caching and concurrent scheduling. Burn this trunk into your head and you will not get lost in any higher-level code.

The first few steps are one-time setup: llama_backend_init starts the global backend; llama_model_load_from_file loads the model by path (if the model is split into shard files, use llama_model_load_from_splits instead); llama_init_from_model builds a context from the model; llama_model_get_vocab pulls out the vocab for later. Across these steps the parameter objects mparams and cparams decide a lot: the model params carry whether to mmap and how many layers to offload to the GPU (echoing L07's -ngl); the context params carry n_ctx (how large the context window is), n_batch (how many tokens at most per batch), and more. In other words, those command-line options all end up as fields in these two structs passed in here.

# pseudocode: one full C-API generation loop (simplified from include/llama.h)
llama_backend_init();                                # global backend init
model = llama_model_load_from_file(path, mparams);   # multi-shard -> llama_model_load_from_splits
ctx   = llama_init_from_model(model, cparams);       # old name llama_new_context_with_model (DEPRECATED)
vocab = llama_model_get_vocab(model);              # read-only vocab
smpl  = llama_sampler_chain_init(sparams);          # build a chain, then add top_k/top_p/temp/dist

n     = llama_tokenize(vocab, prompt, tokens, ...);  # text -> token ids
batch = llama_batch_get_one(tokens, n);             # simplest single-sequence batch
while (more) {
    llama_decode(ctx, batch);                       # run the compute graph
    logits = llama_get_logits(ctx);               # one score per vocab token
    id     = llama_sampler_sample(smpl, ctx, -1);   # the chain picks the next token
    piece  = llama_token_to_piece(vocab, id, ...);   # token -> text piece
    batch  = llama_batch_get_one(&id, 1);           # feed it back, keep looping
}
llama_free(ctx); llama_model_free(model); llama_backend_free();   # free one by one

Then comes the autoregressive main loop: llama_decode runs the graph once, llama_get_logits reads out a score per token for this step, the chain via llama_sampler_sample picks the next token, llama_token_to_piece turns it back into a text piece, and llama_batch_get_one feeds it back to decode the next step. When the loop ends, free in the order context -> model -> backend. Here you see the point of splitting handles again: on every turn of the loop the model never moves, and the only things that change are the KV cache and the position counter inside the context - exactly the payoff of separating the "heavy, constant weights" from the "light, changing state".

A flow chart is still not concrete enough. Let us walk one minimal example - feeding the model a single "Hi" - through the whole chain, to see exactly what we hold at each step: the trace below swaps abstract function names for concrete inputs and outputs, and walking the six stations you find every step is just "take the previous step's product, call one function, get the next step's product" - nothing mysterious.

Why a C ABI

Step back and ask: why expose a C interface rather than more modern C++ classes? The answer is ABI stability. C's function signatures and memory layout are the safest "lowest common denominator" across languages and compilers, and once fixed they rarely change; C++ class layouts and name mangling drift with compiler and version, unfit for a stable cross-language boundary. The same C++ class can produce binary interfaces that do not line up between GCC and Clang; C's calling convention, by contrast, has stayed stable for decades, and nearly every programming language has a built-in ability to "call a C function" - which is what lets one engine be reused from so many languages.

A stable C interface plus opaque pointers doubles the effect: the caller sees only a pointer, never the C++ classes, field layouts, or templates behind it. So the library authors can refactor the inside freely, and as long as those few dozen C function signatures hold, every caller stays unaffected. This is also why llama.cpp can frequently change algorithms, swap data structures, and add new backends internally, while people using the Python bindings outside almost never have to change their code along with it.

It is exactly this "stable surface plus free interior" that lets llama.cpp be embedded almost anywhere: the Python, Go, Rust, and Node bindings are all a thin wrapper over this same set of C functions. One stable C ABI holds up the whole multi-language ecosystem above it - the root reason it "runs everywhere". The various "local LLMs" you see in phone apps, desktop software, and cloud services, dug all the way down, mostly call these same few C functions; because the base is stable enough, this investment pays compounding returns year after year.

Deep dive: the batch and API evolution

Finally, two folds to fill in two details that often trip up newcomers: how tokenized tokens actually get filled into a batch, and why DEPRECATED keeps showing up in the source.

// the input struct llama_decode consumes (simplified from include/llama.h)
struct llama_batch {
    int32_t        n_tokens;   // how many tokens this batch carries
    llama_token  * token;      // the token ids (L18/L20)
    llama_pos    * pos;        // position of each token in its sequence
    llama_seq_id** seq_id;     // which sequence each token belongs to
    int8_t       * logits;     // 1=output logits for this token, 0=skip
};