llama.cpp 是什么What is llama.cpp

llama.cpp is an LLM inference engine written in plain C/C++: it takes an already-trained model (stored as a GGUF file) and runs it efficiently - on an ordinary CPU, even a phone, and faster with a GPU. It does not train models; it only focuses on "turning a model into text". One executable plus one .gguf file lets you chat with an LLM locally, offline, and cheaply - and that is what makes it so appealing.

What problem does it solve

Research models mostly use Python + PyTorch: heavy dependencies, hungry for VRAM, hard to deploy on ordinary devices. Trying to run a 7B model on your own laptop, you often hit "can't install the environment", "not enough VRAM", or "tens of GB of weights to download". llama.cpp aims for the opposite - making inference, that one job, extremely light:

Why C/C++ specifically, and not Python, Rust, or Go? An inference engine cares about two things above all - a zero runtime and being embeddable - and C/C++ delivers both. It compiles straight to machine code, with no Python interpreter and none of the garbage collection or VM runtime of Go/Java, so it computes the moment it starts with predictable latency. It can be statically linked into one standalone executable that runs on another same-architecture machine with zero dependencies installed. Crucially, almost any language can call it through the C ABI, so the engine can be embedded into desktop apps, Android/iOS, the browser (compiled to WASM), even embedded devices - hard to pull off in a heavy-runtime language. Rust could reach a similar goal, but when the project started the C/C++ compilers and each vendor's GPU toolchains were the most mature and the easiest to port to. The official README states the goal up front as "minimal setup with top performance", and plain C/C++ is the linchpin of that sentence.

Three pillars: GGUF - quantization - ggml

The reason llama.cpp can "run anywhere from a single file" is three building blocks fitting together. Here is a first glance at each; later lessons expand every one of them:

(1) GGUF: one file holds the whole model

GGUF is llama.cpp's model file format: it bundles the weights, hyper-parameters (layers, dimensions, vocab size), tokenizer, chat template, and more into a single file. Loading just mmaps it into memory - no extra config files or Python code. Hand the engine one .gguf and it knows "what model this is and how to run it".

What is so good about "one file"? The traditional way needs weight shards, config.json, tokenizer.json, a generation config and more all present at once - miss one and nothing runs, mismatch a version and things break. GGUF welds them all into a single file, so it is config-free: the engine reads the metadata stored as key-value pairs in the file header (hyper-parameters, vocab, chat template) and knows exactly how to run. Loading mmaps the file in on demand - the OS reads a page only when it is touched - so startup is fast and multiple processes can share one read-only copy of the weights to save memory. The most practical payoff: swapping models means swapping one file - replace the .gguf, leave the command-line flags untouched, and you are running a different model. Distribution and management become trivially simple.

(2) Quantization: shrink the weights, keep the accuracy

Raw weights are usually 16-bit floats (FP16), so a 7B model needs about 14 GB. Quantization packs each weight into a lower bit-width (e.g. 4 bits), cutting the size to roughly 1/4 - small enough for an ordinary laptop's RAM. The cost is a tiny accuracy loss, but a "per-block shared scale" design keeps that loss almost unnoticeable:

Why can it "save space yet barely lose quality"? Three things stack up. First, a model has far more weights than runtime activations, so compressing the weights to a low bit-width saves the most space while touching compute precision the least. Second, "predicting the next token" depends on the relative ranking of the candidate logits, which is insensitive to tiny per-weight errors - the jitter from low bit-width mostly washes out. Third, a per-block shared scale lets each small block hug the value range of its own slice, firmly preserving the dynamic range. There is a cost - a little quality loss - but an importance matrix can keep the more critical weights more accurate and push that loss down further.

Put that compression in a real setting and it is a "heavy-to-light" pipeline: the same model shrinks sharply after quantization, and paired with one executable it can move from the data center onto your laptop:

(3) ggml: the low-level tensor engine

ggml is llama.cpp's in-house tensor library: it defines tensors, describes one inference run as a compute graph, then dispatches the ops in that graph (matmul, softmax, rope...) to different backends (CPU SIMD, CUDA, Metal, Vulkan...) to actually compute. By separating "describing the math" from "running it on hardware", the same model code runs unchanged across CPU and all kinds of GPUs.

Structure map: four layers, bottom-up

Stack those three pillars by "who depends on whom" and llama.cpp becomes a clean four-layer tower, from the hardware at the bottom up to the user-facing tools:

Keep this line in mind when reading the source: the backend provides compute, ggml organizes the math into a graph, src/llama-* assembles that graph into "session-level" inference logic, and the outermost tools/ is what you actually type. The next lesson maps these four layers onto concrete directories.

There is a pleasing symmetry to the direction: your request flows top-down through the four layers (a tool receives the prompt -> the inference layer organizes it into a compute graph -> ggml arranges the ops -> a backend lands them on hardware), and the computed result flows bottom-up back into text on your screen. If you ever get lost reading the source, return to this picture and first pin down "which layer am I in" - it usually settles the panic.

Training vs inference: which side is llama.cpp on

An LLM's life has two phases: first training (teaching it), then inference (putting it to work). These are two jobs with completely different needs:

llama.cpp does only the inference half. Precisely because it need not support backprop or optimizers, it can drop the heavy training-framework dependencies and compress the whole engine into a single slab of plain C/C++ - the prerequisite for running light on your machine. How the model is "produced" is not its concern; that is the job of training frameworks like PyTorch.

What does "forward only" concretely mean? To recompute gradients during backprop, training keeps every layer's intermediate results from the forward pass, and maintains optimizer state per weight (e.g. Adam's first- and second-moment momentum) - adding up to several times the memory of the weights themselves. Inference cuts all of that: no backward pass, no gradients, no optimizer state, and the weights are read-only and never change after loading, so memory really only holds two things - weights and the KV cache. Because the weights are read-only, you can safely quantize them to a low bit-width without worrying about training; and because the heavy training burden is gone, the whole thing is light enough to run on a single ordinary CPU. That is the entire confidence that "inference only" buys.

How it differs from PyTorch / transformers / vLLM

All of these deal with LLMs, yet they sit at different positions. A side-by-side comparison makes llama.cpp's distinct niche clear:

In one line: pick PyTorch to train / do research, vLLM for high-concurrency cloud serving, and llama.cpp to run a model lightly on local / edge / embedded devices. They do not replace each other - each has its job.

One often-overlooked fact best captures its "foundation" status: many of the local-LLM desktop tools you know - Ollama, LM Studio, Jan, KoboldCpp, LocalAI and more - are, to a large degree, calling llama.cpp underneath (most of them are listed right in llama.cpp's own README under "UIs"). In other words, you may never have typed its commands directly, yet you very likely use it indirectly every day; it is less a standalone command-line toy and more the engine under the whole local-inference ecosystem.

How to actually run it

The fastest way needs no code at all: download a .gguf and chat right from the command line with llama-cli:

# fastest way to run: one executable + one .gguf
llama-cli -m model.gguf -p "explain quantization in one sentence"

Here -m points at the model file and -p gives the prompt; press enter and the model starts emitting text token by token. Want a browser-accessible "local ChatGPT"? Swap llama-cli for llama-server - it wraps the same inference logic into an HTTP service exposing an OpenAI-compatible API. So what actually happens behind that command? Unpacked, it is just these few steps in the C API:

// simplified minimal inference flow from include/llama.h
llama_backend_init();

llama_model   *model = llama_model_load_from_file("model.gguf", mparams);
llama_context *ctx   = llama_init_from_model(model, cparams);  // new API

const llama_vocab *vocab = llama_model_get_vocab(model);
llama_sampler     *smpl  = llama_sampler_chain_init(llama_sampler_chain_default_params());
llama_sampler_chain_add(smpl, llama_sampler_init_greedy());  // simplest: greedy

// 1) split the prompt into tokens
int n = llama_tokenize(vocab, prompt, /*...*/, tokens, /*...*/);

// 2) autoregressive decode loop
llama_batch batch = llama_batch_get_one(tokens, n);
while (generating) {
    llama_decode(ctx, batch);                 // forward: logits for the next token
    llama_token id = llama_sampler_sample(smpl, ctx, -1);   // sample
    if (llama_vocab_is_eog(vocab, id)) break; // end-of-generation
    batch = llama_batch_get_one(&id, 1);      // feed the new token back
}

llama_model_free(model);

One easily missed detail: each turn of the decode loop only feeds back the single new token rather than recomputing the whole history - thanks to the KV cache, which stores the keys/values already computed for every prior token. That is why the first token only appears after the whole prompt has been passed through once (prefill - the "time to first token"), while generating word by word afterward (decode) is fast, each step doing the forward pass for just one token. llama_decode is "run one forward step" and llama_sampler_sample is "pick a word by probability"; the two alternate to weave the token-by-token output you see.

Go deeper (optional)

Three common questions below - open them if you want to dig in; skip them if you just want the main line.