上下文与会话Context & session

A graph-able model (L14-16) still needs a runtime to actually run, remember conversation progress, and hand results out - that is llama_context. It is a stateful object holding this session's config (cparams), the KV cache (memory), the backend scheduler (sched), and output buffers; llama_decode runs one forward step, llama_get_logits_ith reads the result.

This lesson is the key link that activates "a static model" into "running inference". It also answers a very practical question: why does llama.cpp split "model" and "context" into two objects? Think this through and you understand the underlying reason llama-server can serve many users at once.

model vs context: read-only knowledge vs stateful session

This is the cut to grasp first: weights are read-only knowledge, a session is stateful progress, and the two are deliberately split into two objects.

Why split this way? Because weights are several GB, loaded once and unchanging, and ought to be shared; while "conversation progress" (KV cache, current position) is per-session state that must be stored separately. Splitting unchanging knowledge from changing state, one copy of weights can hold up many concurrent sessions - the basis of multi-user serving.

An analogy: the model is like a dictionary (everyone consults it, its content fixed), the context like each person's scratch paper (each writes their own, none interfering). You would never print a separate dictionary per reader, but everyone needs their own scratch paper. llama.cpp's cut is exactly between "shared knowledge" and "private state".

"One model with several contexts" is no slogan but a daily reality. When llama-server serves many users at once, it is one copy of weights + one context per request; even within a single program, running several different conversations in parallel means opening several contexts. They share the read-only weights, each maintaining its own KV and progress, none interfering. Understand this and you see how "load once, serve many" is done.

This "knowledge vs state" split is actually a very general design principle in computing: separate the stateless, shareable parts from the stateful, must-isolate parts. Web servers separate static assets from session state, databases separate read-only snapshots from transaction state - the same idea. llama.cpp applies it to inference: weights are a stateless "program", the context a stateful "process".

The word "session" here is worth pinning down: it is one "continuous conversation or generation process". Within one session, later words remember what was said before (via the KV cache); a different session is a blank sheet starting over. So a context essentially carries "all the memory of one coherent conversation" - while it lives, the conversation's context lives; once freed, this conversation is forgotten clean.

By the way, "context" here has two easily-confused senses: one is the llama_context object, the other is n_ctx, the "how many tokens it can remember" context length. The former is the container holding session state, the latter how long a conversation that container can hold. This lesson is mainly about the former; the latter's details wait for L19.

What is inside a context

Open up llama_context and it mainly holds four things: config, memory, scheduler, output.

// simplified from src/llama-context.h
struct llama_context {
    llama_cparams          cparams;  // this session's config
    llama_memory_ptr       memory;   // KV cache etc.(L19)
    ggml_backend_sched_ptr sched;    // multi-backend scheduling(L10)
    buffer_view<float>     logits;   // output: next token's scores
};

These four together are exactly the state needed to "run one inference step": cparams says "at what spec to run", memory remembers "what was computed before", sched manages "on which hardware to compute", logits receives "the computed result". Note memory is a generalized "memory" abstraction (not only a raw KV cache, but also variants like recurrent, hybrid, L19), so the field is named memory rather than kv_cache - room left for long context and new architectures.

One more point: the context does not hold the weights - those live in model, and the context only references it. This is exactly the previous section's "split" in action: a lightweight context carries session state and points at the heavy, read-only weights, so opening many contexts costs almost no extra weight memory.

Why is sched (the backend scheduler) also placed in the context, not the model? Because scheduling carries session state: it manages how this session's intermediate-tensor memory is allocated and reused (L10's ggml-alloc), and which graph parts compute on which device. Different sessions run their own graphs, so each naturally needs a scheduler. Placing it in the context aligns exactly with the "one bit of state per session" design.

A word on a context's life and death: it is created by llama_init_from_model(model, cparams) (opening the KV cache etc. per cparams then), and freed by llama_free when done. The model is long-lived (the whole service duration), the context comes and goes (one per request). This "long-lived model + short-lived context" lifecycle pairing is exactly how a server handles wave after wave of requests.

An implementation detail: the llama_context_params you pass in is copied into the context (internally llama_cparams). So once a context is built, its session spec is fixed, not changing because you later edit that outer params struct. Each context remembers its own spec, none interfering - another showing of "per-session independence".

The embd buffer alongside logits deserves a mention: it holds embedding vector output - used for embedding tasks (turning a whole sentence into one vector) instead of logits. So a context actually has two output ports: for "the next word" read logits, for "the sentence's vector representation" read embd. The same decode mechanism, different outputs by task.

cparams: configuring this session

When creating a context, you tell it how to run via llama_context_params (cparams). Most of these parameters trade off VRAM against speed.

// simplified from llama_context_params in include/llama.h
struct llama_context_params {
    uint32_t n_ctx;      uint32_t n_batch;   uint32_t n_ubatch;
    uint32_t n_seq_max;  int32_t  n_threads;
    ggml_type type_k, type_v;   // KV quant type(save VRAM)
    bool offload_kqv;           // KV compute on GPU?
};

The two to build intuition for are n_ctx and type_k/type_v: they directly decide how much VRAM the KV cache eats. Double n_ctx and the KV cache roughly doubles; drop type_k/type_v from 16-bit to 8-bit and KV usage halves again (at a little precision cost). So "how long a context you can open" is not decided by the model alone but tuned by you, given your VRAM, between n_ctx and KV quantization - a thread we pick up again in L19's KV cache.

n_seq_max relates to an often-overlooked capability: one context can run multiple sequences at once. For example, generating answers for several different prompts in a batch can go into the same context, distinguished by different seq_ids (L18), sharing these weights and this scheduling. n_seq_max is the upper bound. This makes "one context serving several concurrent conversations" possible, a more frugal approach than "one conversation per context".

cparams the table omits have specialized uses too, such as pooling_type (how to pool token vectors into one sentence vector for embedding tasks) and various RoPE scaling parameters (extrapolating context beyond the trained length). Plain text generation mostly uses defaults, but their existence shows the context serves not only "generate the next word" but also embedding, long-context extrapolation, and more.

A practical reminder: a context's memory is mostly the KV cache, whose size is set jointly by n_ctx, layer count, KV head count, and type_k/type_v. So when you hit a VRAM wall, shrinking n_ctx or quantizing KV is often more effective than swapping models. We will do this account in more detail in L19.

Why are these parameters configured at context creation, not at model load? Because they are about "how this session runs", not "what the model is". With one model, you can open several contexts with different cparams: one with long context, one short, one many-threaded, one few-threaded, each to its scenario. Decoupling session parameters from the model is exactly for this flexibility.

One inference step and reading the result

With everything ready, running inference is repeatedly calling llama_decode. It eats a batch (L18), internally does graph-building, execution, and KV update in one go, then puts the logits into the context's output buffer.

# pseudocode: one inference step (inside llama_decode)
llama_decode(ctx, batch)              # run one forward step
#   -> split ubatch(L18) -> build_graph(L16) -> sched execute(L10) -> update KV(L19)
#   -> logits written into ctx's output buffer
p = llama_get_logits_ith(ctx, i)    # get the i-th token's logits(n_vocab-dim)

Read the result with llama_get_logits_ith(ctx, i), getting the i-th token's logits - an n_vocab-dimensional vector, the unnormalized scores for "the next word", handed next to sampling (L21) to pick one. This strings L16's graph-building, L10's execution, and L19's KV, through this one llama_decode function, into one complete inference step.

This is also where L03's prefill/decode connects: both phases call llama_decode, differing only in the batch fed. Prefill feeds a whole prompt at once (many tokens), filling their K/V into the KV cache in one go, taking only the last one's logits; decode then feeds one new token each time and takes its logits. One function, two rhythms, all expressed via the batch.

Drawing the whole autoregressive loop: llama_decode computes logits -> sampling (L21) picks a token -> that new token is wrapped into a new batch fed back to llama_decode -> logits again... and so on, popping out the answer word by word. The context is present throughout this loop: its KV cache grows step by step, its output buffer refreshes step by step. Generating one reply is this loop turning many times on one context.

So what this lesson delivers is a leap from "parts" to "machine": earlier lessons built parts like the model (L14-15) and graph (L16); this lesson uses the context to pack them into a loop that can turn repeatedly. By here you can run a full inference in your head: load model -> build context -> repeatedly decode + sample -> output word by word. The remaining lessons take this machine's key components (batch, KV cache, sampling...) apart one by one for the details.

Emphasize once more that "state advances", because it is the key to understanding autoregression. Each llama_decode is not a recompute from scratch but advancing one step on the context's existing state: the KV cache already holds all prior tokens' K/V, and this step only computes the new token and appends its K/V. Precisely because the context keeps the state throughout, decode can be so fast as "compute only one token per step" (L03/L19).

To close on the context's role in one sentence: it is the switch that turns "a static model" into "dynamic inference". Without it, the model is just a pile of resting weights; with it, the weights are driven step by step, popping out token after token. From the next lesson, we dig into this machine's concrete components, starting with the batch fed to decode.