llama-serverllama-server

Last lesson's llama-cli wrapped the shared engine in a "command-line shell"; this lesson's llama-server fits the same engine with an "HTTP shell" - turning inference into a network service: many users and many requests connect at once, and it speaks the OpenAI API, so an existing SDK just needs a new address.

Since cli and server share one engine (revealed in L27), this lesson looks head-on at how that engine runs when it must serve many requests at once. Its most brilliant and most memorable move is continuous batching: one forward pass advances many requests together.

This lesson is an architecture overview only - walk the main road a request takes from door to door, and make the two core concepts, slots and continuous batching, clear. The deeper scheduling trade-offs (how prefill and decode interleave, batch capacity, preemption and fairness) are beyond this overview.

Overall architecture: a request's journey

First walk a request's main road from door to door. An incoming HTTP request is wrapped into a server_task (a to-do) and dropped into the server_queue (a task queue); the scheduler assigns it to an idle server_slot; then the update_slots continuous-batching loop keeps advancing it, producing a server_task_result per token (streamable in pieces); finally the HTTP layer assembles the response back to the client.

This modularity is why server reads well: server-http minds only the network, server-queue only the queue, server-context only inference, server-chat only OpenAI compatibility. Each to its own, untangled - want to see "how requests queue" open queue, want "how it generates" open context, no needle-in-a-haystack in one giant file. This split also echoes L27: what cli reuses is exactly that middle server_context engine. When you read server's source, this module map is your navigation: lost, come back and glance at it, first locate "is what I care about now the network, the queue, inference, or compatibility", then dive into the matching file, rather than gnawing through the whole server front to back in one go.

What a slot is

A slot is the first cornerstone for understanding server. At startup --parallel N opens N slots (called n_parallel in the source), and each slot is an independent parallel sequence: with its own seq_id, its own slice of KV (echoing L19's KV cache), and a small state machine. The total context is divided among slots, and each one's share is n_ctx_slot.

A slot's life is this circle: idle (IDLE) on standby; on a task it enters STARTED and begins eating the prompt (PROCESSING_PROMPT, that is prefill); once the prompt is eaten (DONE_PROMPT) it turns to GENERATING and emits tokens one by one; when generation finishes (an end token or the length cap) it returns to IDLE and waits for the next request. The N slots each turn this circle independently, without interfering - that is the basis for server "running many conversations at once". Each slot has its own KV, so one conversation's context never bleeds into another's. One more thing worth noting: because the slot count is fixed, server carves out these N KV regions once at startup and no longer keeps asking VRAM for memory and freeing it at runtime - fast and stable, but it also means once N is set, how many can run at once is set too, and anything beyond must queue.

Continuous batching (the core)

Now to server's most brilliant move. Picture 3 requests running at once: some slots are in prefill (eating the prompt), some in decode (emitting words). The clumsy way takes them one at a time, the GPU serving only one slot while the rest wait. Continuous batching does the opposite: it packs "the tokens to compute this step" from all currently active slots into the same llama_batch, and one llama_decode advances all their sequences by one step together.

// the heart of continuous batching (condensed from update_slots in server-context.cpp)
common_batch_clear(batch);
for (slot : slots) {                       // iterate all active slots
    if (slot.state == GENERATING || slot.state == PROCESSING_PROMPT)
        common_batch_add(batch, slot.token, slot.pos, { slot.id }); // tag with seq_id=slot.id
}
llama_decode(ctx, batch);                 // one forward, advance all active sequences
for (slot : slots)
    slot.next = common_sampler_sample(slot.smpl, ctx, slot.i_logits); // each reads its own row

The key is that line common_batch_add(batch, token, pos, { slot.id }): it tags each token with "which slot I belong to" as a seq_id. So one batch holds tokens from several slots, llama_decode uses the attention mask to let each sequence see only its own history, and afterward each slot reads its own row of logits to sample. The diagram below freezes "one step": how 3 slots' tokens squeeze into one batch, and how after one decode each gets its own next token. When you read it, look closely at the middle "merged batch" box: it is not split into three request-segments, but truly mixes the three slots' tokens cell by cell, telling them apart only by the seq tag on each cell - it is exactly this "mix into one pan, recognize by tag" that lets one forward feed in the work of three requests at once.

OpenAI compatibility

server has one more feature that makes it especially handy: it grows a set of endpoints identical to the OpenAI API, such as /v1/chat/completions. This means any client, SDK, or front-end written for OpenAI only needs to point its request URL (base URL) at your llama-server to talk to a local model, with almost no code change. The layer doing this "translation" is server-chat: it converts back and forth between OpenAI's JSON schema (messages, tools, and so on) and the engine's internal representation, tool calls included. This translation layer looks humble but is the key to server fitting into the existing ecosystem - your existing tool chains, monitoring dashboards, and client code can almost all be reused without changing a line, cutting the cost of "switching to a local model" to a minimum.

# start a service, then call it like OpenAI
llama-server -m model.gguf --port 8080   # opens N slots by default (--parallel)

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}]}'

Why not just run many processes?

By here a natural question may surface: to serve many requests at once, why not simply run many llama processes, one process per request - is that not simpler? The answer hides in VRAM. Model weights run from a few GB to tens of GB, and each independent process must load another copy of those weights into VRAM - run 8 processes and the same weights are paid for 8 times, which an ordinary GPU cannot bear.

server does the opposite: one set of weights, many sessions. The model loads once (echoing L25's read-only, shareable llama_model), all slots share these same weights, and each only takes one extra small slice of KV (L19). So VRAM cost goes from "weights x process count" to "weights x 1 + KV x slot count" - and one KV slice is far smaller than a full set of weights. That is why "one engine, many slots" fits far more concurrency on a single card than "many processes".

Besides, many processes each run their own forward pass and miss continuous batching's "one forward, serve many" throughput dividend. So the combo of "one engine + many slots + continuous batching" is not for elegance, but the optimum forced by two hard constraints, VRAM and throughput: save VRAM (shared weights) and high throughput (shared forward), two wins at once. Many processes do have merits - cleaner isolation (one crash spares the rest), simpler deployment; but under the most common goal of "serve as many requests as possible on one card", one engine with many slots almost always wins, which is why mainstream inference servers (not just llama.cpp) nearly all take this road.

Deep dive: queueing and throughput

Two final folds answering two questions that cut to the point: what happens when slots run out, and why continuous batching makes throughput high.