llama-clillama-cli

llama-cli is most likely the first llama.cpp program you ever run - one line, llama-cli -m model.gguf -p "Once upon a time", and the model starts continuing the text. It is so simple it barely needs explaining, but that very simplicity makes it the best sample for seeing "how one command turns into a screen full of text".

This lesson digs inside it and takes apart the skeleton of "parse the command line + one generation loop"; it also uses cli to see how llama.cpp's tools actually sit on top of common (L26) and the inference engine - the "way to read" the tool lessons is set here.

One more spoiler up front: a modern llama-cli is no longer a standalone bare llama_decode main loop - it directly reuses llama-server's engine. So this lesson is both "your first hands-on lesson" and a quiet setup for the server lesson next.

A command's journey

The entry point is tools/cli/main.cpp, but it is so thin it is almost one line - it immediately hands off to tools/cli/cli.cpp. cli.cpp first calls common_init() to start the logging system, then uses common_params_parse(argc, argv, params, LLAMA_EXAMPLE_CLI) to pour the command line into common_params (exactly L26's mechanism). Note the 4th argument LLAMA_EXAMPLE_CLI: it tells the parser "recognize parameters by cli's option set", so cli-specific flags and shared flags are both parsed correctly.

// entry: the thin main hands off to cli.cpp (simplified from tools/cli)
common_init();                                  // global init: start logging
if (!common_params_parse(argc, argv, params, /*ex=*/ LLAMA_EXAMPLE_CLI))
     return 1;                                   // argv -> common_params
auto res = common_init_from_params(params);     // conceptually model+ctx+sampler (cli does it via the engine)

With the parameters in place, the rest lines up almost straight: load the model and build the context from common_params, encode the prompt into tokens and feed it in, enter the generation loop, and stream out new words as you sample. The diagram below draws this main line as five steps - the first two should look familiar, they are exactly L25's hand-written sequence, now folded into one line of common_init_from_params.

On this main line, the first two steps (parse, init) are almost entirely packaged by common (L26); the part that truly belongs to cli, its "main act", is step 4's looping generation loop - it threads L19's KV growth, L21's sampling, and L20's detokenize into a closed loop whose output you can watch live. Put plainly, all the internal machinery the first four parts covered finally converges here into a loop you can watch emit text line by line - many people first "fall for" reading llama.cpp's source from cli, because it turns abstract inference into words actually dancing on the screen. Let us pull that loop out and look at it alone.

The generation main loop

Generation is in essence a loop: each round runs one forward pass through the engine (llama_decode) to get "the next word's scores" (logits), uses the sampler (common_sampler, L26/L21) to pick a token, restores it to text with common_token_to_piece and prints it at once, then appends this new token back to the context and goes round again. So it repeats, until it hits one of three "stop" conditions.

// the essence of the generation loop (now inside the engine, server_context)
while (n_remain != 0) {
     llama_decode(ctx, batch);                        // forward: logits at last pos
     llama_token id = common_sampler_sample(smpl, ctx, -1); // sample the next one
     common_sampler_accept(smpl, id, true);            // feedback: penalties/grammar
     if (llama_vocab_is_eog(vocab, id)) break;         // end-of-gen -> stop
     fputs(common_token_to_piece(ctx, id).c_str(), stdout); // stream print
     batch = llama_batch_get_one(&id, 1);              // append new token
     n_remain--;                                         // n_predict countdown
}

Keep the three "stop" reasons in mind: the set length is reached (n_predict counts down to 0), the model itself emits an end-of-generation token (EOG, echoing L20/L21, tested with llama_vocab_is_eog), or in interactive mode it hits the reverse prompt (antiprompt) you set. That common_sampler_accept line cannot be skipped either - it feeds the just-chosen token back to the sampler, updating repetition penalties and advancing grammar state (L23), so the next round samples correctly. This is the easiest thing to overlook as a beginner: many think "sampling" is just picking a word, forgetting the sampler is actually stateful - it must remember which words already appeared to apply repetition penalties, and where the grammar has advanced to constrain the next legal token. Skip accept and that memory freezes in place, so generation soon repeats, drifts, or even stalls.

Walking one turn with a minimal example is the most vivid: suppose the context already holds "The cat", and watch how it generates the next word and streams it to the screen.

Running on the shared engine

Now lift the spoiler from the start. If you open the top of tools/cli/cli.cpp, you will see it directly #includes several of server's headers - server-common.h, server-context.h, server-task.h; and tools/cli/CMakeLists.txt links it against the server-context library. In other words, a modern llama-cli does not write its own bare llama_decode main loop anymore, it reuses server's engine server_context (the slot-and-task machinery, detailed next lesson).

// top of tools/cli/cli.cpp: reuse server's engine directly
#include "server-common.h"
#include "server-context.h"   // server_context: slot / task / KV
#include "server-task.h"

# tools/cli/CMakeLists.txt: link the server-context library
target_link_libraries(${TARGET} PUBLIC server-context llama-common ...)

What does this mean? cli and server actually share one "engine", just wrapped in different "shells": cli's shell is the command line plus an interactive terminal, server's shell is HTTP plus many requests. The engine is identical (loading, KV, batching, sampling all go through the same server_context); the only difference is "how requests are fed in and how results are handed out".

Historically cli was a standalone main.cpp generation loop, and only later was unified onto this shared engine - the gain is one less near-duplicate copy to maintain, and a bug fixed in the engine benefits cli and server alike. For the same reason, many of next lesson's server concepts (slots, continuous batching) you are in fact already using in cli "through the shell", just without noticing.

Interactive mode

Add -i to the command line and cli turns from "one-shot continuation" into "back-and-forth conversation": after you press enter, it encodes your input into the context and keeps generating, taking turns. Interrupting generation is done with the reverse prompt (antiprompt) - you set a string (say "User:"), and the moment the model is about to generate up to it, it stops and hands the microphone back to you. The terminal's coloring, backspace, and UTF-8 input are held up behind the scenes by L26's console.

A few of the flags you deal with most are worth noting against the main loop: -n caps how many tokens to generate (that is n_predict, governing how many times the loop turns), -c sets the context window size (n_ctx, echoing L17/L19), --temp tunes the sampling temperature (L21, governing the pick-a-word step), and -i enters interactive. Match each flag to a step in the loop and you can predict exactly what it changes.

Deep dive: the story of reuse and "when to stop"

Finally two folds for two often-asked points: the backstory of cli reusing server's engine, and "when exactly generation counts as finished". The first is about "why the architecture evolved this way", the second about a detail you meet every run yet may not be able to explain. These two questions look trivial, yet each answers one of the two confusions that most often surface while reading source: one is "why is this code shaped this way" (history and trade-offs), the other "when exactly does it stop" (runtime behavior). Make them clear and you will not get lost when you open cli's real source.