对话模板Chat templates

M4a let the model compute, L20 cut text into tokens, L21 taught it to pick the next word. But one key question remains: you speak sentence by sentence in a chat box, so how does the model know "which line is yours, which is its own, where one turn starts and ends"? This lesson covers chat templates - assembling a list of role-tagged messages (system/user/assistant), in the format this model recognizes, into a prompt string with special markers.

This step looks minor but is crucial: every model was trained with conversations fed in some fixed format (ChatML, Llama-2 differ). At inference you must use the same format for the model to recognize "turns" and "roles". Get the format wrong and the model is, at best, off-topic; at worst, completely out of character. The chat template is what packs messages correctly into "this model's envelope".

Why a template is needed

First see where this step sits in the whole pipeline. Your conversation is a list of structured messages: each has a role (system/user/assistant) and some content. But the model doesn't eat this structure - it eats a long string of tokens (L20). There must be a step that flattens "the message list" into "one string", and that step is the chat template.

The assembled string, besides each message's body, inserts a pile of special markers: marking where each message starts, ends, and who spoke. These markers correspond to special tokens in the vocab (L20), and the model relies on them to recognize "it is the assistant's turn now", "the user is done with this turn".

The order is: message list -> apply the template into a string -> hand to the vocab to tokenize -> into the model. The template handles "structure to text", tokenize handles "text to tokens" - two relays, neither dispensable. This is also why this lesson follows the vocab (L20) closely - its product is exactly tokenize's input.

Conversely, what if you skip the template and tokenize the user's words directly? The model would think it is the continuation of some ordinary text, not "a turn of dialogue asking for a reply". It might keep writing on the user's behalf rather than answer as the assistant - because without the markers delimiting role and turn, it has no idea "it is its turn to speak". The presence or absence of a template directly decides whether the model "completes" or "converses".

Precisely because conversation is encoded as plain text, many interesting things become possible: you can write a "system prompt" into the system message to give the model a persona; you can splice prior turns verbatim so it "remembers" context (really, the history is re-fed each time); you can even insert a fabricated assistant line to steer where it continues. These flexible tricks all rest on the fact that "a conversation is just carefully formatted text".

The built-in template table

// simplified from src/llama-chat.h
enum llm_chat_template {
    LLM_CHAT_TEMPLATE_CHATML,
    LLM_CHAT_TEMPLATE_LLAMA_2,
    LLM_CHAT_TEMPLATE_LLAMA_3,
    LLM_CHAT_TEMPLATE_GEMMA,
    /* ... fifty-odd of them ... */
    LLM_CHAT_TEMPLATE_UNKNOWN,
};

llama.cpp ships a big batch of common models' templates, all listed in the enum llm_chat_template - CHATML, LLAMA_2 (plus several variants), LLAMA_3, GEMMA, MISTRAL, PHI, and so on, fifty-odd in total. Each corresponds to a concrete "markers + assembly rule".

Why hardcode so many? Because each model family's chat format is fixed at its training time, and they vary widely. Building the common ones in means that, given a mainstream model, the engine can mostly auto-detect which format to use, working out of the box without manual specification.

A few representatives make it clear: ChatML (used by many models) wraps messages with <|im_start|>/<|im_end|>; Llama-2 frames user instructions with [INST]/[/INST]; Gemma uses <start_of_turn>; Llama-3 marks roles with <|start_header_id|>. Different markers, same intent: delimit role and turn boundaries.

This "gather every model's format into one enum table" should look familiar - the same idea as L15 gathering architectures into the LLM_ARCH table and L20 gathering tokenizer types into vocab_type: concentrate "the differences that vary" into data, and let generic code act by the table.

You might ask: does the model not know which format to use, needing the engine to guess? Not necessarily. The GGUF file may carry a template field (many new models write one), but plenty of models omit it or write it loosely. So llama.cpp both supports reading the model's own template and builds in these dozens of common formats as a backstop - a two-pronged setup to spare the user manual fuss.

By the way, the enum's LLM_CHAT_TEMPLATE_UNKNOWN sentinel has a purpose too: when detection matches neither a name nor a feature, it lands here. The engine then signals "template not recognized", prompting the user to specify one manually rather than silently using a wrong format. Leaving a clear exit for "cannot recognize" is a common robust-design technique.

An often-overlooked detail is how the system message is handled. Templates diverge most on where the "system prompt" goes and how it is marked: some, like ChatML, list a separate system message; some (like certain Llama-2 variants) fold it into the first user message; some support no standalone system at all. So the same system prompt can land in very different places under different templates - which is also why, switching models, changing the prompt text alone is not enough; the template must place it correctly for you.

Detection and application

# pseudocode: apply a chat template
tmpl = llm_chat_detect_template(template_str)   # by name first, then guess by content
dest = ""
llm_chat_apply_template(tmpl, messages, dest, add_ass=True)
# add_ass: append the assistant start marker so the model writes the reply

With this table, two things remain: recognize which template to use, and apply it to assemble the messages.

Recognizing uses llm_chat_detect_template: it first matches by name exactly (the model's GGUF often carries a template name/string), and failing that, looks for feature substrings like <|im_start|> or [INST] in the template body and guesses by feature. Applying uses llm_chat_apply_template: give it the template enum, the message list, and an output string, and it assembles per that format.

The message's own structure is simple: llama_chat_message has just two fields, role (a role string, e.g. "user") and content (the content). A list of such messages is the input to apply_template; internally, per the chosen template, it wraps each message in its markers, concatenates head to tail, and emits that final prompt. Walk a concrete message set through it to see:

Detection actually hides subtlety. Ideally the GGUF states the template name and a lookup settles it; but in reality it often gives only a template body (Jinja text) with no name. Then one can only infer from "whether certain feature markers appear in the body" - see [INST] and guess Llama-2, see <|im_start|> and guess ChatML. This feature-based heuristic is not 100% reliable but covers the vast majority.

Application is also fussier than it looks. For the same format, the system message may go at the very front, be merged into the first user message, or not be supported at all; in multi-turn dialogue, whether history repeats the markers and how the last turn closes - each template has its own rules. apply_template handles these details per template type, so you just hand in a message list and it returns a precisely formatted prompt.

Two paths: built-in vs Jinja

The built-in templates cover only "the known models". What about a brand-new model bringing its own unique template? That leads to the second path: Jinja.

The built-in path (src/llama-chat.cpp) is pure C++ string assembly - fixed enum, zero deps, fast, but recognizes only the predefined few dozen. The C API llama_chat_apply_template takes this path.

The Jinja path (vendored in common/jinja/) can render any template: the chat_template a model carries in GGUF (often a piece of Jinja text) runs as-is, most faithful. The upper layer common/chat.cpp's common_chat_templates wraps it and also supports tool calls, output parsing, and other advanced tricks.

Why not just use Jinja for everything and skip maintaining dozens of built-in templates? Because Jinja is a full template language, requiring an interpreter and executing arbitrary logic, with non-trivial cost and complexity. For common models whose format is long fixed, a few lines of C++ assembling directly is faster, steadier, and free of the security concerns of running arbitrary templates. So the built-in "fast path" has irreplaceable value.

Conversely, why is Jinja indispensable? Because models keep appearing, and there are always new formats absent from the built-in table, or templates with complex conditional logic ("assemble this way if there is a system message, that way if not", "add a section when tool definitions are present"). A fixed enum simply cannot express these; only a real template engine can execute them. Each path covers the other's weakness; together they are both broad and fast.

Also worth mentioning is the advanced use of tool calling (function calling). When you want the model to call an external tool, the tool definitions, the call format, and the splicing of returns must all be packed into the prompt per a specific convention - far beyond the built-in template's "stitch a few messages", and exactly what the common layer does by combining Jinja with grammar constraints (L23). So chat templates are not only "chat"; they are the foundation of more complex "structured interaction" too.