词表Vocabulary

By now M4a's model can "compute" - loading (L14), architecture (L15), graph-building (L16), context (L17), batching (L18), efficient autoregression via the KV cache (L19), finally emitting a row of logits. But one basic question kept getting skipped: from start to finish the model only knows numbers (token ids); it has no idea what the characters "hi" are. How does text turn into numbers, and numbers back into text? This lesson's star, llama_vocab (the vocabulary), is the sole translator between the text world and the token world.

It does two inverse jobs: tokenize cuts a string into a list of token ids to feed the model; detokenize/token_to_piece turns the token ids the model emits back into text pieces, reassembled into a human-readable sentence. A full chat generation passes through it on the way in (your words become ids) and on the way out (the chosen ids become characters). Without this translation, "the model only does numbers" and "humans only speak text" never connect.

Why a vocabulary is needed

Inside, the model is all numbers. Its input is a list of token ids and its output is a probability distribution over a token id; from end to end it never touches, and never needs, strings. The vocabulary sits exactly on the border between the text world and the token world, and both directions must pass through it - that is why it must exist.

Why not feed the model by "character" or "letter" directly? Too fine and the sequence is too long (one CJK glyph is several bytes, one sentence thousands of steps); too coarse and the vocab explodes (enumerating all words is hopeless). Subword splitting is the compromise: common words get one id whole, rare words split into a few common pieces. So the vocab size stays manageable (tens of thousands) and the sequence length stays reasonable.

This also explains why "the same sentence, split by different models, yields a different token count". The splitting rules are fixed at training time and stored with the weights in GGUF (L13's self-description); at use time you must use the same vocab - the wrong one and the ids no longer match what the model learned, and the output instantly turns to garbage.

From another angle, the vocabulary is really an agreed-upon interface between the model and people. Every piece the model saw repeatedly during training maps to one id in the vocab; all the patterns it learned are built on "those pieces". So once the vocab is fixed, it delimits the model's "known world" - what it handles fluently is always combinations of pieces this vocab can produce.

For that reason the vocab's quality directly affects performance. A well-cut vocab lets common expressions take very few tokens, letting the model spend attention on "meaning" rather than "spelling"; a crude one shatters simple words into fragments, wasting sequence length and raising the learning difficulty. Tokenization is, in a sense, the shared starting line of both training and inference.

Build intuition with a number: an English word averages about 1.3 tokens, a CJK glyph often takes 1 to 2 tokens, and a prompt of a few hundred characters often maps to over a thousand tokens. The model's "context length" (L17's n_ctx) counts tokens, not characters; how much conversation you can fit is ultimately decided by the post-tokenization token count. Grasp this and you see why "the same few screens of text sometimes overflows the context".

What llama_vocab is

// simplified from src/llama-vocab.h
struct llama_vocab {
    uint32_t n_tokens() const;                          // vocab size (source of L15's n_vocab)
    int32_t  tokenize(const char * text, ...) const;       // text -> token id
    int32_t  token_to_piece(llama_token id, char * buf, ...) const; // id -> text piece
private:
    struct impl;                              // pimpl: hides the tokenizer internals
    std::unique_ptr<impl> pimpl;
};

Outwardly llama_vocab is one unified interface: n_tokens() gives the vocab size, tokenize encodes, token_to_piece decodes, plus a batch of methods to query a single token's attributes. But it hides all the "which tokenizer algorithm exactly" implementation detail inside a private impl struct - this is the pimpl (pointer to implementation) idiom.

Why pimpl? Because tokenizer algorithms vary wildly (next section), each with different internal data structures and merge rules. Stuffing them all into impl and exposing only the thin tokenize/token_to_piece interface means the caller feels nothing: whether SPM or BPE underneath, the call is identical. Swapping algorithms or changing internals never disturbs the upper code.

This is the same flavor of decoupling you have seen before: L14's loader splits "parse the format" from "use the model", L17's context splits "read-only knowledge" from "session state". Here it splits "the dirty work of tokenizing" from "the unified interface". A clear boundary keeps the complexity locked in a box.

Look more concretely at this interface's division of labor. On the encoding side, tokenize handles quite a few details: whether to add a space prefix, whether to normalize, how to merge consecutive whitespace - each tokenizer's own "temperament", all gathered into impl. The upper layer just hands in a string and takes out ids, never minding what churns below.

The decoding side is equally subtle. token_to_piece is not a plain "look up a string"; it also handles whether a special token should show, how byte tokens join by UTF-8, whether the first word needs a leading space. Sealing these into the vocab too keeps "restoring text" consistently correct across any model - you just call in a loop and concatenate.

A few tokenizer algorithms

There are only a handful of mainstream tokenizer algorithms, and enum llama_vocab_type lists them out. They differ in "how to split a word into subwords and how to merge", but to the upper layer they are all the same tokenize. GGUF's tokenizer metadata (L13) decides which one this model uses.

SPM (SentencePiece) is the LLaMA-family tradition, based on byte-level BPE with built-in byte fallback; BPE (byte-pair encoding) is the GPT-2-family byte-level merging; WPM (WordPiece) is the BERT family; UGM (Unigram) is the T5 family; RWKV uses greedy matching; and the newer PLAMO2 (Aho-Corasick + dynamic programming). Behind these abbreviations lie different historical ecosystems and language/efficiency trade-offs.

Why so many? Because different model families inherit their own ecosystem's toolchain, and each algorithm has strengths and weaknesses across multilingual coverage, code, and compression rate. llama.cpp does not force uniformity; it makes them all compatible behind one interface (pimpl) - one of the engineering foundations for running dozens of models.

A concrete example brings out the difference. The word "unhappiness", BPE might cut into "un"+"happiness" or "un"+"happy"+"ness", relying on high-frequency merges counted at training time; Unigram instead picks, from a large candidate set, the most probable single split by probability. The two routes converge - both want to cover the most text with the fewest pieces - they just differ in the "philosophy" of choosing pieces.

For a language like Chinese with no natural spaces, tokenization shows its craft. Byte-level schemes first drop a glyph to UTF-8 bytes and then merge, so they work without relying on "word boundaries" - which is also why a model trained mostly on English can often stumble through Chinese: at the bottom it knows bytes, not any language's "words".

Another frequently asked point: how big should the vocab be? Too small and every word splits into many pieces, lengthening sequences and slowing inference; too large and the embedding table and output layer bloat with it, straining VRAM. So vocab size is a compromise, and mainstream models mostly land between tens of thousands and a hundred-odd thousand. Once fixed, it deeply shapes the model's size and speed - once more proof that "the vocab and the model are bound together".

Special tokens and byte fallback

Beyond ordinary "text-piece tokens", the vocab has a class of special tokens: BOS (begin sequence), EOS (end sequence), EOT (end of turn), PAD/SEP/UNK, etc. They map to no specific text but are control markers, fetched by accessors token_bos()/token_eos()/token_eot(). For example, when the model emits EOS it means "I am done", and the upper layer stops accordingly.

So what about a character the vocab simply does not have (rare glyphs, emoji)? Byte fallback: split the character into its UTF-8 bytes and map each byte to a byte token shaped like <0xF0> (carrying the LLAMA_TOKEN_ATTR_BYTE attribute). So any UTF-8 text can, worst case, be encoded byte by byte, and is never "unencodable". This solves the classic OOV (out-of-vocabulary) problem. A round-trip looks roughly like:

# pseudocode: a tokenize round-trip
ids = vocab.tokenize("Hello", add_special=True)   # may auto-prepend BOS
# ids = [<bos>, 9906, ...]
text = ""
for id in ids:
    text += vocab.token_to_piece(id)               # rebuild piece by piece

Walk that "Hello" through it for real and encoding gets concrete:

tokenize can auto-prepend BOS (controlled by that get_add_bos() flag); decoding then calls token_to_piece per token to stitch pieces back. Note that restoring byte tokens means joining several bytes by UTF-8 to form one complete character - which is why decoding accumulates step by step, not "one token, one character".

Special tokens matter because they carry structural information "beyond the text". In a conversation, who spoke, where a turn ends, whether to stop - all are delimited by these markers (L22's chat template, a later lesson, uses them heavily). Think of ordinary tokens as "content" and special tokens as "punctuation and paragraph marks" - without the latter, the model cannot tell the conversation's skeleton.

One more detail to call out: deciding "whether to stop" relies not on a single EOS but on a set of "end-of-generation" (EOG) markers. Different models use different end markers - some EOS, some EOT, some either; the vocab handles them uniformly with a dedicated test (whether a token belongs to the EOG set). The upper layer need only ask "is this token a terminator", finishing correctly without memorizing each model's specific convention.

The C API and the hand-off

Using the vocab from the C API is straightforward: first llama_model_get_vocab(model) to get the vocab from the model, then llama_vocab_n_tokens(vocab) for the size, llama_tokenize to encode, llama_token_to_piece/llama_detokenize to decode. These all take a const llama_vocab *.

Why such a sweeping rename? Because these operations are essentially vocabulary methods, not model ones - folding them from llama_* into llama_vocab_* makes name match substance, echoing how llama_vocab has internally become its own thing. Renames are annoying but make the API clearer.

Connecting this lesson back to the main line: your input text is first formatted by L22's chat template, then tokenized into ids by the vocab, enters the model to compute logits (L17), the sampler (L21) picks the next id in this vocabulary's token space, and finally token_to_piece turns it back into text shown to you. The vocab is exactly the two gates, in and out, of this loop.

A common doubt worth clearing: is tokenize's result unique? For a fixed vocab and the same rules, yes - the same input always cuts into the same id sequence, which is exactly the premise that encoding/decoding round-trips reliably. The randomness sampling (L21) brings happens at the "pick the next token" step, unrelated to tokenization; tokenization itself is fully deterministic.

One last whole-picture view: the vocab is one of the few parts of this inference system "a human can read directly". Weights are a pile of floats, the compute graph a chain of operators; only the vocab lets you look ids back into text and see with your own eyes "what the model read, what it wants to say". When debugging model behavior, printing the tokens first is often the fastest way in.