多模态Multimodal

Up to here, the LLM in your head still only reads text: a prompt is split into tokens, each looked up to an embedding, then run through dozens of transformer layers. Yet today's models routinely "talk about pictures" - you send an image and they describe it, answer questions, read tables. How do they fit "an image" into a model that only knows tokens? The answer is surprisingly plain: the LLM's input layer does not care whether the embedding vectors fed in came from text or from an image. Multimodality just turns "an image" into a run of embeddings too, splices them into the same sequence as the text tokens' embeddings, and sends the lot into llama_decode.

This lesson looks at how llama.cpp's mtmd (multimodal) subsystem does it. The core is one sentence: the model body needs not a single change; multimodality rides entirely on a "translation" pipeline bolted in front of it - a vision encoder (clip / ViT) compresses the image into visual features, and the projector (mmproj) projects those features into the LLM's embedding space, yielding N vectors that "look just like token embeddings", spliced in where the prompt's <image> marker sits. The LLM, handed this run of vectors, cannot tell which came from text and which from the image - it just computes on as usual. This gives pause at first: if the model clearly "understood" the image, how can it "not know it was an image"? But that is precisely the cleverest part of the design - "understanding the picture" is fully outsourced to the vision pipeline in front, and the LLM does only what it is best at: reasoning over a sequence of vectors.

Roadmap: first the whole mtmd pipeline (split into chunks -> encode -> get embeddings -> interleave with text and decode), with a trace of "one image entering the LLM"; then a dedicated look at why the projector bridge is indispensable; and finally two folds digging into clip's ViT internals and how image embeddings "take positions" in the sequence and enter the KV cache.

The whole pipeline: how mtmd feeds an image into the LLM

First walk the whole pipeline. The user gives an "interleaved image-text" input - some text carrying an <image> marker, plus the pixels of one (or a few) images. mtmd runs it in five steps: (1) mtmd_init_from_file loads the projector (that separate mmproj file); (2) mtmd_tokenize splits the input into a run of chunks - text passages are TEXT chunks, each image becomes an IMAGE chunk (audio is AUDIO); (3) each image chunk is run through mtmd_encode_chunk (internally clip + projector); (4) mtmd_get_output_embd pulls out the encoded visual embeddings; (5) the text chunks' tokens and the image chunks' embeddings are interleaved in original order and fed segment by segment into llama_decode. First, the splitting:

// split "text + <image> marker + image bitmap" into ordered chunks (simplified from tools/mtmd/mtmd.h)
mtmd_input_chunks * chunks = mtmd_input_chunks_init();
mtmd_tokenize(mtmd_ctx, chunks, &text, bitmaps, n_bitmaps);
// chunks now hold, in original order:
//   [TEXT "this image is"] [IMAGE one picture] [TEXT "what is in it?"]
// each chunk carries a type: TEXT / IMAGE / AUDIO

Note the <image> in the prompt (the source's default marker is actually <__media__>): it is just a placeholder telling mtmd "where in the text this image should be inserted". The source's comment gives a plain example: a line like "here is an image: <__media__> ..." splits into three chunks - the text before the marker, the image itself, the text after - in exact original order. Once chunks are split, the real show is encoding the image chunk into embeddings and decoding it interleaved with text. llama.cpp packs this logic into a helper whose comment is practically the whole pipeline's pseudo-code:

// per chunk: text decodes directly, image encodes first then decodes (simplified from mtmd-helper.cpp)
for (each chunk : chunks) {
    if (type(chunk) == MTMD_INPUT_CHUNK_TYPE_TEXT) {
        llama_decode(lctx, batch_of(chunk.tokens));   // text: old path, look up embeddings then compute
    } else {                                          // IMAGE / AUDIO chunk:
        mtmd_encode_chunk(mtmd_ctx, chunk);          // 1) internally runs clip(ViT) + projector
        float * embd = mtmd_get_output_embd(mtmd_ctx); // 2) pull out N visual embeddings
        llama_decode(lctx, batch_of_embd(embd));     // 3) feed the embeddings straight in
    }
}

See the trick? Text and image both land on the same llama_decode - the only difference is "whether you feed in tokens (and let the model look up embeddings) or already-computed embeddings (used directly)". llama_batch has long supported both inputs (remember from L18 that a batch can carry either token or embd?), so image embeddings slot seamlessly into the sequence, with not one line of the model body changed. The beauty of this design is reuse: mtmd does not open a second inference path for images, it "disguises" images as embeddings and reuses the entire text machinery of batching, KV cache, and sampling - so multimodality becomes a "preprocessing" problem, not a "rewrite the engine" problem. Freezing one image's whole journey into the LLM as a pipeline:

Worth noting: this "split chunks -> encode -> interleave" mechanism treats audio identically - cut speech into frames, run an audio encoder (like Whisper's front end), then a projector to embeddings, taking exactly the same path as images. That is why mtmd's chunk types put AUDIO right beside IMAGE. In other words, mtmd's design goal from the start was not "support images" but "support any modality that can be encoded into embeddings". Once you understand the image path, audio - and more modalities to come - are cast from the same mold.

The projector: translating "what is seen" into the LLM's mother tongue

That mtmd_encode_chunk step splits internally into two halves: first clip(ViT) "sees" the image as visual features, then the projector "translates" those features into embeddings the LLM can read. Why is this second step mandatory? Because clip's visual features and the LLM's token embeddings are simply not in the same space - the dimensions may differ (clip might output 1024-d, the LLM wants 4096-d), and the value distributions and semantics are two different systems entirely. Feeding clip's output straight into the LLM is like handing an untranslated foreign passage to an expert who only reads Chinese - blank stares. The projector (that separate mmproj file) is the bridge: a small network that projects visual features into the LLM's embedding dimension and space, making them "look and behave just like a token embedding". By analogy, clip's output is a kind of "visual shorthand" whose numbers mean things arranged for a vision task; the LLM's embedding space grew out of a language task, so vectors of the same length live in completely different "coordinate systems". The projector does exactly that change of coordinates: re-expressing the visual shorthand into the language coordinate system, so "there is a cat in the image" lands near the patch of vector space the LLM has always used for "cat".

Two key "alignments" are guarded by two functions: clip_n_mmproj_embd(ctx) returns the projector's output dimension - it must equal the LLM's embedding dimension, or the vectors will not fit into the sequence; clip_n_output_tokens(ctx, img) returns how many embedding tokens this one image occupies (the N in the earlier trace). That N is not arbitrary: bigger image, more patches, larger N; some projectors (resampler types) even actively compress N, aggregating hundreds of patch features into a few dozen embeddings, saving KV cache and compute. This N directly sets an image's "cost": N embeddings take N sequence positions and write N entries of KV - so for the same image, whether the projector squeezes it to 64 embeddings or lays out 576 changes VRAM and speed by an order of magnitude. This is one of the core tradeoffs of high-resolution multimodal models: the finer it sees (more patches, larger N) the more accurate, but the longer the sequence, the slower and more VRAM-hungry.

// inside the projector: clip encodes first, output dim set by mmproj (simplified from tools/mtmd/clip.h)
int n_embd   = clip_n_mmproj_embd(clip_ctx);       // projector output dim == LLM embedding dim
int n_tokens = clip_n_output_tokens(clip_ctx, img); // how many embedding tokens this image takes
std::vector<float> out_vec;
clip_image_encode(clip_ctx, n_threads, img, out_vec); // run ViT + projector, output n_tokens x n_embd
// out_vec is now n_tokens visual embeddings, each n_embd-dim,
// exactly the shape of n_tokens token embeddings -> straight into the sequence

Common projectors come in three tiers of complexity: a linear layer (one matmul, simplest, early LLaVA), a two-layer MLP (one more nonlinearity, better alignment, common today), and a resampler / cross-attention (a set of learned queries "resamples" the variable-length patch features into a fixed number of embeddings, compressing N and handling arbitrary resolution, used by Qwen-VL etc). Which tier is a tradeoff of accuracy and cost: linear is cheapest but least expressive, the resampler is most flexible but carries its own pile of parameters and compute. Whichever tier, its job is the same: align visual features into the LLM's embedding space. This is also why the mmproj is a separate file, loaded separately - it is a bridge specifically trained for one "this vision encoder + this LLM" pairing; swap the LLM or the clip and the bridge must be retrained. Grasp this and you see why, when downloading a multimodal model, besides the big main GGUF you also need a tiny mmproj file: without that bridge, the model keeps only its "read text" skill, and "see images" is off the table. A practical tip: in a multimodal model's GGUF repo on HuggingFace, the small file with mmproj in its name is exactly this, usually on the order of a few hundred MB - do not forget to grab it.

Deeper: clip's eyes, and an image's "position"

Two folds for two details you cannot avoid when really deploying multimodality.