LoRA 适配器LoRA adapters

All through Part 4, you can now load, infer, tokenize, sample, apply chat formats, and constrain output (L14-L23). The last piece of the puzzle: what if you want the model to learn a new style or task? Full fine-tuning means retraining and storing a whole set of tens-of-GB weights - expensive and clumsy. This lesson covers LoRA - a far lighter approach that patches the weights with two small matrices, changing model behavior in mere megabytes.

LoRA's essence is low rank: it leaves the original weights untouched and learns only a low-rank delta (the product of two small matrices A and B), added to the original weights' output. Because the rank is low, these two matrices are tiny (an adapter is often just a few MB), yet they approximate full fine-tuning's effect. Better still, it is plug-and-play - attach the adapter for a style you want, drop it anytime, and even stack several.

Why LoRA is needed

First, see full fine-tuning's pain point clearly. To adapt a billions-of-parameters model to a new task, the traditional way is to keep training and update all its weights, then store this whole new weight set. The problem: each task needs its own tens-of-GB model, training takes lots of VRAM - expensive, space-hungry, and hard to share.

LoRA takes a different tack: freeze the original weights, not a byte changed, and learn just two small matrices A, B alongside. To use it, temporarily add the delta computed from A and B onto the original weights. So the "adapter" you store, share, and load is just those two small matrices A and B, often only a few MB - compared to tens of GB of full fine-tuning, several orders of magnitude smaller.

The key assumption behind this is: the change fine-tuning brings to the weights often concentrates in a low-dimensional subspace. In other words, the adjustment needed to "make the model learn a task" has not that many degrees of freedom, and a low-rank matrix approximates it well. It is exactly this insight that makes "learn just two small matrices" possible, and surprisingly effective.

A more down-to-earth analogy. Full fine-tuning is like reprinting a whole book to change one sentence; LoRA is like sticking a few notes onto the original book - the book is untouched, yet the notes suffice to express your edits. To change an edit, peel the notes and swap a new batch; the original book is forever that one book. This "original untouched, edits attached" thinking is the source of all of LoRA's convenience.

The collaboration dividend is large too. In the community, what people share is no longer the tens-of-GB whole model but few-MB adapters - fast to download, cheap to store, and freely combinable like plugins. A popular base model is often surrounded by hundreds or thousands of LoRAs each with its own trick, a flourishing bought precisely by "lightweight and shareable".

Of course LoRA is not omnipotent. It excels at "stylistic, domain-specific adjustments on top of existing ability", but to make the model learn a brand-new skill it utterly lacks, the low-rank delta's expressiveness may fall short, and heavier training is then needed. Knowing its boundary lets you use it where it counts - most needs at the "tone, format, domain" level, LoRA catches beautifully.

The LoRA math

How exactly is it computed? Let the original weight be W and the input x. The base output is W*x (an ordinary matmul). LoRA adds a low-rank branch alongside it: first A drops x to a very low dimension (rank r), then B lifts it back to the original dimension, giving B*(A*x); multiply by a scale factor and add onto W*x. The final output = W*x + scale*B*A*x.

Draw this formula by "dimension" and the low-rank bottleneck pops out: the base W*x keeps its width; the bypass is first squeezed by A to a narrow rank r, lifted back by B, scaled, and added to the base.

// simplified from src/llama-graph.cpp build_lora_mm
res = ggml_mul_mat(w, cur);                  // base weight output W*x
for (lora : active_adapters) {
    ab = ggml_mul_mat(b, ggml_mul_mat(a, cur)); // low-rank two steps B*(A*x)
    ab  = ggml_scale(ab, scale);             // scale = alpha/rank * user ratio
    res = ggml_add(res, ab);                 // add the delta
}

The snippet above (simplified from src/llama-graph.cpp's build_lora_mm) is what it does at graph-build time (L16): first compute the base mul_mat(w, cur), then for each active adapter compute B*(A*cur), multiply by scale, and add back. Note all this happens in the compute graph - the delta is computed and added on the fly, never actually modifying those GB of W weights.

That scale matters too: it is computed from the adapter's alpha and the rank r (roughly alpha/rank, times a user-given ratio). This ratio is the "strength" knob you can tune at attach time - turn it up for a stronger adapter influence, down to stay closer to the original model. Making strength adjustable lets you glide smoothly between "as-is" and "fully transformed".

Let me spell out the word "rank" a bit more. A matrix's rank is, roughly, how many "truly independent directions" it has. Full rank means all directions are used, maximal information but also maximal parameters; low rank says "actually only a few directions suffice to describe this change". LoRA's bet is exactly that the change needed to adapt a task is essentially low-rank, so r directions (the inner dimension of A and B is r) approximate it well enough.

One often-missed detail: the initialization of A and B is asymmetric. Usually B is initialized to all zeros and A randomly, so at the start of training the delta B*A is zero - that is, attaching an untrained LoRA has no effect on the model, the same as attaching none. Only training gradually grows useful directions in this delta. This "start from zero, join smoothly" design makes LoRA training both stable and safe.

Loading and applying

# pseudocode: load and attach a LoRA
adapter = llama_adapter_lora_init(model, "style.gguf")   # read A/B tensors
llama_set_adapters_lora(ctx, [adapter], n=1, scales=[0.8])  # batch attach, each with scale
# ... decode a few steps, output carries this style ...
llama_set_adapters_lora(ctx, [], n=0, NULL)              # n=0 => clear, detach all

It is simple to use: llama_adapter_lora_init reads the A, B tensors from a .gguf adapter file, giving an adapter object; then llama_set_adapters_lora attaches it to the context (L17) with a scale. Every subsequent decode automatically folds this adapter's delta in at graph-build time.

"Attach several at once, each with a scale" is no ornament but the basis of stacking abilities: you can attach a "Chinese style" and a "legal domain" LoRA at once, each with a weight, giving the model both specialties. The batched interface naturally supports this combination - which is why it is designed as a set of {adapter, scale}, not one-at-a-time.

Stress that "no weight copy" again: attaching merely records on the context "which adapters are active now, each at what scale"; the real addition happens at each decode's graph build (build_lora_mm). So attaching and detaching a LoRA is nearly free - involving no copy or modification of those GB of weights, switching styles as fast as swapping a filter.

Matching by target tensor name at attach time also echoes L15's naming convention. Each LoRA tensor's name records which layer's which weight it modifies (say a layer's attn_q). At graph build, when build_lora_mm reaches that weight, it looks up the adapter by name for a matching A, B, and adds the delta if found. Names once again become the key that pairs "weight" with "patch".

This design brings another benefit: the same LoRA file can apply to any structurally compatible base model - because its modification targets are specified by name, not bound to a specific model instance. So a community LoRA trained for one architecture can often be used directly on different fine-tuned versions of that architecture. Name-driven loose coupling vastly expands an adapter's reuse range.

A common combination in practice worth mentioning: many people run a quantized base model (L12) + a LoRA adapter, enjoying the VRAM saved by quantization while gaining task specialty from the adapter. llama.cpp supports this "quantized base + LoRA" - the adapter's delta is added on demand at graph build, largely orthogonal to how the base is quantized. Saving VRAM and staying customizable, you can have both.

Control vectors and the hand-off

Besides LoRA, there is an even lighter "seasoning" means: the control vector (cvec). It touches no weights but adds a fixed-direction vector directly onto the activations (the residual stream) of certain layers - like nudging the model's "train of thought", tilting it overall toward some tone or tendency (more formal, more optimistic, and so on).

The difference is worth remembering: LoRA changes weights (adding a low-rank delta to matmul, affecting that layer's entire computation), expressive, able to learn complex adaptations; the control vector changes activations (shifting the residual stream along one direction), lighter, more like "seasoning", good at fine-tuning style along a semantic direction.

Connecting this lesson back to Part 4's main line: the adapter is attached to llama_context (L17), and at each llama_decode graph build (L16), build_lora_mm folds the delta into the relevant matmul. So LoRA/cvec is not a new system started from scratch but a thin layer of "behavior tuning" embedded in the existing inference loop - reusing the graph-build and context mechanisms you learned earlier.

How a control vector is "computed" is worth a mention. It often needs no training but uses a contrastive method: take a batch of "positive examples" (say formal-toned text) and a batch of "negative examples" (casual text), run each through the model and take a layer's activations; the direction of the difference between the two activation sets is roughly the direction of the concept "formal" inside the model. Add this direction vector to the residual stream and you push the output toward "more formal". Simple, direct, and training-free.

Step back to the common ground of LoRA and control vectors: both practice the same principle - the base model is read-only, the changes are attached and stackable. This is of one piece with L17 separating "read-only weights" from "session state" and L21 making the sampling strategy a pluggable chain. All of Part 4, really, plays this one theme over and over: settle the invariant, split out the mutable, so the system is both solid and flexible.

Part 4 ends here. Beyond it (Part 5) we step out of llama's internals to see how these abilities are exposed to you through the public API and command-line tools - having understood the machinery under the hood, next is learning to drive the car.