后端调度Backends & dispatch

The last two lessons watched CPU (L31) and CUDA (L32) each compute ops on their own. But in reality a machine often has a CPU and one or more GPUs at once - how does ggml manage them uniformly, and how does it decide which op goes to whom? This lesson looks at the ggml-backend abstraction: the close of Part 6, and the key link tying all the earlier "how to compute" back to "where to compute".

Recall the compute graph from L09/L10: a chain of ops flowing from input to output. Now the question: within that chain, some ops are best on the GPU (matmul), some are simpler left on the CPU; and a tensor may be in VRAM now, in RAM later. Who decides where each op goes, and who moves tensors between devices? The answer is ggml's backend abstraction and scheduler. This lesson sees how it hides hardware differences behind one uniform interface, how it loads backends dynamically at runtime, and how it dispatches a whole graph for execution. A most relatable scenario: you run a big model on a card whose VRAM is a bit short, so you put some layers on the GPU and leave the rest on the CPU - now every pass of the model shuttles data back and forth between CPU and GPU several times, all riding on the backend and scheduler this lesson covers. Understand it and you truly see what happens behind that -ngl flag.

Roadmap: first the backend abstraction (one uniform interface), then registry and dynamic loading (one binary picks backends by hardware), then the scheduler (dispatch a graph's ops to backends and move tensors as needed), and finally a quick tour of the other backends ggml supports.

The backend abstraction: one uniform interface for all hardware

A ggml_backend represents "a device that can run ops", bundling three things: a device handle, a set of buffers (memory to allocate/read/write tensors on that device), and a set of op implementations (how this hardware computes matmul, softmax, ...). Every backend (CPU, CUDA, Metal, ...) implements the same interface (ggml_backend_i in ggml-backend-impl.h), so the upper compute graph only writes to this abstraction and never minds which hardware is underneath. The buffer here is an easily-overlooked key: it decides whether a tensor's data sits in CPU RAM or GPU VRAM; the same tensor is two different chunks of memory in a CPU buffer versus a GPU buffer, and what the scheduler later moves across devices is exactly the data in these buffers. Devices also carry a type, so the upper layer can "pick by need":

// device types (ggml-backend.h)
enum ggml_backend_dev_type {
    GGML_BACKEND_DEVICE_TYPE_CPU,    // CPU
    GGML_BACKEND_DEVICE_TYPE_GPU,    // discrete GPU
    GGML_BACKEND_DEVICE_TYPE_ACCEL,  // accelerator (used with the CPU, e.g. BLAS/AMX)
    // ... also IGPU (integrated GPU) / META, etc.
};
// pick a GPU device; fall back to CPU if none (usage in ggml-backend-reg.cpp)
ggml_backend_dev_t dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_GPU);
if (!dev) dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU);

Half the payoff of this design already showed up in L31/L32: when the upper layer calls matmul it never writes "AVX2 or CUDA" - that lives inside each backend's implementation. The backend abstraction formalizes this: the compute graph (L09/L10) describes "what to do", the backend handles "how and where". Graph and hardware are fully decoupled:

Registry and dynamic loading: one binary, backends by hardware

We have the abstraction, but how does the program know which backends this machine has? The answer is runtime dynamic loading. Each backend compiles to a shared library (.so/.dll); at startup the program calls ggml_backend_load_all() to scan and load the usable backend libraries one by one and register them in a registry. The low-level worker is dl_load_library:

// load all available backends at runtime (ggml-backend-reg.cpp)
void ggml_backend_load_all();   // scan and load cuda/metal/vulkan... shared libs

// low-level single-library load (ggml-backend-dl.cpp), two platform paths
// POSIX:
handle = dlopen(path, RTLD_NOW | RTLD_LOCAL);
// Windows:
handle = LoadLibraryW(path);

Once loaded, the program can ask the registry "how many backends, how many devices now": ggml_backend_reg_count(), ggml_backend_dev_count(), then pick a device with last section's ggml_backend_dev_by_type. The point of this dynamic loading is concrete: one main binary loads the matching backend at runtime per the machine's actual hardware - load the CUDA backend if there is a CUDA card, skip it otherwise, never failing to start over a missing library. Think of the registry as a "phone book": each loaded backend writes one entry - "who I am, what devices I have, how to reach me". When the upper layer wants a GPU, it does not go straight to CUDA but looks it up in the book by type - it is exactly this indirection that lets "the upper layer not know the concrete hardware" actually hold.

Scheduling: dispatching a graph's ops across backends

With several backends, the last question is: in one compute graph (L09/L10), who decides which backend each op runs on, and who moves tensors between backends? That is ggml_backend_sched's (the scheduler's) job. Its two main interfaces:

// build a scheduler, hand it a set of backends (ggml-backend.h)
sched = ggml_backend_sched_new({backend_gpu, backend_cpu}, ...);
// dispatch the whole compute graph across backends
ggml_backend_sched_graph_compute(sched, graph);

Given the graph, the scheduler walks each op: which backend's buffer are its input tensors in now? Which backend best suits this op (matmul prefers GPU, say)? It assigns the op to a backend accordingly; if some input is still on another device (the op runs on GPU but the input is still in CPU memory), the scheduler first inserts a cross-device copy to move the input over, then executes. When the whole graph is done, results land where they should be. Worth noting: the scheduler does not dispatch each op in isolation - cross-device copies are expensive (memory traffic again, echoing L30/L32), so it tries to give consecutive ops that can run on the same device to one backend in segments, cutting the back-and-forth. In other words, it weighs not just "where does this op run fastest" but "how to cut this graph so total cross-device movement is least" - so it leans toward giving consecutive work that can run on the same device to one backend in segments, rather than cutting it up layer by layer. Freezing "dispatching one op" into a flow shows it clearest:

This is exactly where L09/L10's "static graph" connects to "dynamic execution": the graph describes dependencies, and the scheduler, following them and each tensor's actual location, drops the ops one by one onto concrete hardware. When you use -ngl N to put the first N layers on the GPU and leave the rest on CPU, this scheduler is what dispatches by layer and shuttles boundary tensors between GPU and CPU. And when the GPU's VRAM truly cannot hold the whole model, this CPU+GPU hybrid execution is often the only way to run at all - at the cost of a few cross-device copies at the boundary, but far better than not running. It also explains why turning -ngl up or down trades speed against VRAM use: the more layers on the GPU, the faster it computes, but the more VRAM it takes, and the CPU-GPU handoff points shift too.

One "how the whole graph is cut" picture makes -ngl's effect concrete - take a 32-layer model with -ngl 20:

This picture also explains why "place in segments" beats "scatter": chaining the GPU layers into one run leaves the whole graph with a single cross-device boundary and one copy; interleave GPU and CPU layers and every switch costs a copy, and that meager PCIe bandwidth is soon eaten by copying. The scheduler defaults to "the first N layers as one GPU run" precisely to keep such boundaries minimal.

A tour of the other backends

Beyond the CPU (L31) and CUDA (L32) we examined closely, ggml implements a whole crowd of backends covering everyone's hardware. They all follow the same ggml_backend_i interface, so upper-layer code barely changes - switching hardware just means loading a different backend:

This table best shows the value of the backend abstraction: from Apple's Metal to Huawei's CANN, from a local GPU to remote RPC, the hardware is worlds apart, yet to the upper layer each is just "a device implementing ggml_backend_i". Supporting a new piece of hardware is essentially writing one more backend implementation and registering it - not a single line of the upper model code needs to change. In fact, many of these backends were contributed by the community or hardware vendors: precisely because the interface is uniform, Huawei can write CANN themselves and Intel can write SYCL, without touching ggml's core. A good abstract interface opens a door for the whole ecosystem - "you adapt the hardware, I guarantee the upper layer stays put" - and that is the organizational reason an open-source project can support so much hardware so fast.

Deeper: special backends and how to add one

Two last folds: two "not very GPU-like" special backends, and roughly what adding a new backend takes.

Part 6 closes here. From L31's CPU instructions (scalar, SIMD, multithreading), through L32's CUDA threads (grid/block, tiling, the memory hierarchy), to this lesson's backend abstraction and scheduling, the thread of "how one op finally lands on hardware to compute" has been told end to end. Zooming back out: the model (Part 4) describes what to compute, the compute graph (Part 3) organizes it into dependencies, and the backend (this part) lands the graph on concrete hardware - join the three layers and the whole picture of how llama.cpp runs a model is clear. Next stop, Part 7, where we look at some advanced topics. By here, you have taken llama.cpp from "a black box that runs models" to "a stack of legible layers" - and that is the biggest reward of reading source: however complex, a system is never a tangle but a stack of abstractions, each solving one problem and exposing one clean interface to the layer above. Carry this lens away and you will read any large project far more calmly than before.