--n-gpu-layers in llama.cpp: how to set -ngl

Oct 4, 2026 · 7 min read

-ngl (also --gpu-layers and --n-gpu-layers) is the most layers llama.cpp will store in VRAM. The layers it leaves out stay in system RAM, which is how llama.cpp runs a model larger than the GPU's memory.

Source links point at llama.cpp build b11390 (October 4, 2026). The default has changed twice, from 0 (all layers on Metal) to every layer in build b6325 (#15434) and then to a value fitted to free memory in b7410, so check your build with --version.

What -ngl counts

llama.cpp counts each repeating block of the model as one layer and the output layer as one more. A GGUF with 40 blocks has 41 offloadable layers, and the load log reports offloaded 41/41 layers to GPU when all of them are on the GPU.

The input layer, the token embedding table, is not in the count. The loader keeps it on the CPU because there is “very little benefit” to offloading it.

Order matters when you set a number. Since build b7470 the output layer goes first, then blocks from the last one backward (#18148): on a 40-block model, -ngl 1 offloads only the output layer, and -ngl 33 offloads it plus blocks 8 to 39. Earlier builds offloaded blocks first and the output layer only when -ngl exceeded the block count.

-ngl 0: no layers on the GPU. The build docs use it to turn off GPU inference on Metal.
-ngl N: the output layer plus the last N-1 blocks. A value above the layer count is capped, so -ngl 999 means every layer.
-ngl all: every layer.
-ngl auto: the default. --fit picks the number, or every layer goes with --fit off.

A build without a GPU backend ignores the flag and warns no usable GPU found. llama.cpp reaches NVIDIA GPUs through CUDA, AMD GPUs through HIP (ROCm), and others through Vulkan, Metal and SYCL (backends).

Where the memory goes

An offloaded layer's weights go to VRAM, and so does its part of the KV cache: llama.cpp allocates the cache one layer at a time on the device that holds the layer. -nkvo (--no-kv-offload) keeps the whole cache in system RAM instead.

Each device also gets a compute buffer for intermediate results. llama.cpp sizes it when it creates the context and logs it as compute buffer size; it is the one part of the memory bill you cannot work out from the model file.

Blocks left behind, and the embedding table, stay in system RAM, and llama.cpp schedules each operation on the device that holds its weights. With --op-offload, on by default, a GPU backend can still claim work on those weights; the CUDA backend does once a batch reaches a minimum size.

With several GPUs, the default --split-mode layer divides the offloaded layers between them in proportion to each device's free memory. --tensor-split 3,1 sets the proportions yourself.

Let --fit choose

Since build b7410, llama.cpp fits the settings you leave unset to the free memory on each device, and --fit is on by default (#16653).

It starts from the defaults, every layer on the GPU at the full trained context. If that leaves less than 1,024 MiB free on a device (--fit-target), it shrinks the context, no lower than 4,096 tokens (--fit-ctx), and only then moves layers to system RAM, starting with the expert weights of a mixture-of-experts model.

It leaves alone anything you set. Pass -c and the context stays; pass any -ngl, all and 999 included, and when memory runs short fit logs n_gpu_layers already set by user and loads your value as given.

llama-fit-params -m model.gguf

llama-fit-params prints the flags fit would use, -ngl among them, so you can pin them in a script.

Programs built on the C API alone skip this step, because fit lives in llama.cpp's common library and not in llama.h. There, llama_model_default_params() sets n_gpu_layers to -1, which llama.h defines as all layers.

Work it out by hand

The example is Qwen3-14B-f16.gguf from ggml-org/Qwen3-14B-GGUF on Hugging Face, whose file viewer lists every tensor's shape and type. These are the numbers that matter:

Quantity	Value	From the GGUF
Blocks	40, so 41 offloadable layers	qwen3.block_count
One block	630.04 MiB	blk.N.* tensors
Output layer	1,483.77 MiB	output.weight, output_norm.weight
Embedding table (stays on the CPU)	1,483.75 MiB	token_embd.weight
KV heads	8, each 128 wide for K and for V	qwen3.attention.head_count_kv, key_length, value_length
Trained context	40,960 tokens	qwen3.context_length

One block: attn_q and attn_output are 5120 × 5120, attn_k and attn_v 5120 × 1024, and ffn_gate, ffn_up and ffn_down hold 5120 × 17408 values each, all f16 at 2 bytes, plus 41 KiB of f32 norms. The output layer is 5120 × 151936 at f16.

The KV cache holds a K and a V vector per KV head for every token in every layer, at 2 bytes per value for the default f16 cache type: (8 × 128 + 8 × 128) × 2 = 4,096 bytes per token per layer. At -c 8192 that is 32 MiB per layer; leave -c unset and one sequence gets the trained 40,960 tokens, 160 MiB per layer.

For the GPU, llama.cpp's fit-params example shows a 24 GB card reporting 24,077 MiB free. llama-server --list-devices prints the total and free MiB of each device on yours.

  24,077.00  free VRAM
-  1,024.00  margin --fit keeps free (--fit-target)
-  1,483.77  output layer
= 21,569.23  MiB for blocks

-c 8192:   630.04 + 32 = 662.04 MiB per block
           21,569.23 / 662.04 = 32.6  ->  32 blocks  ->  -ngl 33

-c 40960:  630.04 + 160 = 790.04 MiB per block
           21,569.23 / 790.04 = 27.3  ->  27 blocks  ->  -ngl 28

These are ceilings, because the compute buffer is still missing. At -c 8192, 32 blocks leave 383.95 MiB above the margin; if the compute buffer size llama.cpp logs is larger, take off one block per 662.04 MiB of overshoot, rounding up.

At -ngl 33 the 8 blocks left behind need 5,040.32 MiB of system RAM, plus 1,483.75 MiB for the embedding table and 256 MiB for their part of the KV cache.

The same repository's Qwen3-14B-Q4_K_M.gguf needs 8,161.75 MiB for everything but the embedding table, and the full 40,960-token cache adds 6,400 MiB. Against 23,053 MiB (24,077 free less the margin), that 14,561.75 MiB leaves 8,491.25 MiB for the compute buffer, so the answer there is -ngl all.

Quantized blocks are not all one size. In that file they run from 177.23 to 200.43 MiB, because half of them store attn_v and ffn_down as Q6_K, so add them up from the last block backward, the order they offload in.

To see what each layer buys on your hardware, measure it. llama-bench takes a list and reports tokens per second for each value:

llama-bench -m model.gguf -ngl 0,10,20,30

When -ngl is too high

A number above the layer count is harmless, since llama.cpp caps it. A number whose layers do not fit fails at load.

Take -ngl 41 -c 8192 on the f16 example: the weights alone are 26,685.37 MiB against 24,077 MiB free. Because you set -ngl, fit gives up, and on the CUDA backend the load ends like this:

common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 41, abort
ggml_backend_cuda_buffer_type_alloc_buffer: allocating <size> MiB on device 0: cudaMalloc failed: <error>
llama_model_load: error loading model: unable to allocate CUDA0 buffer
llama_model_load_from_file_impl: failed to load model

A HIP (ROCm) build compiles the same backend source, so it logs the same lines with the device buffer named ROCm0. If the weights fit but the KV cache or the compute buffer does not, the model loads and creating the context fails instead, with failed to allocate buffer for kv cache or failed to allocate compute buffers.

Ways out:

Drop -ngl and let --fit choose, or lower it.
Lower -c. In the f16 example every 1,024 tokens of context cost 4 MiB per offloaded layer.
Quantize the cache with -ctk q8_0 -ctv q8_0: ggml stores q8_0 as 34 bytes per 32 values, against 64 for f16. A quantized V cache needs Flash Attention, which llama.cpp turns on for it unless you turned it off.
Keep the KV cache in system RAM with -nkvo.
On a mixture-of-experts model, --cpu-moe keeps all expert weights in system RAM, and --n-cpu-moe N does it for the first N layers.
On Linux with the CUDA backend, GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 lets allocations spill into system RAM instead of crashing. The build docs warn that on HIP it hurts performance on GPUs that are not integrated.

Every layer on one GPU

When the model, its KV cache and the compute buffer fit on one GPU, -ngl all or the auto default puts every layer there. OpenRelay rents GPU VMs, with each GPU passed through whole, and container pods by the hour; the GPU catalog lists each GPU's memory for the arithmetic above.