Configure GPU Layer Offloading via the -ngl Parameter

To enable GPU acceleration, you must explicitly instruct llama.cpp to offload model layers to your graphics hardware.

1. Open the terminal or command prompt window used to launch your model.
2. Identify your current execution command for the llama.cpp server or main binary.
3. Append the -ngl (number of GPU layers) flag to your command. For example, to offload all layers, use: -ngl 999.
4. Execute the updated command.
5. Verify the fix by observing the console startup output; ensure it indicates that layers are being successfully offloaded to your GPU (e.g., 'X/X layers offloaded').
6. Confirm GPU utilization by monitoring your hardware activity using nvidia-smi or an equivalent GPU monitoring tool during inference.

Sources