Intermediate
What is GGUF?
The file format that made running LLMs locally practical.
GGUF (GPT-Generated Unified Format) is a file format for storing quantized language models, designed for fast loading and efficient CPU/GPU inference. It's the standard format used by llama.cpp and its many frontends.
A GGUF file bundles the model weights at a chosen quantization level (like Q4_K_M or Q8_0) plus metadata. You download one file and run it — no Python environment or complex setup required.
When you see 'Llama 3 8B Q4_K_M GGUF' on Hugging Face, that's a 4-bit quantized Llama 3 8B packaged for local inference, typically needing ~5 GB of RAM.
Key points
- Standard format for quantized local models
- Used by llama.cpp and many apps
- One file = runnable model
- Q4_K_M etc. denote quantization level