Q4, Q5, and Q8 quantization comparison in Ollama
Choosing an Ollama quantization level determines your local model's speed, memory footprint, and output quality all at once. Here’s how Q4, Q5, and Q8 differ, and how to pick the right level for your own hardware.
Ollama quantization tags can look confusing at first glance: the same model comes in variants like q4_K_M, q5_K_M, and q8_0. Quantization means storing model weights using fewer bits. As the bit count drops, the model uses less memory and runs faster on most hardware—though beyond a certain point, output quality also drops.
What do the levels change?
- Q4 (e.g., q4_K_M): the smallest and usually fastest option. An 8-billion-parameter model takes about 5 GB.
- Q5 (e.g., q5_K_M): slightly larger (around 5.5–6 GB for an 8B model), with less quality loss than Q4.
- Q8 (q8_0): closest to the original model; around 8.5 GB for an 8B model and requires noticeably more memory bandwidth.
Memory allocated for the context window adds on top of these sizes. If you use long contexts, leave at least a few gigabytes of headroom on your GPU (VRAM); if the model doesn't fit in memory, part of it offloads to the CPU and speed drops sharply.
| Level | Approx. size (8B model) | Speed | Quality | Best for |
|---|---|---|---|---|
| Q4 (q4_K_M) | ~5 GB | Highest | Acceptable loss | Limited memory, fast responses |
| Q5 (q5_K_M) | ~5.5–6 GB | High | Better than Q4 | General use, classification |
| Q8 (q8_0) | ~8.5 GB | Lower | Closest to original | Long, detailed text |
In the tags, “K” stands for the k-quant method and “M” stands for a medium mix: some layers of the model are stored at higher precision and others at lower precision. At the same bit count, “K_M” variants generally deliver better quality than older methods.
How to test on your own hardware
General tables are a good starting point, but a quick test with your own workload is much more reliable for making a decision. Pull all three versions of the same model and run them against the same prompt set:
ollama pull llama3.1:8b-instruct-q4_K_M
ollama pull llama3.1:8b-instruct-q5_K_M
ollama pull llama3.1:8b-instruct-q8_0
# --verbose, yanıtın sonunda saniyedeki token sayısını (eval rate) yazar
ollama run llama3.1:8b-instruct-q5_K_M --verbose- Pick 15–20 prompts from your actual workload and run each one at least three times.
- Note the tokens per second and memory usage (ollama ps).
- If you require structured output, separately track the valid JSON rate.
Results vary by GPU, driver, and model. Benchmarks from one machine don't translate directly to another.
Use these commands to check whether the model fits entirely in memory and inspect the context window:
# Yüklü modeller, bellek kullanımı ve GPU / CPU payı
ollama ps
# Bağlam penceresini sabitleyen özel model
cat > Modelfile <<'EOF'
FROM llama3.1:8b-instruct-q5_K_M
PARAMETER num_ctx 8192
PARAMETER temperature 0.2
EOF
ollama create llama31-q5-8k -f ModelfileTo evaluate quality, a simple scoring rubric is enough. Score each response from 1 to 3 using the same criteria:
- Accuracy: does the response provide the requested information accurately?
- Format: does it follow the requested structure (list, JSON, length)?
- Consistency: are the results similar when the same prompt is repeated?
Which level should you choose?
For classification, summarization, and short structured outputs, Q5 is often the sweet spot. Start with Q4 if memory is tight or speed is your top priority. Q8 makes sense when you have plenty of memory and the quality of long, detailed text is critical. For strictly formatted outputs like JSON, you'll need a validation layer regardless of the quantization level; see my note on getting reliable JSON output from local LLMs for details.
You can see how I made this choice in a real-world application in the Local AI Product Image Generation project.
Conclusion
Choosing an Ollama quantization level is about trade-offs rather than a single right answer: you need to find the right balance between memory, speed, and quality for your workload. For help with hardware selection, model deployment, and benchmarking, check out my local AI setup service.
