Understand quantization

Quantization represents weights with fewer bits.

A full-precision release might use BF16 or FP16. Local variants commonly use 8, 6, 5, 4, or fewer bits per weight.

Reducing precision makes the file smaller and usually reduces memory use. It may also improve speed when the hardware and runtime have efficient kernels for that format.

The tradeoff is quantization error. Two nearby original values may become the same lower-precision value. Good methods choose scales and blocks carefully so useful behavior survives.

Names such as Q4_K_M describe a particular GGUF quantization recipe. The number is not a complete quality score. A strong 4-bit quantization can be a better practical choice than a poorly supported higher-precision file on your machine.

My advice is to start around 4 bits for a local experiment. Move upward when you have enough memory and your evaluation shows a meaningful improvement. Move downward only when fitting the model matters more than the quality you lose.

Always compare the exact artifacts. “The 8B model” is incomplete when one test used BF16 and another used a 4-bit conversion.

Notice that quantization is not magic compression. It is a lossy encoding of learned numbers. Summarization might survive Q4 fine. Delicate reasoning or rare code patterns might not. Your task decides whether the loss matters.

When Ollama pulls gemma3:1b, you receive a Q4_K_M artifact by default. That tag is part of your test configuration. Record it beside every evaluation result so you can reproduce the run later.

If a model loads but answers feel dull, try the next step up in bit width on the same architecture before you jump to a much larger parameter count. Sometimes precision was the bottleneck, not capacity.

Try this on your own project: run the same five prompts against two quantizations of the same base model and note where factual errors appear. That diff is worth more than any generic “Q4 is fine” claim.