Choose a model that fits
Understand quantization
Use lower-precision weights to reduce memory while treating quality and runtime compatibility as measured tradeoffs.
Quantization represents weights with fewer bits.
A full-precision release might use BF16 or FP16. Local variants commonly use 8, 6, 5, 4, or fewer bits per weight.
Reducing precision makes the file smaller and usually reduces memory use. It may also improve speed when the hardware and runtime have efficient kernels for that format.
The tradeoff is quantization error. Two nearby original values may become the same lower-precision value. Good methods choose scales and blocks carefully so useful behavior survives.
Names such as Q4_K_M describe a particular GGUF quantization recipe. The number is not a complete quality score. A strong 4-bit quantization can be a better practical choice than a poorly supported higher-precision file.
My advice is to start around 4 bits for a local experiment. Move upward when you have enough memory and your evaluation shows a meaningful improvement. Move downward only when fitting the model matters more than the quality you lose.
Always compare the exact artifacts. “The 8B model” is incomplete when one test used BF16 and another used a 4-bit conversion.
Lesson completed