What quantization really is

What is quantization?

You download a 12-billion-parameter model. The card says it is “open.” Your GPU has 16 GB. The loader asks for something closer to 27 GB and dies. However somehow a different filename ending in q4_0 works flawlessly. Same model name, a quarter of the weight file. The work has a name.

Quantization means representing model values (usually weights, sometimes activations or the KV cache) with fewer bits via a scale and optional zero-point. Same architecture; smaller numerical ruler – not a smaller network.

That last sentence is the part people get wrong. A 4-bit checkpoint still has the same layers, attention, vocabulary, and parameter count as its 16-bit twin. Nothing was “distilled away.” You did not turn a 12B model into a 4B model. You changed how many distinct numbers each weight is allowed to take, and you stored a scale so those integers can be turned back into approximate floats when the runtime needs them.

Why the tip exists: memory first

Local inference usually fails on memory before it fails on FLOPs. A rough lower bound for the weight footprint is:

Ignore metadata, scratch buffers, the KV cache, and the runtime for a moment. A 12B model still needs about 24 GB of weight storage at 16 bits, and about 6 GB at 4 bits. Google’s Gemma 4 docs put the real loading numbers at 26.7 GB in BF16 and 6.7 GB in Q4_0, including ~20% overhead – roughly a 75% cut in the static footprint, not a 75% cut in intelligence. (Gemma 4 model overview)

There is a second payoff once the model fits: generation rereads those weights token by token. Fewer bytes to move can mean higher tokens/s – if the runtime has efficient low-bit kernels. Hardware, batch size, and context length decide whether you feel that. Fitting at all is the first win; speed is a maybe.

How the ruler works

So how do you actually put a float onto a smaller ruler? In symmetric quantization you pick a scale from the largest absolute value and map each value to an integer:

For signed N-bit integers, a common choice is:

Four bits give you 16 codes. Sixteen-bit float gives you a much wider dynamic range. The difference is the approximation error:

You do not need every error to be zero. You need errors not to pile up where the task cares.

When values are not centered on zero, asymmetric quantization adds a zero-point:

A shared scale and zero-point can cover a whole tensor, a channel, a token, or a small group – that is granularity / group-wise quantization. Finer scales usually preserve quality; they also cost more metadata and can be harder for kernels. These are the standard uniform formulas used across LLM papers, including SmoothQuant (paper).

A tiny example makes the failure mode obvious. Take:

Signed 4-bit, max magnitude 1.00 → $s \approx 1/7 \approx 0.1429$. Then $0.71$ becomes:

Fine. Now replace one value with an outlier of 10.0 while the rest stay near zero. The scale jumps to cover that spike, and many ordinary values collapse onto the same few codes. That is why “float → int” is incomplete: distribution, granularity, clipping, and calibration – measuring real ranges on representative inputs – decide whether the smaller ruler still measures the parts of the model that matter.

What you choose to shrink

Once the mapping is clear, the next question is which tensors get the smaller ruler.

• Weight-only (e.g. W4A16) – 4-bit weights, 16-bit activations. The usual local-LLM choice: weights dominate storage and the recipe is comparatively robust.
• Weight + activation (e.g. W8A8) – also low-bit activations. Can unlock integer matmul; activations are dynamic and often full of outliers, so this is harder.
• KV cache quantization – shrinks the growing conversational state. Separate decision from quantizing the static weights you downloaded.

The letters are a contract: W4A16 means 4-bit weights and 16-bit activations. A bare Q4 in a filename is incomplete until you know format and kernel – which is why the Discord tip worked only because everyone already assumed GGUF + llama.cpp.

When you apply it: after training or during

Two production paths, same goal.

PTQ (post-training quantization): train normally, convert afterward. Cheap. Aggressive conversion can hurt, especially with activations or coarse scales.

QAT (quantization-aware training): train while simulating rounding and clipping so weights adapt to the deployment constraint. Google’s Gemma 4 QAT checkpoints are built this way to keep quality while cutting memory. (QAT announcement)

One naming trap: a QAT checkpoint may ship as half-precision weights for further conversion, as a quantized GGUF for llama.cpp / LM Studio, or as compressed tensors for vLLM / SGLang. “Unquantized QAT” does not mean “trained without quantization.” It means half-precision weights from a QAT pipeline.

How you actually do it

Where the model runs picks the packaging. Local CPU / Apple Silicon / consumer GPU: a pre-quantized GGUF and a llama.cpp-compatible runtime is usually the least painful. Production GPU serving: use what the engine expects – Google routes Gemma QAT GGUF to llama.cpp / LM Studio and W4A16 compressed tensors to vLLM / SGLang. (Deployment routing)

A generic PTQ checklist, in the order you will actually care about:

1. Target representation (W8A16, W4A16, …).
2. granularity (per-tensor, per-channel, group-wise).
3. calibration data that looks like real prompts.
4. Measure ranges, pick scales, optionally clip outliers.
5. Quantize weights; keep scale metadata.
6. Export (GGUF, GPTQ, AWQ, compressed tensors, …).
7. Evaluate on standard benchmarks and your real prompts.
8. Measure peak memory, prefill, generation, and quality on the target box.

The core of symmetric weight quantization, as a teaching sketch:

import torch
 
def quantize_symmetric(weights, bits=4):
    qmax = 2 ** (bits - 1) - 1
    scale = weights.abs().amax().clamp_min(1e-8) / qmax
    quantized = torch.round(weights / scale).clamp(-qmax - 1, qmax)
    return quantized.to(torch.int8), scale
 
def dequantize(quantized, scale):
    return quantized.float() * scale

Real formats pack multiple low-bit values into bytes, store per-group scales, and need optimized kernels. A mathematically correct conversion can still be slow if the runtime unpacks inefficiently – which brings us back to why “it fits” and “it is fast” are different claims.

Gemma 4 puts numbers on the tip

Dense Gemma 4 12B instruction-tuned is a clean case. Baseline: google/gemma-4-12B-it in BF16. Quantized path: official QAT Q4_0, typically as GGUF (google/gemma-4-12B-it-qat-q4_0-gguf).

Google’s figures include ~20% overhead. Q4_0 saves ~20 GB vs BF16. Context, KV cache, framework overhead, and multimodal inputs sit on top – a “6.7 GB model” does not mean a machine with exactly 6.7 GB of VRAM will run it happily. (Memory requirements)

Parameter count alone is incomplete in another way too. Gemma 4 includes a 26B A4B MoE that activates ~4B parameters per token, but all experts still load for fast routing. Quantization shrinks the storage cost of those weights; it does not make inactive experts disappear. Same lesson as the opening: smaller ruler, not fewer layers.

Quality is a claim you have to earn

There is no universal “quantization accuracy penalty.” Bit width, quantizer, calibration, group size, runtime, and task all move the needle. A 4-bit model can look identical to baseline on one workload and stumble on precise arithmetic, long reasoning chains, or rare tokens.

For Gemma 4 specifically: Google publishes family capability scores, but the public card does not give a controlled BF16-vs-Q4_0 table. QAT is described as staying close to BF16; the docs give hard memory numbers. That supports a deployment claim about memory. It does not license inventing a quality delta.

Resist filling that last row with a number from a different runtime, prompt template, revision, or recipe. Useful as engineering notes; not a controlled comparison.

To earn your own numbers, keep everything except precision fixed:

# Matching chat templates and the same prompt set.
llama-bench -m gemma-4-12b-it-bf16.gguf -p 512 -n 128
llama-bench -m gemma-4-12b-it-qat-q4_0.gguf -p 512 -n 128

Record prefill throughput, generation throughput, peak RSS, time-to-first-token, and quality on a fixed eval set. Report model revision, quant format, runtime version, hardware, context, batch size, sampling, and dataset – or “Q4 is 2× faster” is just another Discord tip without receipts.

When it changes what can run at all

Quantization matters most when it changes the set of machines that can host the model. Google’s Gemma 4 estimates put 31B at 69.9 GB BF16 vs 17.5 GB Q4_0 – multi-GPU vs something that can fit a high-end 24 GB card before context and runtime. (Model overview)

Speed stays less predictable: good kernels and less traffic help; bad unpack paths or prefill-dominated workloads do not. Energy and cost follow the systems story – fewer GPUs, lower-power devices, more concurrent requests – not promises glued to the string Q4 in a filename.

The whole loop, in one picture:

Spend bits where they protect quality; remove them where they mostly encode redundant detail. The best quantized model is not the one with the fewest bits – it is the one that meets quality, latency, memory, and cost for the job you opened Discord about.

Takeaways

1. Same architecture, smaller ruler – that is the definition, and it is enough to kill the “smaller model” myth.
2. Memory is why the tip exists; kernels decide whether speed follows.
3. outliers, calibration, and granularity decide whether the tip still answers well.
4. Gemma 4 makes the payoff concrete: 26.7 GB BF16 → 6.7 GB Q4_0 for 12B. Measure quality yourself; do not invent the delta Google did not publish.

Sources

• Google: Gemma 4 model overview and memory requirements
• Google DeepMind: Gemma 4 with quantization-aware training
• Hugging Face: Gemma 4 12B QAT unquantized checkpoint
• Xiao et al.: SmoothQuant, PMLR 2023
• Gemma 4 Developer Hub: quantization formats