Blog

Quantizing a 64B Llama Model Without Wrecking It

How to quantize a 64B-parameter Llama model to 4-bit: GPTQ vs AWQ vs GGUF, memory math, calibration, outlier channels, and what breaks in production.

Quantizing a 64B-parameter Llama model means converting its weights from 16-bit floats to lower-precision integers, typically 4-bit, to cut memory footprint from roughly 128 GB down to 32-36 GB so it fits on a single high-end GPU. The hard part isn't the arithmetic, it's preserving accuracy on the small fraction of outlier weight channels that dominate model quality at low bit widths.

The memory wall: why a 64B model forces the issue#

A 64-billion-parameter Llama-family model stored in fp16 costs 2 bytes per parameter. That's 128 GB just for the weights, before you add a single token of KV cache or any activation memory. No single accelerator on the market holds that: an H100 or A100 tops out at 80 GB of HBM, so full-precision inference means tensor-parallel sharding across at least two GPUs, and in practice three or four once you leave headroom for the KV cache at any real batch size or context length.

Drop to int8 and the weight footprint halves to roughly 64 GB. Drop to 4-bit and it halves again to about 32 GB, small enough to fit alongside a healthy KV cache on a single 40 GB or 48 GB card. That's the entire economic argument for quantization: it's not about making inference faster in an abstract sense, it's about collapsing a multi-GPU serving problem into a single-GPU one, which changes your fleet cost, your failure domain, and your ability to autoscale a fleet of replicas instead of babysitting a tensor-parallel group.

The catch is that quantization error doesn't distribute evenly across a transformer. A handful of channels in a handful of layers carry disproportionate magnitude, and naive round-to-nearest quantization on those channels can blow up perplexity far more than the average-case error would suggest. Everything interesting in a real quantization pipeline is about handling that long tail correctly.

Picking a scheme: GPTQ, AWQ, bitsandbytes, and GGUF#

Four schemes dominate open-weight LLM quantization today, and they solve the outlier problem differently.

GPTQ (Frantar et al. 2022) is a post-training, layer-by-layer method. For each linear layer it quantizes weights column by column and uses the inverse Hessian of the layer's activations (estimated from a small calibration set) to update the remaining unquantized columns, compensating for the error just introduced. It's the same idea as Optimal Brain Compression applied at scale, and it's what let the original paper quantize models up to 175B parameters to 3 or 4 bits on a single GPU in a few GPU-hours.

AWQ (Lin et al. 2023) takes a different angle: instead of correcting error after the fact, it identifies the small percentage of weight channels that are 'salient' based on activation magnitude, and applies a per-channel scaling factor before quantization so those channels see less relative error. No backward pass, no Hessian, and in the paper's numbers it holds up better than GPTQ at 3-4 bits on several model families, while being noticeably cheaper to run because there's no iterative reconstruction.

bitsandbytes NF4 (Dettmers et al. 2023, the QLoRA paper) is a 4-bit NormalFloat data type tuned to the actual distribution of neural network weights (roughly zero-centered, roughly Gaussian) rather than a uniform int4 grid, paired with double quantization of the quantization constants themselves. It's the natural choice if you're also planning to fine-tune with LoRA adapters on top of the frozen quantized base, since it was built for exactly that workflow.

GGUF / llama.cpp k-quants (llama.cpp) is a file format plus a family of quantization schemes (Q4_K_M, Q5_K_M, Q3_K_S, and so on) built for CPU and hybrid CPU/GPU inference outside the CUDA-only Python stack. The 'K' variants use per-block scales at finer granularity than plain int4 and let you mix precision by layer, trading a larger file for less quality loss on the layers that need it.

For a 64B model destined for GPU serving, GPTQ and AWQ are the practical defaults; GGUF matters when you need CPU fallback or edge deployment on machines without a datacenter GPU.

Pipeline: calibration, quantize, pack, serve#

The pipeline looks the same regardless of which scheme you pick, only the middle box changes:

Calibration architecture#

Pipeline diagram#

The calibration set matters more than people expect. It has to represent the distribution of prompts the model will actually see in production, because the Hessian estimate (GPTQ) or the per-channel activation statistics (AWQ) are computed against it. Calibrate a chat model with a code-heavy calibration set and you'll under-protect the channels that matter for conversational output, and vice versa. Most published recipes use 128 to 512 sequences of a few thousand tokens each pulled from something like C4 or a slice of the model's own training-adjacent distribution, which is usually enough since the goal is capturing per-channel activation statistics, not training the model.

Group size, symmetric vs asymmetric, and the accuracy/throughput knob#

Every practical scheme quantizes in groups, not per-tensor. A group size of 128 means every 128 consecutive weights in a row share one scale factor (and, for asymmetric quantization, one zero-point). Smaller groups track local weight distribution more precisely and lose less accuracy; larger groups mean fewer scale factors to store and slightly faster dequantization.

The overhead is arithmetic you can actually compute. With group size 128 and a fp16 scale (2 bytes) plus an int4 zero-point packed alongside, you're adding roughly 16 bits of metadata per 128 weights, or about 0.125 bits per weight, on top of the 4 bits of quantized weight itself. For a 64B model that's an extra ~1 GB on top of the ~32 GB of raw 4-bit weights, so real-world GPTQ/AWQ checkpoints for a model this size land around 33-36 GB depending on group size and whether the embedding and lm_head layers are left in higher precision (they usually are, because those layers are disproportionately sensitive and cheap enough in absolute terms not to bother quantizing).

Group size 128 is the de facto default because it's the point where quality loss flattens out for most 65-70B-class dense models in published GPTQ/AWQ results, while group size 32 buys a small additional accuracy gain at a real cost in packed-tensor overhead and kernel complexity. Symmetric quantization (zero-point fixed at the grid midpoint) is simpler and slightly faster to dequantize; asymmetric quantization fits skewed weight distributions better but costs an extra zero-point per group. AWQ's checkpoints and most GPTQ configs default to asymmetric for exactly that reason.

Serving: kernels, batching, and the KV cache decision#

A quantized checkpoint is inert without a kernel that can multiply int4-packed weights against fp16 activations without first materializing the full dequantized tensor. That's the job of kernels like Marlin and ExLlamaV2: they fuse the dequantization into the matmul itself, so the GPU never allocates a full fp16 copy of the weight matrix in HBM. Without a fused kernel, naive dequant-then-matmul can actually be slower than running the fp16 model, because you pay the dequant cost on every forward pass and still need the fp16 memory transiently. This is the most common way teams get quantization 'wrong' operationally: they cut the checkpoint size in half but forget the serving stack needs a matching kernel or they get none of the throughput benefit.

Batching interacts with this in a specific way: quantized kernels tend to show their biggest win at small-to-medium batch sizes, where the workload is memory-bandwidth-bound rather than compute-bound, because 4-bit weights move a quarter of the bytes off HBM per matmul compared to fp16. At large batch sizes the workload shifts toward compute-bound and the relative advantage of quantization narrows, since you're bottlenecked on FLOPs, not bandwidth.

The KV cache is a separate decision from weight quantization and it's often the one teams get wrong first. Leaving it in fp16/bf16 is the safe default. Quantizing it too (FP8 KV cache, which vLLM supports as a runtime flag) buys you more context length per GPU but the accuracy risk compounds with weight quantization error in ways that are harder to predict from perplexity alone, since KV cache precision affects every subsequent token's attention pattern, not just the layer it's computed in. Treat it as its own experiment with its own eval, not a rider on the weight-quantization decision.

What breaks: outliers, evals, and the layers you shouldn't touch#

The single biggest source of visible quality regression in 4-bit Llama quantization is under-protected outlier channels, usually concentrated in a handful of MLP down-projection layers and occasionally in early attention layers. AWQ exists specifically because round-to-nearest quantization treats every channel the same, and a small number of channels with activation magnitudes an order of magnitude above the rest will dominate the quantization error budget if you let them share a scale with everything else.

A few failure modes that show up consistently:

Calibration/production distribution mismatch. A calibration set that doesn't resemble production traffic gives you a checkpoint that looks fine on perplexity benchmarks and degrades on your actual workload, especially for structured outputs, code, or long-context retrieval where a small number of tokens carry most of the semantic weight.
Embedding and lm_head left unquantized, but not verified. Most pipelines skip quantizing these by default, but if you're using a custom quantization script it's worth confirming, since they're a common place for silent regressions if someone quantizes 'everything' for a marginal memory win.
Perplexity looking fine while task accuracy drops. Perplexity is a smooth, aggregate metric; it can look nearly identical to the fp16 baseline while multi-step reasoning, tool-call formatting, or long-context retrieval degrade measurably. Any quantization rollout needs its own task-specific eval suite, not just a perplexity delta on a held-out set.
Group size chosen for storage, not accuracy. It's tempting to pick group size 256 or per-tensor scales to shrink the checkpoint further; on a 64B model that saves single-digit GB and can cost more in accuracy than it's worth, particularly on the layers with the highest outlier concentration.
Kernel/format mismatch at load time. GPTQ, AWQ, and GGUF checkpoints are not interchangeable; loading an AWQ checkpoint with a GPTQ-only kernel (or vice versa) either fails outright or, worse, silently falls back to a slow dequant path that erases the throughput gain you quantized for in the first place.

The operational lesson worth carrying forward: never ship a quantized 64B checkpoint on a perplexity number alone. Run the same task eval suite you'd run against a fine-tune, on the same held-out set, and diff it against the fp16 baseline before it goes anywhere near production traffic.

Key takeaways#

A 64B Llama model costs about 128 GB in fp16, about 64 GB in int8, and about 32-36 GB in 4-bit including group-scale overhead, which is the difference between multi-GPU tensor parallel and a single 40-48 GB GPU.
GPTQ corrects quantization error after the fact using a Hessian-based per-layer reconstruction; AWQ prevents it up front by scaling salient channels before quantizing. Both outperform naive round-to-nearest at 4 bits.
Group size 128 with asymmetric quantization is the practical default for 65-70B-class dense models; smaller groups buy marginal accuracy at real storage cost, larger groups save little and cost more accuracy.
A quantized checkpoint is useless without a fused dequant kernel like Marlin or ExLlamaV2; loading it with a naive dequant-then-matmul path can be slower than running the fp16 model.
KV cache quantization is a separate decision from weight quantization with its own accuracy risk; don't bundle it into the same rollout without a dedicated eval.
Perplexity deltas hide task-specific regressions. Always run the same downstream eval suite (reasoning, tool calls, long-context retrieval) against fp16 and quantized checkpoints before shipping.

FAQ#

Can a 64B Llama model run on a single GPU after quantization?#

Yes at 4-bit. Weights land around 32-36 GB depending on group size, which fits on a single 40 GB or 48 GB GPU with room left for KV cache, versus roughly 128 GB in fp16 which requires multi-GPU tensor parallel.

What's the practical difference between GPTQ and AWQ for a 64B model?#

GPTQ uses a Hessian-based per-layer reconstruction to compensate for quantization error after it's introduced; AWQ scales salient weight channels before quantizing so they see less error in the first place. AWQ is cheaper to compute since it skips the iterative reconstruction, and its paper reports better accuracy than GPTQ at 3-4 bits on several model families.

Does 4-bit quantization noticeably hurt Llama output quality?#

Perplexity degradation from GPTQ or AWQ at 4-bit is typically small on published benchmarks for 65-70B-class models, but perplexity is an aggregate metric and can mask regressions in task-specific behavior like multi-step reasoning or structured output formatting, so a dedicated eval suite is necessary before trusting a quantized checkpoint in production.

Should the KV cache be quantized along with the model weights?#

Not automatically. KV cache precision affects every subsequent token's attention computation, so error compounds differently than weight quantization error. FP8 KV cache (supported by serving engines like vLLM) can extend usable context length per GPU, but treat it as a separate experiment with its own eval, not a default paired with weight quantization.

What is GGUF and when should you use it instead of GPTQ or AWQ?#

GGUF is the model file format used by llama.cpp, paired with k-quant schemes like Q4_K_M that use finer-grained per-block scales. Use it when you need CPU or hybrid CPU/GPU inference outside a CUDA-only stack; use GPTQ or AWQ when serving purely on GPU with a kernel like Marlin or ExLlamaV2.