LLM Quantization: A Deep Dive into GPTQ, AWQ, and Post-Training Quantization

Large Language Models (LLMs) are massive, requiring billions of parameters to achieve state-of-the-art reasoning. Storing and serving these models is computationally expensive:

• A 70-billion parameter model stored in FP16 (16-bit Floating Point) requires 140 GB of VRAM just to load.
• This exceeds the capacity of standard consumer GPUs, requiring enterprise-grade hardware cluster setups.

Quantization reduces this memory requirement by converting model weights from high-precision formats (like FP16) to lower-precision formats (like INT4 or INT8).

LLM Weight Quantization (GPTQ / AWQ)

The Math of Quantization

Quantization maps a continuous set of float values x to a discrete set of integer values q. The linear quantization mapping formula is:

q = \text{round}\left( \frac{x}{S} \right) + Z

where S is the scale factor (a float value) and Z is the zero-point offset (an integer). The dequantization process back to float is:

x_{approx} = (q - Z) \cdot S

When quantizing an entire model, the goal is to minimize the reconstruction error of the network activations.

GPTQ vs. AWQ

There are two primary algorithms for Post-Training Quantization (PTQ) of LLM weights:

1. GPTQ (Generalized Post-Training Quantization):

   • Uses second-order information (inverse Hessian matrix) to adjust remaining weights after quantizing a specific weight row.
   • It performs row-by-row optimization, minimizing the squared error of layer activations.
   • Provides excellent accuracy for 4-bit quantization but can suffer from outlier activation errors.

2. AWQ (Activation-aware Weight Quantization):

   • Recognizes that not all weights are equally important. Only 1% of weights (salient weights) dictate the majority of the model's accuracy.
   • Protects these salient weights by keeping them in higher precision or scaling them up, while quantizing the remaining 99% of weights.
   • This selective approach maintains accuracy close to FP16 levels without needing complex inverse Hessian computations.

Quantization Formats Comparison

| Format | Precision | Target Hardware | Primary Use Case | |---|---|---|---| | FP16 | 16-bit Float | Nvidia H100 / A100 | Training & High-end Serving | | INT8 | 8-bit Integer | Standard Server GPUs | Mid-range deployment | | INT4 (GPTQ/AWQ) | 4-bit Integer | Consumer GPUs / Edge | Local deployment & Low cost |

Running a Quantized Model in Python using vLLM

Here is how you can initialize and serve an AWQ-quantized model using the vllm library:

# Initializing vLLM with an AWQ model
# from vllm import LLM, SamplingParams
# 
# llm = LLM(
#     model="TheBloke/Llama-2-7B-Chat-AWQ",
#     quantization="awq",
#     dtype="half"
# )

Conclusion

Quantization is a key technique for scaling LLM deployments. By shrinking models to INT4 via GPTQ or AWQ, developers can run large models on cheaper, consumer-grade hardware while maintaining high accuracy and performance.