Can I read Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM on EtoBox?

Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM by Zhang, Luoming; Fei, Wen; Wu, Weijia; He, Yefei; Lou, Zhenyu; Zhou, Hong is a scholarly article available to read on EtoBox.

What is Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM about?

Large Language Models (LLMs) pose significant hardware challenges related to memory requirements and computational ability. There are two mainstream quantization schemes for LLMs: coarse-grained ($\textit{e.g.,}$ channel-wise) quantization and fine-grained ($\textit{e.g.,}$ group-wise) quantization. Fine-grained quantization has smaller quantization loss, consequently achieving superior performance. However, when applied to weight-activation quantization, it disrupts continuous integer matrix multiplication, leading to inefficient inference. In this paper, we introduce Dual Grained Quantization (DGQ), a novel A8W4 quantization for LLM that maintains superior performance while ensuring fast inference speed. DSQ dequantizes the fine-grained INT4 weight into coarse-grained INT8 representation and preform matrix multiplication using INT8 kernels. Besides, we develop a two-phase grid search algorithm to simplify the determination of fine-grained and coarse-grained quantization scales. We also devise a percentile clipping schema for smoothing the activation outliers without the need for complex optimization techniques. Experimental results demonstrate that DGQ consistently outperforms pr

Author

Zhang, Luoming; Fei, Wen; Wu, Weijia; He, Yefei; Lou, Zhenyu; Zhou, Hong

Published

2023

Language

EN