Can I read SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models on EtoBox?

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models by Huang, Wei; Qin, Haotong; Liu, Yangdong; Li, Yawei; Liu, Xianglong; Benini, Luca; Magno, Michele; Qi, Xiaojuan is a scholarly article available to read on EtoBox.

What is SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models about?

Large language models (LLMs) achieve remarkable performance in natural language understanding but require substantial computation and memory resources. Post-training quantization (PTQ) is a powerful compression technique extensively investigated in LLMs. However, existing PTQ methods are still not ideal in terms of accuracy and efficiency, especially with below 4 bit-widths. Standard PTQ methods using group-wise quantization suffer difficulties in quantizing LLMs accurately to such low-bit, but advanced methods remaining high-precision weights element-wisely are hard to realize their theoretical hardware efficiency. This paper presents a Salience-Driven Mixed-Precision Quantization scheme for LLMs, namely SliM-LLM. The scheme exploits the salience distribution of weights to determine optimal bit-width and quantizers for accurate LLM quantization, while aligning bit-width partition to groups for compact memory usage and fast integer inference. Specifically, the proposed SliM-LLM mainly relies on two novel techniques: (1) Salience-Determined Bit Allocation utilizes the clustering characteristics of salience distribution to allocate the bit-widths of each group, increasing the accurac

Author

Huang, Wei; Qin, Haotong; Liu, Yangdong; Li, Yawei; Liu, Xianglong; Benini, Luca; Magno, Michele; Qi, Xiaojuan

Published

2024

Language

EN