Can I read Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs on EtoBox?

Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs by Gope, Dibakar; Mansell, David; Loh, Danny; Bratt, Ian is a scholarly article available to read on EtoBox.

What is Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs about?

Large language models (LLMs) have transformed the way we think about language understanding and generation, enthralling both researchers and developers. However, deploying LLMs for inference has been a significant challenge due to their unprecedented size and resource requirements. While quantizing model weights to sub-byte precision has emerged as a promising solution to ease memory pressure, the group quantization formats commonly used for LLM quantization have significant compute overheads and a resource-intensive dequantization process. As a result, a higher proportion of compute instructions do not perform multiplies, i.e., real work, rendering them unsuitable for meeting the required latency requirements for LLMs deployed on commodity CPUs. In this work, we propose a set of highly optimized kernels to accelerate LLM inference and unleash the full potential of CPUs, particularly Arm CPUs. These kernels amortize the cost of loading the operands and the cost of weight unpacking across multiple output rows. This, along with the introduction of an optimized interleaved group data layout for weights and decompression path optimizations to reduce unnecessary operations and dequantiz

Author

Gope, Dibakar; Mansell, David; Loh, Danny; Bratt, Ian

Published

2024

Language

EN