Can I read GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments on EtoBox?

GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments by Chen, Yanyu; Huang, Ganhong is a scholarly article available to read on EtoBox.

What is GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments about?

Efficiently deploying large language models (LLMs) in real-world scenarios remains a critical challenge, primarily due to hardware heterogeneity, inference framework limitations, and workload complexities.Efficiently deploying large language models (LLMs) in real-world scenarios remains a critical challenge, primarily due to hardware heterogeneity, inference framework limitations, and workload complexities. These challenges often lead to inefficiencies in memory utilization, latency, and throughput, hindering the effective deployment of LLMs, especially for non-experts. Through extensive experiments, we identify key performance bottlenecks, including sudden drops in memory utilization, latency fluctuations with varying batch sizes, and inefficiencies in multi-GPU configurations. These insights reveal a vast optimization space shaped by the intricate interplay of hardware, frameworks, and workload parameters. This underscores the need for a systematic approach to optimize LLM inference, motivating the design of our framework, GUIDE. GUIDE leverages dynamic modeling and simulation-based optimization to address these issues, achieving prediction errors between 9.9% and 42.3% for key m

Author

Chen, Yanyu; Huang, Ganhong

Published

2024

Language

EN