Can I read Disaggregated LLM Inference Optimization on EtoBox?
Disaggregated LLM Inference Optimization by pyryezor is a document available to read on EtoBox.
What is Disaggregated LLM Inference Optimization about?
The document discusses interference issues that can occur when running large language model inference requests of different types and lengths together on shared hardware. It presents TetriInfer, a system that aims to minimize such interference by carefully scheduling and grouping requests based on their characteristics such as input prompt length and generated token length. It partitions prompts, separates prefill and decode phases, and uses a two-level scheduling algorithm to improve metrics like latency,
- Author
- pyryezor
- Language
- EN