Can I read SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading on EtoBox?

SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading by Chen, Qiaoling; Li, Shenggui; Gao, Wei; Sun, Peng; Wen, Yonggang; Zhang, Tianwei is a scholarly article available to read on EtoBox.

What is SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading about?

In recent years, Large Language Models (LLMs) have exhibited remarkable capabilities, driving advancements in real-world applications. However, training LLMs on increasingly long input sequences imposes significant challenges due to high GPU memory and computational demands. Existing solutions face two key limitations: (1) memory reduction techniques, such as activation recomputation and CPU offloading, compromise training efficiency; (2) distributed parallelism strategies require excessive GPU resources, limiting the scalability of input sequence length. To address these gaps, we propose Adaptive Sequence Pipeline Parallel Offloading (SPPO), a novel LLM training framework that optimizes memory and computational resource efficiency for long-sequence training. SPPO introduces adaptive offloading, leveraging sequence-aware offloading, and two-level activation management to reduce GPU memory consumption without degrading the training efficiency. Additionally, SPPO develops an adaptive pipeline scheduling approach with a heuristic solver and multiplexed sequence partitioning to improve computational resource efficiency. Experimental results demonstrate that SPPO achieves up to 3.38x th

Author

Chen, Qiaoling; Li, Shenggui; Gao, Wei; Sun, Peng; Wen, Yonggang; Zhang, Tianwei

Published

2025

Language

EN