Can I read M E-I: Efficient Moe Inference On Personal Machines With Sparsity-Aware Expert Cache on EtoBox?

M E-I: Efficient Moe Inference On Personal Machines With Sparsity-Aware Expert Cache by Pratham Dave is a document available to read on EtoBox.

What is M E-I: Efficient Moe Inference On Personal Machines With Sparsity-Aware Expert Cache about?

The document introduces M O E-I NFINITY, an efficient MoE inference system tailored for personal machines with limited GPU memory, achieving 3.1–16.7× latency reductions compared to existing systems. It utilizes a sparsity-aware expert cache that optimizes expert selection based on activation patterns during inference, addressing the inefficiencies of current caching methods. The system

Author

Pratham Dave

Language

EN