Skip to content

Opening book details…

Can I read Extreme Compression of Sentence-transformer Ranker Models: Faster Inference, Longer Battery Life, and Less Storage on Edge Devices on EtoBox?

Extreme Compression of Sentence-transformer Ranker Models: Faster Inference, Longer Battery Life, and Less Storage on Edge Devices by Chaulwar, Amit; Malik, Lukas; Krajewski, Maciej; Reichel, Felix; Lundbæk, Leif-Nissen; Huth, Michael; Matejczyk, Bartlomiej is a scholarly article available to read on EtoBox.

What is Extreme Compression of Sentence-transformer Ranker Models: Faster Inference, Longer Battery Life, and Less Storage on Edge Devices about?

Modern search systems use several large ranker models with transformer architectures. These models require large computational resources and are not suitable for usage on devices with limited computational resources. Knowledge distillation is a popular compression technique that can reduce the resource needs of such models, where a large teacher model transfers knowledge to a small student model. To drastically reduce memory requirements and energy consumption, we propose two extensions for a popular sentence-transformer distillation procedure: generation of an optimal size vocabulary and dimensionality reduction of the embedding dimension of teachers prior to distillation. We evaluate these extensions on two different types of ranker models. This results in extremely compressed student models whose analysis on a test dataset shows the significance and utility of our proposed extensions.

Author
Chaulwar, Amit; Malik, Lukas; Krajewski, Maciej; Reichel, Felix; Lundbæk, Leif-Nissen; Huth, Michael; Matejczyk, Bartlomiej
Published
2022
Language
EN

More by Chaulwar, Amit; Malik, Lukas; Krajewski, Maciej; Reichel, Felix; Lundbæk, Leif-Nissen; Huth, Michael; Matejczyk, Bartlomiej

Browse all works by Chaulwar, Amit; Malik, Lukas; Krajewski, Maciej; Reichel, Felix; Lundbæk, Leif-Nissen; Huth, Michael; Matejczyk, Bartlomiej