Can I read Krony-PT: GPT2 compressed with Kronecker Products on EtoBox?

Krony-PT: GPT2 compressed with Kronecker Products by Ayad, M. Ayoub Ben; Mitrovic, Jelena; Granitzer, Michael is a scholarly article available to read on EtoBox.

What is Krony-PT: GPT2 compressed with Kronecker Products about?

We introduce Krony-PT, a compression technique of GPT2 \citep{radford2019language} based on Kronecker Products. We specifically target the MLP layers of each transformer layer, and systematically compress the feed forward layer matrices to various degrees. We introduce a modified Van Loan decomposition to initialize the new factors, and also introduce a new pruning-based initialization trick. Our method compresses the original 124M parameter GPT2 to various smaller models, with 80M being the smallest, and 96M being the largest compressed model. Our 81M model variant outperforms distilgpt2 on next-token prediction on all standard language modeling datasets, and shows competitive scores or performs on par with other Kronecker Products based compressed models of GPT2 that are significantly higher in size.

Author

Ayad, M. Ayoub Ben; Mitrovic, Jelena; Granitzer, Michael

Published

2024

Language

EN