Can I read LMFusion: Adapting Pretrained Language Models for Multimodal Generation on EtoBox?

LMFusion: Adapting Pretrained Language Models for Multimodal Generation by Shi, Weijia; Han, Xiaochuang; Zhou, Chunting; Liang, Weixin; Lin, Xi Victoria; Zettlemoyer, Luke; Yu, Lili is a scholarly article available to read on EtoBox.

What is LMFusion: Adapting Pretrained Language Models for Multimodal Generation about?

We present LMFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them to understand and generate both text and images in arbitrary sequences. LMFusion leverages existing Llama-3's weights for processing texts autoregressively while introducing additional and parallel transformer modules for processing images with diffusion. During training, the data from each modality is routed to its dedicated modules: modality-specific feedforward layers, query-key-value projections, and normalization layers process each modality independently, while the shared self-attention layers allow interactions across text and image features. By freezing the text-specific modules and only training the image-specific modules, LMFusion preserves the language capabilities of text-only LLMs while developing strong visual understanding and generation abilities. Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that, LMFusion improves image understanding by 20% and image generation by 3.6% using only 50% of the FLOPs while maintaining Llama-3's language capabilities. We also d

Author

Shi, Weijia; Han, Xiaochuang; Zhou, Chunting; Liang, Weixin; Lin, Xi Victoria; Zettlemoyer, Luke; Yu, Lili

Published

2024

Language

EN