Can I read ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion on EtoBox?

ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion by Khan, Rana Muhammad Shahroz; Tang, Dongwen; Li, Pingzhi; Wang, Kai; Chen, Tianlong is a scholarly article available to read on EtoBox.

What is ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion about?

Parameter generation has emerged as a novel paradigm for neural network development, offering an alternative to traditional neural network training by synthesizing high-quality model weights directly. In the context of Low-Rank Adaptation (LoRA) for evolving ($\textit{i.e.}$, constantly updated) large language models (LLMs), this approach promises efficient adaptation without costly retraining. However, existing methods face critical limitations in simultaneously achieving scalability and controllability. In this paper, we introduce $\texttt{ORAL}$, a novel $\textbf{conditional recurrent diffusion}$ framework that addresses these challenges. $\texttt{ORAL}$ incorporates a novel conditioning mechanism that integrates model architecture and textual task specifications, enabling the generation of task-specific LoRA parameters that can seamlessly transfer across evolving foundation models. Our approach successfully scales to billions-of-parameter LLMs and maintains controllability. Through extensive experiments across seven language tasks, four vision tasks, and three multimodal tasks using five pre-trained LLMs, we demonstrate that $\texttt{ORAL}$ generates high-quality LoRA parameter

Author

Khan, Rana Muhammad Shahroz; Tang, Dongwen; Li, Pingzhi; Wang, Kai; Chen, Tianlong

Published

2025

Language

EN