Can I read 2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization on EtoBox?

2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization by Li, Mengyang; Zhang, Zhong is a scholarly article available to read on EtoBox.

What is 2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization about?

Aligning large language models with human preferences is crucial for their safe deployment. While Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning from human feedback, traditional DPO methods are limited by their reliance on single preference pairs. Recent work like Curriculum-DPO integrates multiple pairs using a one-dimensional difficulty curriculum based on pairwise distinguishability (PD), but overlooks the complexity of the input prompt itself. To address this, we propose 2D-Curri-DPO, a novel framework employing a two-dimensional curriculum that jointly models Prompt Complexity (PC) and Pairwise Distinguishability. This framework introduces dual difficulty metrics to quantify prompt semantic complexity and response preference clarity, defines a curriculum strategy space encompassing multiple selectable strategies for task adaptation, and incorporates a KL-divergence-based adaptive mechanism for dynamic reference model updates to enhance training stability. Comprehensive experiments demonstrate that 2D-Curri-DPO significantly outperforms standard DPO and prior curriculum methods across multiple benchmarks, including MT-Bench, Vicun

Author

Li, Mengyang; Zhang, Zhong

Published

2025

Language

EN