arXiv AI
Language Chain in Alignment: Cross-Lingual Ranking Preference Optimization
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 28
Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
The paper introduces Cross‑lingual Ranking Preference Optimization (CRPO), a framework that uses high‑quality English preference data to improve alignment of large language models in other languages. CRPO builds a hierarchical structure over parallel preference pairs, jointly optimizing intra‑ and inter‑lingual preferences and providing a relative ranking signal beyond binary comparisons. Experiments on five languages show consistent gains in instruction‑following and knowledge utilization, with robust performance across different weighting schemes and improved reward margins and log‑probabilities of desirable responses.
arXiv AI
Sep 3
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
CroCo introduces cross‑lingual contrastive preference tuning on self‑generations, extending prior English‑only methods to 14 high‑ and low‑resource languages. A reward model trained solely on English preferences, applied to a multilingual base, yields effective within‑language rankings and improves performance in both monolingual and multilingual settings without catastrophic forgetting. The approach requires on‑policy data; off‑policy responses and online preference optimization offer limited gains, yet on structured tasks CroCo matches or surpasses the base model in most languages, and on open‑ended generation it wins 28/30 judge evaluations across 15 languages.
Hugging Face Trending Papers
Aug 25
Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilit...
arXiv AI
Jun 30
Aligning Language Model Benchmarks with Pairwise Preferences
arXiv:2602. 02898v3 Announce Type: replace Abstract: Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance.
arXiv AI
Aug 26
Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
The paper introduces BALIGN, a balanced data selection strategy designed to reduce catastrophic forgetting—referred to as the alignment tax—in large language models during preference-based alignment. By analyzing preference optimization gradients, the authors identify three data-centric features that influence parameter drift: the reference model's log-probability margin, token length differences between chosen and rejected responses, and TF‑IDF similarity to general capability corpora. BALIGN aggregates these features into a composite risk score to filter out high-risk preference samples, thereby preserving foundational capabilities while maintaining alignment gains with minimal computational overhead.
arXiv Machine Learning
Jun 30
REAR: Test-time Preference Realignment through Reward Decomposition
arXiv:2606. 30339v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task.