arXiv AI August 28, 2026 By Seungyoon Lee, Minhyuk Kim, Jungseob Lee, Heuiseok Lim

Language Chain in Alignment: Cross-lingual Ranking Preference Optimization

The paper introduces Cross‑lingual Ranking Preference Optimization (CRPO), a framework that uses high‑quality English preference data to improve alignment of large language models in other languages. CRPO builds a hierarchical structure over parallel preference pairs, jointly optimizing intra‑ and inter‑lingual preferences and providing a relative ranking signal beyond binary comparisons. Experiments on five languages show consistent gains in instruction‑following and knowledge utilization, with robust performance across different weighting schemes and improved reward margins and log‑probabilities of desirable responses.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 25

Language Chain in Alignment: Cross-Lingual Ranking Preference Optimization

arXiv:2608.23149v1 Announce Type: cross Abstract: The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in...

By Seungyoon Lee, Minhyuk Kim, Jungseob Lee, Heuiseok Lim

llms safety

arXiv AI

Sep 3

CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

CroCo introduces cross‑lingual contrastive preference tuning on self‑generations, extending prior English‑only methods to 14 high‑ and low‑resource languages. A reward model trained solely on English preferences, applied to a multilingual base, yields effective within‑language rankings and improves performance in both monolingual and multilingual settings without catastrophic forgetting. The approach requires on‑policy data; off‑policy responses and online preference optimization offer limited gains, yet on structured tasks CroCo matches or surpasses the base model in most languages, and on open‑ended generation it wins 28/30 judge evaluations across 15 languages.

By Mike Zhang, Ali Basirat, Desmond Elliott

llms reinforcement-learning fine-tuning

arXiv AI

Jun 30

Aligning Language Model Benchmarks with Pairwise Preferences

arXiv:2602. 02898v3 Announce Type: replace Abstract: Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance.

By Marco Gutierrez, Xinyi Leng, Hannah Cyberey, Jonathan Richard Schwarz, Ahmed Alaa, Thomas Hartvigsen

llms benchmarks safety

Hugging Face Trending Papers

Aug 25

Preference Data Selection for Mitigating the Alignment Tax in Large Language Models

Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilit...

llms safety

arXiv AI

Jun 11

Toward Preference-aligned Large Language Models via Residual-based Model Steering

arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.

By Lucio La Cava, Andrea Tagarelli

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 30

REAR: Test-time Preference Realignment through Reward Decomposition

arXiv:2606. 30339v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task.

By Fuxiang Zhang, Pengcheng Wang, Chenran Li, Yi-Chen Li, Yuxin Chen, Lang Feng, Chenfeng Xu, Masayoshi Tomizuka, Bo An

llms safety

One email a morning, machine-written

Related stories

Language Chain in Alignment: Cross-Lingual Ranking Preference Optimization

CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

Aligning Language Model Benchmarks with Pairwise Preferences

Preference Data Selection for Mitigating the Alignment Tax in Large Language Models

Toward Preference-aligned Large Language Models via Residual-based Model Steering

REAR: Test-time Preference Realignment through Reward Decomposition