arXiv AI
Jul 23
Rewarding Better Thinking for LLM Preference Alignment
arXiv:2607. 19824v1 Announce Type: new Abstract: LLM preference alignment aims to optimize models toward human preferences across diverse user instructions.
Multi-LLM Collaborative Alignment via Stackelberg Games
The paper introduces Stackelberg Alignment, a leader‑follower framework that lets a pool of language models collaborate and improve by learning from each other’s responses. An EXP3 bandit leader adaptively selects instructions based on difficulty and discriminability, while the models act as followers, evaluating peers and learning via DPO or GRPO with Elo‑style reputation weighting and opponent matching. Experiments on diverse benchmarks show that this adaptive curriculum outperforms static baselines by up to 12‑25% and improves multi‑LLM evolution.
Related stories
arXiv Computation and Language
Sep 1
Small Reward Models via Backward Inference
arXiv:2602.13551v3 Announce Type: replace Abstract: Reward models (RMs) play a central role throughout the language model (LM) pipeline, particularly in non-verifiable domains. However, the dominant...
arXiv AI
Jun 19
AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Models
arXiv:2509. 25148v2 Announce Type: replace Abstract: Post-training alignment of large language models often combines supervised fine-tuning (SFT) on expert demonstrations with reinforcement learning (RL) from preference or verifiable feedback.
arXiv AI
Aug 12
Multiplayer Nash Preference Optimization
arXiv:2509. 23102v4 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences.
arXiv AI
Jul 3
Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
arXiv:2607. 01612v1 Announce Type: new Abstract: Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks.
arXiv Machine Learning
Sep 23
PACT: From Credit Assignment to Critic Alignment
arXiv:2609.26355v1 Announce Type: new Abstract: Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted ma...
arXiv Machine Learning
Jun 2
Enhancing LLM Metacognition via Cognitive Pairwise Training
arXiv:2606. 00869v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become central to LLM reasoning, but its outcome-level rewards can make models more willing to give confident answers when evidence or reasoning is unreliable.
arXiv AI
Jun 26
NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research
arXiv:2606. 26671v1 Announce Type: new Abstract: Post-training alignment determines the reasoning and human preference following capabilities of large language models, yet most existing works withhold detailed data construction, filtering rules and training recipes, which hinders community reproducibility and lightweight model optimization.
arXiv Machine Learning
Aug 18
Towards a theory of inference-time alignment with unknown rewards
arXiv:2608. 15402v1 Announce Type: new Abstract: Generative model alignment has received broad interest, and significant progress has been made in supervised fine-tuning and inference-time computation.
arXiv AI
Jun 9
Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
arXiv:2602. 02572v2 Announce Type: replace-cross Abstract: Existing alignment methods directly use the reward model learned from user preference data to optimize an LLM policy, subject to KL regularization with respect to the base policy.
arXiv AI
Jun 4
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
arXiv:2606. 04807v1 Announce Type: new Abstract: Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape.
arXiv AI
Jun 2
SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
arXiv:2604. 10688v2 Announce Type: replace-cross Abstract: On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult.