arXiv AI

You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs

The study shows that instruction‑tuned language models, when asked to generate responses with varying emotional intensity, consistently undershoot the requested affect. By conditioning a model on continuous Valence‑Arousal targets and measuring output with a frozen regressor, the authors find that the gain for valence is only 0.26 and for arousal 0.13 on Llama‑3.1‑8B, far below the ideal value of 1. They trace this undershoot to the preference‑learning pipeline: training data such as EmoBank are neutral‑heavy and the candidate pool rarely contains extreme affect, so Direct Preference Optimization lacks examples to learn from. Expanding the target space uniformly and sampling a hotter, larger candidate pool raises valence gain to 0.40 and improves extrapolation with minimal in‑distribution cost, a result that also holds for Qwen3‑8B. Arousal remains more variable because the base model rarely generates highly aroused candidates.

arXiv AI

Aug 24

VA-DPO: Valence-Arousal Direct Preference Optimization for Controllable Emotion Generation in Language Models

The paper introduces VA‑DPO, a method that trains language models to generate text with a specified continuous affect point in the Valence‑Arousal plane. By using a frozen VA regressor to score candidate generations and selecting pairs with a distance margin, the approach modifies Direct Preference Optimization to better hit target emotions. Experiments on Llama‑3.1‑8B‑Instruct show a 33% reduction in mean VA distance compared to system‑prompting and 25% over few‑shot prompting, while maintaining performance on benchmarks like MMLU, HellaSwag, and TruthfulQA.

By Hyunwoo Kim

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 29

Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization

arXiv:2607. 25136v1 Announce Type: new Abstract: Research on preference optimization often varies the training objective while holding the data fixed.

By Zhengtao Yao, Runhao Li, Xupeng Chen, Jiayi Cheng, Chenqian Le, Michael Yue, Siheng Wang, Haoyan Xu, Yuqi Li, Chenhao Wei, Zhengdao Li, Rongchao Zhang, Guang Yang, Yidong Wang, Junhao Dong

llms safety

arXiv AI

Jul 21

OR Else: A Differentiable Trust Region for Policy Optimization

arXiv:2607. 18163v1 Announce Type: cross Abstract: PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative.

By Chinmay Rane, Kanishka Tyagi, Michael Manry

llms reinforcement-learning

arXiv AI

Jun 29

When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model

arXiv:2606. 21641v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have been proposed as hyperparameter-optimization (HPO) advisors that "warm-start" search from prior knowledge, proposing strong configurations in very few evaluations.

By Carson Rodrigues, Oysturn Vas, Isaiah Abner DCosta, Nithish Kumar Prabhakaran

llms benchmarks

arXiv AI

Jul 28

Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models

arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?

By Tapan Parikh

llms rag benchmarks

arXiv AI

Jul 7

Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning

arXiv:2607. 04470v1 Announce Type: cross Abstract: Large Language Models (LLMs) offer a natural interface for translating human objectives into reward signals for cooperative multi-agent reinforcement learning (MARL), yet the training-time dynamics of this integration remain poorly understood.

By Faid Keddouri, Sohaib Houhou, Aissa Boulmerka, Nadir Farhi

llms agents reinforcement-learning

arXiv Machine Learning

Sep 22

A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation

The paper investigates selective on‑policy distillation, where a student model is trained only on token positions chosen by a selector. It demonstrates that the commonly used shared learning rate is not neutral: performance varies significantly with the learning rate for different selectors, leading to inconsistent comparisons. The authors attribute this selector‑rate entanglement to the selection process itself and recommend reporting the full arm‑by‑rate matrix for fair evaluation.

By Chencheng Zhu

fine-tuning efficiency benchmarks

Hugging Face Trending Papers

Jul 27

Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models

Appending a two-word confirmation tag to a decision question -- "Is X the better choice? " versus "X is the better choice, right?

llms rag benchmarks

arXiv AI

Jun 12

Boosting Direct Preference Optimization with Penalization

arXiv:2606. 12505v1 Announce Type: cross Abstract: Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset.

By Pengwei Sun

llms reinforcement-learning

arXiv AI

Jul 15

Sensitivity to Subjective Expected Utility Maximization: A Methodological Study, with an Illustrative Application to LLM Decision-Making

arXiv:2607. 11920v1 Announce Type: cross Abstract: Evaluating decisions made under uncertainty is hard when labeled outcomes are scarce, costly, or confounded with luck.

By Jeff Helzner

llms agents

arXiv Computation and Language

5d ago

Over-Personalization Is a Decision Failure: Generation-Induced Apply Bias in LLMs

arXiv:2609.34284v2 Announce Type: replace Abstract: Personalized LLMs must decide, for each stored preference, whether the current context calls for applying or suppressing it, which we call its appl...

By Haeun Jang, Yonghyun Jun, Hwanhee Lee

llms benchmarks safety

arXiv AI

Sep 10

ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR

arXiv:2609.09075v1 Announce Type: cross Abstract: In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term st...

By Tommy Sha, Skylar Zhai, Siqi Zhao

reinforcement-learning

Related stories