arXiv AI September 10, 2026 By Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela

KTO: Model Alignment as Prospect Theoretic Optimization

The paper introduces KTO, a new loss function that aligns large language models with human preferences by directly maximizing a prospect-theoretic utility function rather than log-likelihood of preferences. It builds on the idea that existing alignment objectives implicitly use human-aware losses (HALOs) that reflect biases such as loss aversion. KTO achieves performance comparable to or better than preference-based methods across 1B to 30B parameter models while only requiring a binary desirability signal, and it highlights that no single HALO is universally best, emphasizing the importance of choosing the right inductive biases for a given setting.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 2

The Representation-Rationalizability Tradeoff in Reward Learning

arXiv:2606. 00291v1 Announce Type: cross Abstract: In RLHF, each training example contains a prompt $x$ and two candidate responses $y,y'$, and annotators provide pairwise preferences between these responses.

By Jing Dong, Yaoliang Yu, Pascal Pourpart

rag reinforcement-learning

arXiv AI

Jul 7

Out-of-Distribution Generalization of Risk Aversion in Language Models

arXiv:2607. 02755v1 Announce Type: cross Abstract: Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned.

By Kristina Zhang, Junior Chinomso Okoroafor, Benjamin Maltbie, Andrew Lin, Abhitej Bokka, Elliott Thornley

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 24

Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner

arXiv:2604. 18239v4 Announce Type: replace-cross Abstract: Preference optimization is widely used to align large language models (LLMs) with human preferences.

By Wei Chen, Yubing Wu, Junmei Yang, Delu Zeng, Qibin Zhao, John Paisley, Min Chen, Zhou Wang

llms

arXiv AI

Jun 24

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

arXiv:2606. 22974v2 Announce Type: replace Abstract: Recent work on preference elicitation in large language models (LLMs) has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure.

By Yujun Zhou, Christopher M. Ackerman

llms safety

arXiv AI

Jun 11

Autoregressive Direct Preference Optimization

arXiv:2602. 09533v2 Announce Type: replace Abstract: Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences.

By Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, Nakamasa Inoue

llms reinforcement-learning

arXiv AI

Jul 14

Reproducing human biases in route choice using large language models: Toward scalable behavioral modeling

arXiv:2607. 11632v1 Announce Type: new Abstract: Human choice behavior, including route choice, exhibits systematic behavioral biases that deviate from the assumptions of full rationality.

By Jiangtao Han, Shoufeng Ma, Shuxian Xu, Geng Li, Shuai Ling, Ning Jia, Zhengbing He

llms agents

One email a morning, machine-written

Related stories

The Representation-Rationalizability Tradeoff in Reward Learning

Out-of-Distribution Generalization of Risk Aversion in Language Models

Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

Autoregressive Direct Preference Optimization

Reproducing human biases in route choice using large language models: Toward scalable behavioral modeling