arXiv Machine Learning

Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users

arXiv:2606. 20482v1 Announce Type: cross Abstract: To align a Large Language Model (LLM), most existing methods collect explicit human feedback and train a reward model to predict the human preference based on the response text.

arXiv AI

Jun 8

Re-Centering Humans in LLM Personalization

arXiv:2606. 06614v1 Announce Type: cross Abstract: Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data.

By Lechen Zhang, Jiarui Liu, Tal August

llms

arXiv AI

Jul 3

HAL: Inducing Human-likeness in LLMs with Alignment

arXiv:2601. 02813v3 Announce Type: replace Abstract: Aligning language models to qualitative behavioral traits, such as human-likeness, remains difficult because they are hard to define, measure, and optimize.

By Masum Hasan, Junjie Zhao, Ehsan Hoque

llms safety

arXiv AI

6d ago

DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

The paper introduces DIAL, a framework that uses large language models (LLMs) as judges while mitigating position bias and aligning their judgments with human preferences. DIAL separates judge‑specific position effects, learns shared structure in debiased LLM preferences, and adaptively calibrates this structure toward human targets using limited human comparisons. Experiments on simulations and three human‑preference benchmarks show that DIAL remains robust to unbalanced response order, achieves strong human‑aligned rankings with few labels, and adapts when LLM information is imperfect, supported by a real‑data study of over 410K judgments from 21 LLM judges.

By Zesheng Cai, Yingqi Fan, Sichang Chen, Jin-Hong Du

llms benchmarks safety

arXiv AI

Aug 12

Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes

arXiv:2608. 10209v1 Announce Type: new Abstract: Feedback signals used to train Large Language Models (LLMs) are the primary driver of their behavior and our main lever for instilling alignment with human values and objectives.

By Alec Harris, Kasey Corra, Archie Chaudhury, Yixiong Hao

llms reinforcement-learning fine-tuning safety

arXiv AI

Jul 20

Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling

arXiv:2607. 15740v1 Announce Type: cross Abstract: As Text-to-Image (T2I) systems rapidly advance, evaluating the cultural authenticity of synthesized content has become increasingly important for fair and trustworthy generative AI.

By Bo-An Chang, Yu-Chih Chen

llms diffusion nlp reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jul 28

What do Reward Models Memorize?

arXiv:2607. 24484v1 Announce Type: new Abstract: This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets.

By Ivo Verhoeven, Pushkar Mishra, Ekaterina Shutova

arXiv AI

Jul 7

Attention Limited Reward Learning

arXiv:2607. 04590v1 Announce Type: new Abstract: Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences.

By Wenqian Xing

reinforcement-learning safety

arXiv Computation and Language

Sep 10

HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization

HyperTrace is a training‑free framework that personalizes large language models by tracing latent user preferences online. It maintains interpretable natural‑language hypotheses about short‑term intent and long‑term preferences, updating them with an SMC‑style reweighting process driven by an LLM‑based surrogate choice model. Experiments on PRISM and PersonaMem‑v2 demonstrate that HyperTrace improves response alignment, preference prediction, and profile consistency compared to strong online baselines.

By Jianzhi Shen, Keyu Mao, Minghao Shao, Chuanyang Jin, Yusong Wang, Ailiang Lin, Kotaro Funakoshi, Manabu Okumura, Tianmin Shu, Muhammad Shafique

llms safety

arXiv Machine Learning

Jul 31

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.

By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao

llms reinforcement-learning benchmarks safety

Hugging Face Trending Papers

Jun 28

Cognitive World Models for Process-Level Social Influence Evaluation

Social influence dialogue changes user behavior by altering internal cognitive states. The central evaluation question is whether the user's beliefs, desires, intentions, and emotions measurably change over the course of conversation, a process-oriented criterion that neither surface-level text metrics (BLEU/ROUGE) nor single-score LLM judgments can capture.

llms agents

arXiv Computation and Language

4d ago

Evaluating Alignment of Behavioral Dispositions in LLMs

arXiv:2602.11328v2 Announce Type: replace Abstract: As people turn to LLMs for social advice, understanding their behavior in such contexts becomes essential. In this work, we focus on behavioral dis...

By Amir Taubenfeld, Zorik Gekhman, Lior Nezry, Omri Feldman, Natalie Harris, Shashir Reddy, Romina Stella, Ariel Goldstein, Marian Croak, Yossi Matias, Amir Feder

llms safety

Hugging Face Trending Papers

Jul 27

What do Reward Models Memorize?

This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets. We show that RMs 1) misallocate memorization to easy, high margin preference pairs, 2) memorize dataset-specific shortcuts (e.

Related stories