arXiv Machine Learning

A Probabilistic Approach for Model Alignment with Human Comparisons

The paper proposes a two‑stage framework, SL+LHF, that first learns low‑dimensional representations from noisy labeled data and then refines model alignment using human comparison feedback via a probabilistic bisection approach. It introduces the label‑noise‑to‑comparison‑accuracy (LNCA) ratio to theoretically identify when this framework outperforms pure supervised learning, showing that trading labels for comparisons reduces sample complexity when labels are scarce. Experiments on a high‑dimensional crowdfunding prediction task and an Amazon Mechanical Turk study confirm that incorporating human or large language model evaluators improves accuracy under a fixed query budget.

arXiv Computation and Language

Sep 2

Post-hoc Alignment of LLM-judges to Human Judgment Distribution

The paper introduces NAPHA, a lightweight post‑hoc alignment method that improves large language model (LLM) predictions of human judgment distributions (HJD) by matching LLM output distributions to HJD through entropy‑based class assignment and specialized alignment models. Experiments on five datasets show that while LLMs perform near human‑level on hard‑label tasks, they struggle with soft‑label predictions, and NAPHA consistently enhances soft‑label accuracy, especially on high‑entropy instances. The study also demonstrates that better entropy class prediction can further boost NAPHA’s effectiveness.

By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani

llms safety

arXiv Machine Learning

Jul 15

Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints for Reliable Decision-Making

arXiv:2602. 07008v3 Announce Type: replace-cross Abstract: Reliable models should not only predict correctly, but also justify decisions with acceptable evidence.

By Ruoyu Chen, Shangquan Sun, Xiaoqing Guo, Sanyi Zhang, Kangwei Liu, Shiming Liu, Zhangcheng Wang, Qunli Zhang, Wei Wang, Hua Zhang, Xiaochun Cao

agents computer-vision safety

arXiv Machine Learning

Jul 2

REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations

arXiv:2604. 17289v2 Announce Type: replace Abstract: Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise.

By Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, Ramtin Pedarsani

llms nlp fine-tuning benchmarks safety

arXiv AI

Jul 28

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.

By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

llms reinforcement-learning benchmarks safety data-engineering

arXiv Computer Vision

Aug 25

CBX-Bench: A Human-Aligned MLLM Council for Benchmarking Concept Bottleneck Model Explanations

arXiv:2608.15404v2 Announce Type: replace Abstract: Concept Bottleneck Models (CBMs) are designed to make visual classification interpretable by expressing predictions through human-understandable co...

By Yusuf Meric Karadag, Gulay Oklan, Seref Baris Cagliyan, Umut Ozdemir, Emre Akbas

llms multimodal benchmarks safety

arXiv AI

Jul 7

Aligning Language Models with Selective Prediction

arXiv:2607. 03528v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as critical decision-making components in high-stakes real-world AI systems, rendering LLM reliability a foremost practical concern.

By Gaoxiang Luo, Yifan Wu, Sinian Zhang, Aryan Deshwal, Ju Sun

llms reinforcement-learning safety

arXiv AI

Jun 11

GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human

arXiv:2605. 28882v2 Announce Type: replace-cross Abstract: With the rapid advancement of large language models, evaluating human-likeness in open-ended conversation has become increasingly important.

By Yihang Lin, Yunze Gao, Zeyang Lin, Dongbo Li, Kun Peng, Yue Liu

llms agents benchmarks safety

arXiv AI

Aug 12

Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons

arXiv:2608. 10045v1 Announce Type: cross Abstract: The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models.

By Kaustubh Shivshankar Shejole, Tanish Agarwal, Arpit Agarwal, Avishek Ghosh

llms fine-tuning safety

arXiv AI

2d ago

PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

PADM'E is a method for synthesizing preference‑aligned data to meta‑evaluate language‑model (LM) evaluators of agentic behaviors. It reframes meta‑evaluation as a preference judgment problem, generating criterion‑based data with small LMs and no human involvement. In a prototype, PADM'E produced 1,000 samples across four domains and three criteria, and human validation showed agreement with human judgment rising from 73% to 85% compared to a naive baseline.

By Cheng Chang, Yining Mao, Peng Qi

llms agents safety

arXiv AI

Aug 18

Leveraging Machine Unlearning for Cost-Efficient Preference Alignment

arXiv:2504. 06659v2 Announce Type: replace-cross Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges.

By Xiaohua Feng, Yuyuan Li, Huwei Ji, Jiaming Zhang, Li Zhang, Tianyu Du, Chaochao Chen

llms reinforcement-learning safety

arXiv AI

Jun 9

Explaining Black-Box Language Models: Learning to Optimize Linguistically-Structured Word Subsets

arXiv:2606. 08497v1 Announce Type: new Abstract: As deep language models (DLMs) are increasingly deployed in high-stakes domains such as healthcare, understanding their decision rationale becomes paramount for ensuring trust, safety, and accountability.

By Minyoung Hwang, Seokhyun Lee, Changhee Lee

llms benchmarks safety

arXiv Machine Learning

Jun 5

SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs

arXiv:2606. 05376v1 Announce Type: new Abstract: Many human-centered tasks, including natural language inference (NLI) and emotion recognition (ER), have multiple plausible interpretations, leading to label ambiguity and challenging disagreements across human annotators.

By Jingyao Wu, Ashley Wang, Keane Ong, Paul Pu Liang, Rosalind Picard

llms reinforcement-learning benchmarks safety

Related stories