arXiv Computation and Language October 16, 2026 By Junru Wu, Jingting Zheng, Yuqi Ren, Deyi Xiong

VALCON: Benchmarking and Enhancing Value Consistency in Large Language Models via Dual-Role Variation

Read the original on arXiv Computation and Language →

VALCON introduces a role-driven framework to benchmark and improve value consistency in large language models (LLMs) by evaluating how well an LLM’s generative outputs align with its evaluative judgments when the same model alternates between generator and evaluator roles. The framework creates three benchmark suites—VAB-Consistency, VAB-Boundary, and VAB-Challenge—using dual-role variation to expose hidden value discrepancies. Experiments across 28 LLMs reveal widespread cross-role inconsistency, and mechanistic probing identifies a role-dependent representational decoupling; a representation-directed intervention is then proposed to mitigate these failures, establishing value consistency as a distinct alignment dimension.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Oct 8

Latent Performance Profiling of Large Language Models

arXiv:2605.30018v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...

By Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya, Partha Pratim Chakrabarti, Amlan Chakrabarti, Supratik Chakraborty, Partha Pratim Das, Lipika Dey, Richa Singh, Mayank Vatsa

llms benchmarks safety

arXiv AI

Oct 9

Predicting Alignment Generalization with Value Representations

The paper introduces the task of predicting alignment generalization, which measures how fine‑tuning a language model to follow a specific value affects its behavior across a broad set of other values. Through a large‑scale study of 66 values, the authors show that representations derived from model activations when applying values in context outperform textual description‑based methods, achieving a correlation of 0.45 versus 0.05. They further demonstrate that these representations can predict downstream robustness and help construct a taxonomy of LLM values based on empirical generalization dynamics.

By Andy Liu, Mehar Bhatia, Karolina Stanczak, Mona Diab, Vered Shwartz, Daniel Fried

llms fine-tuning benchmarks safety

arXiv AI

Jul 28

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.

By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

llms reinforcement-learning benchmarks safety data-engineering

arXiv AI

Sep 30

PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

PADM'E is a method for synthesizing preference‑aligned data to meta‑evaluate language‑model (LM) evaluators of agentic behaviors. It reframes meta‑evaluation as a preference judgment problem, generating criterion‑based data with small LMs and no human involvement. In a prototype, PADM'E produced 1,000 samples across four domains and three criteria, and human validation showed agreement with human judgment rising from 73% to 85% compared to a naive baseline.

By Cheng Chang, Yining Mao, Peng Qi

llms agents safety

arXiv AI

Jul 10

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

arXiv:2601. 22588v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design.

By Zhuochun Li, Yong Zhang, Ming Li, Yuelyu Ji, Yiming Zeng, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao, Daqing He

llms benchmarks

arXiv AI

Jun 8

VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models

arXiv:2602. 03160v2 Announce Type: replace Abstract: Aligning Large Language Models (LLMs) with the diverse spectrum of human values remains a central challenge: preference-based methods often fail to capture deeper motivational principles.

By Woojin Kim, Sieun Hyeon, Jusang Oh, Jaeyoung Do

llms rag safety

One email a morning, machine-written

Related stories

Latent Performance Profiling of Large Language Models

Predicting Alignment Generalization with Value Representations

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models