AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Computation and Language
Oct 21

RealControlBench: A Benchmark for Interpretability-Informed Fine-Grained Control of LLMs and Agents

RealControlBench is a new benchmark that evaluates fine‑grained steering of large language models and agents across diverse behavioral targets and interaction settings. It extends beyond isolated generation tasks to include complex, execution‑grounded, multi‑turn agent interactions, covering instruction‑following control, tool‑use decision control, reasoning performance, reasoning length, and safety awareness. The benchmark jointly measures target controllability and capability preservation, testing robustness across prompts, domains, and model architectures, and finds that current steering methods are often unreliable in realistic scenarios.

By Xinying Cai, Bin Hu, Mingyu Jin, Qiming Zhang, Yang Ouyang, Bangwei Guo, Minghao Guo, Yutong Yin, Lang Gao, Feijiang Han, Sheldon Yu, Kaiqu Liang, Qingcheng Zeng, Jingcheng Niu, Fan Yang, Wei Cheng, Zhen Xiang, Julian McAuley, Zhaoran Wang
arXiv Computation and Language
Oct 21

Detecting Errors in Large Language Model Responses: The Incremental Predictive Value of Hidden States

The paper investigates whether the internal hidden states of large language models can improve the detection of hallucinations in model responses. By integrating hidden states with an output‑side baseline and evaluating on both open‑ended generation and multiple‑choice tasks, the authors find that adding hidden states yields modest AUROC gains of about 0.004 and 0.039. These results suggest that while hidden states contain additional predictive information, the overall improvement over strong standalone detection methods remains limited.

By Ningzhi Li, Xiaojun Ma, Xiaofei Wang
arXiv Computation and Language
Oct 21

Judging the Justification: Decision Certificate Supervision for Generative Reward Models

The paper introduces DeCert, a framework that evaluates the validity and verdict consistency of rationales generated by generative reward models (GenRMs) without needing reference explanations. DeCert translates these evaluations into preference supervision that considers both verdict correctness and justification quality, enabling the construction of rationale preference pairs that differentiate stronger from weaker justifications. Experiments across multiple model families and benchmarks show that DeCert outperforms strong generative judge baselines, improves rationale quality, and mitigates positional bias, raising position-consistent accuracy by up to 14.8 points.

By Guiquan Sun, Bing Bai, Martin Renqiang Min, Dongjin Song
arXiv Computation and Language
Oct 21

From DQI to C-DQI: Automatic Codebook Generation and Evaluation for Measuring Computational Discourse Quality Index

The paper introduces a protocol for generating and evaluating codebooks for measuring Computational Discourse Quality Index (DQI) using Large Language Models (LLMs). It compares top‑down, bottom‑up, and hybrid approaches, finding that inductive, data‑driven schemas yield higher inter‑model agreement while remaining valid. Experiments on 100 Israeli parliament transcripts support the reliability of these data‑driven schemas for large‑scale political discourse analysis.

By Esther Shizgal, Ron Keinan, Daniel Shtauber, Roee Tzvi Esquira, Ori Aronson, Matan Toledano, Or Rappel-Kroyzer, Liron Lavi, Renana Keydar, Ittai Bar-Siman-Tov, Reut Tsarfaty
arXiv AI
Oct 21

How Fine-Tuning Objectives Shape Layer-Wise Information in LLM Hallucination Detection

The study compares two fine‑tuning paradigms for detecting hallucinations in large language models: a classifier‑head (“cls”) and a sequence‑to‑sequence (“s2s”) approach. “Cls” consistently outperforms “s2s” because it induces a clear two‑cluster geometry in hidden states, enabling stronger class separation and linear decodability. Analysis of training dynamics reveals a narrow upper‑middle “InfoWindow” where task information emerges; truncating layers above this window removes 31% of the network without loss of accuracy and reduces latency by 1.4×, a strategy that transfers across multiple models and tasks.

By Kaiyang Wan, Forrest Sheng Bao, Amin Ahmad, Yuxia Wang
arXiv AI
Oct 21

Large language model and reinforcement learning based autonomous construction and progress updates

The paper proposes an autonomous construction system that combines a large language model (ChatGPT‑5) with a reinforcement learning algorithm to control a mobile crane during retaining wall construction. An enhanced soft actor‑critic model plans crane actions, while robot kinematics ensure these actions are executable. The system also uses a custom embedding model to extract relevant construction data and generates progress updates, safety information, and reports.

By Yifei Xiao, Huasheng Lin, T. Y. Yang
arXiv AI
Oct 21

MedPAO-Assist: Edge-Deployable, Modality-Aware Agentic AI for Radiology Report Structuring

MedPAO-Assist is an edge‑deployable, agentic AI framework that structures radiology reports from radiologist dictation and voice commands using only about 3.3 GB of RAM and an average latency of 31 seconds on a standard laptop CPU. It supports four imaging modalities—Chest X‑ray, Chest CT, Abdomen CT, and Brain MRI—and operates entirely with a 0.6 B‑parameter small language model, avoiding cloud APIs or high‑end GPUs. The system follows an Extract‑Ground‑Generate pipeline, performing concept extraction, SNOMED‑CT grounding, structured report generation, and a speech‑driven human‑in‑the‑loop correction phase, enabling a fully “no‑touch” reporting workflow that requires no keyboard input or internet connectivity.

By Shrish Shrinath Vaidya, Minmini Selvam, Ganapathy Krishnamurthi
arXiv AI
Oct 21

MAMath-Bench: Process-Aware and Robust Evaluation of Multimodal Multi-Agent Mathematical Reasoning

MAMath-Bench is a new benchmark that evaluates multimodal mathematical reasoning in multi‑agent systems (MASs). It contains 1,003 problems across 11 categories, multiple difficulty levels, and modalities such as text, images, and audio, with fine‑grained milestone annotations for correctness, reasoning‑process reliability, and robustness to adversarial perturbations. The study finds that while collaboration can improve intermediate reasoning coverage, it does not consistently boost final‑answer accuracy and can increase redundancy, and current systems remain vulnerable to subtle adversarial errors.

By Xinping Wang, Jie Ma, Rui Xing, Wensong Guo, Xiayi Zhang, Jianhao Deng, Zihao Zhou, Boqian Yang, Feijuan He