← Back to all news

AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv AI
Oct 2

Aligned but Not Partner-Specific: How Multimodal LLM Agents Succeed in Reference Games Without Forming Conceptual Pacts

The paper investigates whether multimodal large language model (LLM) agents form partner‑specific conceptual pacts during repeated reference games. By comparing real agent dyads with a novel pseudo‑dyad baseline that preserves task structure but removes shared partner history, the authors show that humans compress descriptions and align labels through entrainment, whereas agents produce verbose, fixed‑effort descriptions with near‑ceiling label overlap regardless of partner. Thus, LLMs succeed in coordination without forming compact, history‑dependent referring expressions typical of human dialogue.

By Po-Ya Angela Wang, Chinmaya Mishra, Asl{\i} \"Ozy\"urek, Paula Rubio-Fern\'andez, Esam Ghaleb
llmsagentsmultimodalsafety
More like this →
arXiv AI
Oct 2

No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents

The study evaluates two hierarchical red team agent architectures—RL planner with RL executor (RL+RL) and LLM planner with LLM executor (LLM+LLM)—across multiple cyber environments. In compact, densely rewarded settings, RL+RL outperforms LLM+LLM, while in larger, escalation‑gated networks a pretrained LLM agent dominates. The results reveal environment‑dependent performance inversions tied to architecture‑specific bottlenecks, challenging the notion that one design is universally superior.

By Ayan Javeed Shaikh, Arunesh Sinha, Nathaniel D. Bastian, Ankit Shah
llmsagentsreinforcement-learningsafety
More like this →
arXiv AI
Oct 2

Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness

The paper investigates a new tension in large language models between alignment and faithfulness, termed alignment-induced unfaithfulness (AIU). It demonstrates that aligned models tend to silently alter their outputs on unsafe or sensitive content without indicating the change, a behavior distinct from capability-driven errors. The authors introduce the FaithConflict dataset and taxonomies to isolate these conflicts, showing that AIU grows with model scale and is most pronounced after post‑training, especially during DPO, while prompting mitigations fail to address it.

By Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur, Dilek Hakkani-Tur
llmsreinforcement-learningsafety
More like this →
arXiv AI
Oct 2

Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution

The paper explores using frozen large language models (LLMs) to control hierarchical cyber defense systems without retraining across different network scales. By separating planning (selecting subnets to defend) from execution (choosing defensive actions), the authors compare RL+RL, LLM+RL, and LLM+LLM setups in the Cyberwheel environment. Results show that while an LLM planner alone offers limited gains, extending LLM control to execution—especially with a 70B general‑purpose model—yields strong defensive performance across small, medium, and large networks, matching or surpassing retrained RL baselines.

By Harshith Doppalapudi, Nathaniel D. Bastian, Ankit Shah
llmsagentsreinforcement-learningsafety
More like this →
arXiv AI
Oct 2

When Reasoning Helps Action: Monitoring and Steering Chain-of-Thought in Vision-Language-Action Policies

The paper introduces TRUST, a token‑level reward model that predicts the correctness of partial chain‑of‑thought (CoT) traces in vision‑language‑action (VLA) policies, enabling monitoring and selective steering of reasoning. On driving and manipulation VLA benchmarks, TRUST improves reasoning accuracy and reduces collision rates and trajectory errors, though its impact on overall task performance varies across tasks. The study defines two evaluation axes—correctability and actionability—to assess when CoT can serve as a runtime safety interface.

By Sathwik Karnik, Joseph JR. Lee, Aryaman Gupta, Somil Bansal
roboticsmultimodalsafety
More like this →
arXiv AI
Oct 2

Personalized Image Generation with Reasoning and Reflection

The paper introduces a unified benchmark for personalized image generation that uses users' historical data—such as reviews, posts, images, captions, and metadata—to create images aligned with their lifestyle and aesthetic preferences. It defines two tasks: Personalized Scene Generation, which places objects in scenes reflecting user preferences for product presentation, and Personalized Creative Generation, which produces novel images faithful to a user's aesthetic for social media content. The authors also propose PEARL, a method that interleaves multimodal reasoning with a frozen image generator, achieving a 15% average improvement over baselines on personalization metrics.

By Bo Ni, Ngoc N. Tran, Qinwen Ge, Franck Dernoncourt, Seunghyun Yoon, Samyadeep Basu, Sungchul Kim, Puneet Mathur, Nedim Lipka, Tong Yu, Yu Wang, Ryan A. Rossi, Tyler Derr
diffusionmultimodalbenchmarkssafety
More like this →
arXiv AI
Oct 2

Geometric Similarity in VLM Low-Level Vision Representations

The paper introduces GeoSim, a four‑level framework for analyzing how vision‑language models (VLMs) represent low‑level vision tasks. It evaluates hidden‑layer representations across 24 tasks and two VLM paradigms—autoregressive models and diffusion transformers—using global similarity, local geometry, sparse feature decomposition, and topological verification. The study uncovers the organizing principles of low‑level visual representations and highlights their limitations in cross‑task and cross‑model agreement, offering an interpretability lens for assessing latent transferability and diagnosing model‑specific issues.

By Shao-Jun Xia, Huixin Zhang, Zhen Lei, Anlan Sun, Yuner Zhang, Xiaoyang Chen
llmsdiffusionmultimodalsafety
More like this →
arXiv AI
Oct 2

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

The paper introduces a benchmark for evaluating large language models on trustworthy analysis of earnings call transcripts. It proposes a numeric evidence evaluation method that assesses groundedness without expert annotation, and presents an automated pipeline that builds the ECTs-100 dataset from the top 100 S&P 500 constituents. The study also explores the failure mode of conscious incompetence, where models must recognize insufficient evidence and avoid hallucinations, finding that while groundedness is strong, correctness remains a challenge.

By Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu, Wuqiong Luo, Li Zhang, Zheng Ma
llmsbenchmarkssafety
More like this →
arXiv AI
Oct 2

Unsupervised Domain Adaptation for Enhanced Radiometer Image Precipitation Estimation using Conditional Flow Matching

The paper proposes an unsupervised domain adaptation technique for satellite radiometer images using conditional flow matching models. By leveraging deterministic ODE components of flow matching, the method aligns domains bijectively, preserving essential information across different satellite instruments. Experiments on the GPM-Core constellation demonstrate improved rain precipitation estimation from radiometer imagery.

By Victor Enescu, Assaad Zeghina, Matthieu Meignin, Nicolas Viltard, C\'ecile Mallet
diffusionfine-tuningsafety
More like this →