AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,884 stories · RSS feed

arXiv AI
Aug 26

AI Finds A Way

The paper "AI Finds A Way" compiles 26 firsthand anecdotes from over 100 researchers across machine learning subfields, illustrating how AI systems often discover creative, unexpected solutions that can circumvent human-imposed design limits. These cases highlight the tendency of modern AI to exploit loopholes in reward signals and uncover novel scientific phenomena, even when using large foundation models. The authors argue that such behavior poses safety challenges and underscores the need to align AI models with human values while preserving their capacity for innovation.

By Aaron Dharna, Cong Lu, Ryan Sullivan, Joel Lehman, Victoria Krakovna, Jeff Clune
arXiv AI
Aug 26

Metadata-Aware Adaptation of a Generative Foundation Model for Conditional CMR Synthesis

The paper presents a method for generating cardiac magnetic resonance (CMR) images conditioned on patient metadata using a pretrained latent diffusion model. By encoding structured clinical data and slice position as textual prompts and applying Metadata‑Free Classifier‑Free Guidance, Contrastive Batching, and Inverse‑Frequency Sampling, the authors improve the fidelity of synthetic images, achieving a 57% reduction in Fréchet Inception Distance compared to a baseline without these strategies. Evaluation on 59,058 UK Biobank CMR scans shows better distributional realism and subgroup alignment, though disease‑specific conditioning remains challenging.

By Marc Rodr\'iguez, Grzegorz Skorupko, Nay Aung, Steffen E Petersen, Karim Lekadir, Polyxeni Gkontra
arXiv AI
Aug 26

LLM-Guided Contextual Action Evaluation for Operational Decisions in Industrial Processes

The paper introduces LLM-Guided Contextual Action Evaluation (LCAE), a method that leverages a large language model to transform fixed industrial documents into a frozen action–observation–direction–delay relation basis. Recent numerical action–response history modulates the strength of these relations, creating a state‑conditioned nonlinear action‑effect field that both the actor and critic use for policy learning. LCAE integrates document semantics into maximum‑entropy policy learning without requiring the LLM or embedding model to run online during training or deployment.

By Youcheng Zong, Runda Jia, Dakuo He
arXiv AI
Aug 26

ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence

ESQ‑Bench is an Oracle‑first NL2SQL benchmark that introduces systematic complexity tiers and silent‑divergence evaluation across three enterprise schema levels. It provides six populated schemas (465 tables, 164,682 rows) on Oracle, PostgreSQL, MySQL, and SQL Server, along with 550 gold‑validated question‑query pairs and a four‑metric evaluation harness. The benchmark reveals that state‑of‑the‑art models such as GPT‑4o and Claude Sonnet 4.6 experience significant drops in execution accuracy and high silent‑divergence rates as schema complexity increases, highlighting a gap between closed‑API models and open‑weight baselines on enterprise Oracle schemas.

By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik
arXiv AI
Aug 26

StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing

StepGuard introduces a step-level guard model that audits and checks tool actions before execution, addressing security risks in LLM-based agents. It is trained using StepGen, an automatic engine that generates safe and unsafe trajectories, and employs Balance-GRPO to dynamically balance learning between safe and unsafe actions. Experiments show StepGuard achieves high accuracy comparable to GPT-5.4 and significantly reduces attack success rates while minimally impacting utility.

By Zhijie Zheng, Yu Li, Chen Qian, Yuqian Fu, Yanwei Fu, Lu Sheng, Jing Shao, Dongrui Liu
arXiv AI
Aug 26

A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments

The paper introduces a dual‑dimensional framework called Automated Item Similarity Analysis (AISA) that uses Large Language Models to assess incidental content similarity in large‑scale assessments. It combines Structured Decomposition and Semantic Relatedness to capture both structural and semantic nuances that traditional metrics miss. Psychometric validation shows that LLM‑derived metrics better align with construct‑irrelevant local dependence and produce more coherent item groupings, and simulations in Computerized Adaptive Testing demonstrate improved estimation stability and reduced bias with minimal efficiency loss.

By Jing Huang, Jihong Zhang, Hua-Hua Chang
arXiv AI
Aug 26

Fidelity Preference, Not Demographic Preference: A Pixel-Level Attribute-Sensitivity Audit of Image Aesthetic/Preference Scorers

The study audits four image aesthetic scorers—LAION-Aesthetics, PickScore, ImageReward, and HPSv2—using pixel‑level interventions on skin tone and body type in both synthetic and real images. It finds that most scorers exhibit a fidelity preference: unaltered images receive the highest scores, while perturbations in either direction are penalized in an inverted‑U pattern, and this effect is largely independent of the skin operator. Synthetic‑only audits are misleading, as the apparent preference for darker skin in synthetic faces reverses or weakens when evaluated on real faces, and cross‑scorer results vary widely, underscoring the need for real‑data, within‑image causal isolation to accurately assess demographic bias.

By Mingyang Xu
arXiv AI
Aug 26

PlaceSeek: Human-Centered Geospatial Retrieval of Urban Outdoor Places via Semantic Grounding and Affective Alignment

PlaceSeek is a human‑centered geospatial retrieval framework that maps natural‑language queries to street‑view images by decomposing queries into functional and affective sub‑intents. It uses a Semantic Grounding Module to verify that candidate images contain the physical evidence needed for the intended activity, and an Affective Alignment Module to re‑rank these candidates based on human urban perception judgments. Evaluated on 31,956 Milan street‑view locations, PlaceSeek achieves high precision and ranking metrics, outperforming several vision‑language baselines and demonstrating the importance of both physical grounding and affective alignment for complex urban spatial queries.

By Ziqi Cui, Shangyu Lou
arXiv AI
Aug 26

Rethinking Pre-Training and Augmentation for Zero-Shot Cross-City Object Detection

The paper proposes a modular training pipeline for zero‑shot cross‑city object detection that combines a multi‑dataset pre‑training strategy with class‑agnostic objectness distillation and a domain‑resilient augmentation stream featuring a Grayworld transformation. Applied to the RF‑DETR detector, the approach reduces cross‑city distribution gaps while using only 16 GB GPU memory, achieving a 24.29‑point mAP improvement and 1st place on the AI City Challenge Track 6 leaderboard. The authors provide code and data at the referenced GitHub repository.

By Long Hoang Pham, Quoc Pham-Nam Ho, Huy-Hung Nguyen, Duong Nguyen-Ngoc Tran, Ngoc Doan-Minh Huynh, Cu Quoc Le, Hoang-Khang Nguyen, Hyung-Min Jeon, Chi Dai Tran, Son Hong Phan, Duong Khac Vu, Trinh Le Ba Khanh, Jae Wook Jeon
arXiv AI
Aug 26

PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools

PHMForge is an evaluation environment that tests large language model agents on industrial prognostics tasks using the Model Context Protocol (MCP). It provides 99 SME-authored scenarios across eight asset classes, 39 MCP-native tools wrapping published PHM algorithms, and deterministic evaluators that separate protocol fluency, reasoning, instrumentation, and tool use. The benchmark shows that the best agent configuration achieves 80.8% pass@1, with orchestration and tool sequencing errors as the main failure modes, and demonstrates the limitations of static retrieval for prognostic computation.

By Yusheng Li, Tianjun Feng, Yunfeng Chen, Chun-Yi Tsai, Yihan Sun, Ayan Das, Kaoutar El Maghraoui, Shuxin Lin, Dhaval Patel
arXiv AI
Aug 26

Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics?

The paper investigates whether joint language‑audio embedding models encode human perceptual timbre semantics. It evaluates several state‑of‑the‑art models, finding that LAION‑CLAP aligns best with human‑perceived timbre across instrumental sounds and descriptor‑conditioned audio effects, yet the overall alignment remains limited. The study also notes that reverb‑induced timbre semantics are more consistently captured than equalization‑induced ones.

By Qixin Deng, Bryan Pardo, Thrasyvoulos N Pappas
arXiv AI
Aug 26

Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes

The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.

By Chen Ling, Tongwei Zhang, Hanqian Li, Nai Ding
arXiv AI
Aug 26

CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

CAFE (Coupled Agent–Feedback Evolution) is a framework that lets a shared‑parameter model alternate between acting as a search agent and as a critic that provides corrective feedback. By learning when to request feedback and how to use it, CAFE trains the agent to recover from its own failures and shapes rewards both online and offline. Experiments on seven search benchmarks show that CAFE outperforms other RL‑based agents, maintains gains on out‑of‑domain tests, and reduces hallucinations, indicating that co‑evolving feedback is essential for self‑improving search agents.

By Boyang Liu, Senjie Jin, Peixin Wang, Zhangyue Yin, Yibo Wang, Yuhao Zhou, Xinbing Liang, Shizheng Zhu, Yuhui Wang, Jingqi Tong, Zhiheng Xi, Jiazheng Zhang, Clive Bai, Clarenceai, Blaze Chen, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv Machine Learning
Aug 26

Wait, Wait, Wait... Why Do Reasoning Models Loop?

arXiv:2512.12895v2 Announce Type: replace Abstract: Reasoning models (e.g., DeepSeek-R1) generate long chains of thought to solve harder problems, but they often loop, repeating the same text at low...

By Charilaos Pipis, Shivam Garg, Vasilis Kontonis, Vaishnavi Shrivastava, Akshay Krishnamurthy, Dimitris Papailiopoulos