AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

10,801 stories · RSS feed

arXiv AI

Jun 10

Conformal Prediction for Neural Operators: Distribution-Free Uncertainty Quantification in Physics Simulation

arXiv:2606. 09923v1 Announce Type: cross Abstract: Neural operators such as the Fourier Neural Operator (FNO) have emerged as powerful surrogates for solving partial differential equations (PDEs), achieving speedups of several orders of magnitude over traditional numerical solvers.

By Michael Chin

benchmarks safety

arXiv AI

Jun 10

Bypassing Copyright Protection in Diffusion-based Customization via Two-Stage Latent Feature Optimization

arXiv:2606. 09909v1 Announce Type: cross Abstract: With the growing concerns over copyright infringement in diffusion-based customization, adversarial attacks have emerged as a prominent defense strategy to prevent malicious content forgery in personalized image generation.

By Ziang Xu, Wenbo Yu, Hongyao Yu, Hao Fang, Jiawei Kong, Bin Chen, Hao Wu, Shu-Tao Xia, Zhiyong Wu

diffusion benchmarks safety

arXiv AI

Jun 10

From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs

arXiv:2606. 10147v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) can listen and see, but how do audio and visual signals actually travel through the network to shape an answer?

By Wish Suharitdamrong, Muhammad Awais, Xiatian Zhu, Sara Atito

llms multimodal safety

arXiv AI

Jun 10

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

arXiv:2606. 10504v1 Announce Type: new Abstract: Cross-modal knowledge distillation (CMKD) studies how a (large) teacher model trained on one type of data (e.

By Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham, Phi Le Nguyen, Trong Nghia Hoang

efficiency multimodal benchmarks safety

arXiv AI

Jun 10

A Sober Look at Agentic Misalignment in Automated Workflows

arXiv:2605. 24197v2 Announce Type: replace Abstract: We study a class of emergent misalignment in multi-agent systems (MAS), with a focus on automated workflows, which we refer to agentic misalignment.

By Wenqian Ye, Bo Yuan, Zhichao Xu, Yijun Tian, Yawei Wang, Henry Kautz, Aidong Zhang

agents safety

arXiv Machine Learning

Jun 10

Gradient-Guided Reward Optimization for Inference-time Alignment

arXiv:2606. 09635v1 Announce Type: cross Abstract: Ensuring the reliability of Large Language Models (LLMs) under distribution drift requires inference-time adaptation.

By Hankun Lin, Ruqi Zhang

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 10

When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models

arXiv:2606. 10740v1 Announce Type: new Abstract: Failures in multi-turn reasoning models are largely invisible to terminal-score evaluation.

By Sai Kartheek Reddy Kasu, Nils Lukas, Samuele Poppi

safety

arXiv AI

Jun 10

From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG

arXiv:2605. 18271v2 Announce Type: replace-cross Abstract: With the rapid emergence of personal AI agents based on Large Language Models (LLMs), implementing them on-device has become essential for privacy and responsiveness.

By Changmin Lee, Jaemin Kim, Taesik Gong

llms rag agents benchmarks safety

arXiv AI

Jun 10

Local Is Not a Sufficient Privacy Boundary: Governing OS-Integrated On-Device AI

arXiv:2606. 10173v1 Announce Type: cross Abstract: As AI systems move into operating systems, privacy no longer turns only on whether a model runs locally.

By Jonghyun Chung, Sanket Badhe

llms rag safety

arXiv AI

Jun 10

Test-time Adversarial Takeover: A Real-time Hijacking Interface against Robotic Diffusion Policies

arXiv:2606. 10371v1 Announce Type: cross Abstract: Diffusion-based action generation has become a foundational component of embodied AI, but its reliance on visual conditioning leaves deployed visuomotor policies vulnerable to adversarial manipulation.

By Zi Yin, Peilin Chai, Siyuan Huang, Zhanhao Hu

llms diffusion robotics safety

arXiv AI

Jun 10

An LLM-Native Psychometric Instrument Does Not Predict LLM Behavior: Evidence Across 25 Models

arXiv:2606. 09843v1 Announce Type: cross Abstract: Large language models (LLMs) produce stable self-reports on personality inventories, but these self-reports do not predict observed behavior.

By Juan Manuel Contreras

llms safety

arXiv AI

Jun 10

One Lens, Many Worlds : A Capability-Typed Interface for World-Model Interpretability

arXiv:2606. 09936v1 Announce Type: cross Abstract: World models are now built on substantially different computational substrates.

By Bhavith Chandra Challagundla, Sanskar Pandey, Param Thakkar, Rishikesh Mallagundla, Yugandhar Reddy Gogireddy, Wenhao Lu, Hindol Roy Choudhury, Shravani Challagundla, Mohamed Deraz Nasr, Spursh Deshpande

llms rag diffusion safety

arXiv AI

Jun 10

Durable Evaluation Framework: Adversarial Arbitration for Sycophancy Reduction in Large Language Models

arXiv:2606. 07532v2 Announce Type: replace-cross Abstract: RLHF-trained models are systematically biased toward agreement over accuracy, a structural property of the training process.

By Sam Ryan

llms agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 10

HiGR: Industrial-Scale Hierarchical Generative Slate Recommendation Framework in Tencent

arXiv:2512. 24787v3 Announce Type: replace-cross Abstract: Slate recommendation, which presents users with a ranked item list in a single display, is ubiquitous across mainstream online platforms.

By Yunsheng Pang, Zijian Liu, Yudong Li, Shaojie Zhu, Zijian Luo, Chenyun Yu, Sikai Wu, Shichen Shen, Cong Xu, Bin Wang, Kai Jiang, Chengxiang Zhuo, Zang Li

rag efficiency benchmarks safety

arXiv AI

Jun 10

Exploring Accurate and Transparent Domain Adaptation in Predictive Healthcare via Concept-Grounded Orthogonal Inference

arXiv:2602. 12542v2 Announce Type: replace-cross Abstract: Deep learning models for clinical event prediction on electronic health records (EHR) often suffer performance degradation when deployed under different data distributions.

By Pengfei Hu, Chang Lu, Feifan Liu, Yue Ning

fine-tuning safety

arXiv AI

Jun 10

When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models

arXiv:2512. 06343v3 Announce Type: replace-cross Abstract: Reward models are central to Large Language Model (LLM) alignment within the framework of RLHF.

By Tong Xie, Andrew Bai, Yuanhao Ban, Yunqi Hong, Haoyu Li, Cho-Jui Hsieh

llms reinforcement-learning safety

arXiv AI

Jun 10

Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs

arXiv:2602. 02788v2 Announce Type: replace-cross Abstract: We aim to develop physics foundation models for science and engineering that provide real-time solutions to Partial Differential Equations (PDEs) which preserve structure and accuracy under adaptation to unseen geometries.

By Benjamin D. Shaffer, Shawn Koohy, Brooks Kinch, M. Ani Hsieh, Nathaniel Trask

llms benchmarks safety

arXiv AI

Jun 10

Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning

arXiv:2606. 10250v1 Announce Type: cross Abstract: Class imbalance is a common problem in deep learning that severely degrades performance.

By Haengbok Chung, Jae Sung Lee

benchmarks safety

arXiv AI

Jun 10

Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

arXiv:2606. 10917v1 Announce Type: new Abstract: Although Large Language Model (LLM) agents have demonstrated strong performance on complex tasks, their learning is often limited by inefficient interaction feedback and static training environments, which hinder broader generalization.

By Xucong Wang, Ziyu Ma, Shidong Yang, Tongwen Huang, Pengkun Wang, Yong Wang, Xiangxiang Chu

llms agents benchmarks safety

arXiv AI

Jun 10

Deployment-Time Memorization in Foundation-Model Agents

arXiv:2606. 10062v1 Announce Type: new Abstract: Foundation-model agents are increasingly long-lived systems that remember users across interactions, making memorization an explicit deployment-time function rather than solely a property of model weights.

By Lei (Rachel), Chen, Guilin Zhang, Kai Zhao, Dalmo Cirne, Andy Olsen, Xu Chu, Zeke Miller, Alet Blanken, Amine Anoun, Jerry Ting

llms agents nlp safety

Get the daily briefing