AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

10,300 stories · RSS feed

arXiv Machine Learning

Jul 1

Explaining Machine Learning and Memorization with Statistical Mechanics

arXiv:2606. 31110v1 Announce Type: new Abstract: Artificial neural networks (NNs) and machine learning (ML) algorithms are poorly understood from a theoretical perspective, which makes it difficult to fully realize their potential and overcome their weaknesses.

By Robin Theriault

safety

arXiv AI

Jul 1

IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO

arXiv:2606. 23032v3 Announce Type: replace Abstract: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating both Anthropic Claude and OpenAI ChatGPT frontier language models on financial tasks.

By Mostapha Benhenda

llms agents benchmarks safety

arXiv AI

Jul 1

Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning

arXiv:2502. 00684v2 Announce Type: replace-cross Abstract: Deep reinforcement learning (DRL) has successfully addressed many complex control problems.

By Zeyu Jiang, Hai Huang, Xingquan Zuo

agents computer-vision reinforcement-learning safety

arXiv AI

Jul 1

LabGuard: Grounding Natural-Language Laboratory Rules into Runtime Guards for Embodied Laboratory Agents

arXiv:2606. 31045v1 Announce Type: new Abstract: Scientific embodied agents are increasingly capable of carrying out laboratory procedures, but executing these procedures safely in dynamic laboratory environments remains challenging.

By Jingpu Yang, Fengxian Ji, Zhengzhao Lai, Zhexuan Cui, Guangxian Ouyang, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, Preslav Nakov, Zhuohan Xie

agents robotics safety

arXiv Machine Learning

Jul 1

DISCOVER: A Solver for Distributional Counterfactual Explanations

arXiv:2603. 16436v2 Announce Type: replace Abstract: Counterfactual explanations (CE) explain model decisions by identifying input modifications that lead to different predictions.

By Yikai Gu, Lele Cao, Bo Zhao, Lei Lei, Lei You

safety

arXiv Machine Learning

Jul 1

WIDER-FAIR: An Annotated Version of the WIDER-FACE Dataset for Fairness Evaluation

arXiv:2606. 31704v1 Announce Type: cross Abstract: The deployment of face detection models in real-world applications raises important fairness concerns, as these systems may showcase performance disparities across demographic groups.

By Maxime Moussi, Beno\^it Ronval, Siegfried Nijssen, F\'elicien Schiltz

rag benchmarks safety

arXiv AI

Jul 1

TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels

arXiv:2504. 12557v3 Announce Type: replace-cross Abstract: Ensuring safe behavior in reinforcement learning (RL) is challenging when safety constraints are implicit and cannot be densely measured.

By Siow Meng Low, Ze Gong, Akshat Kumar

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 1

Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks

arXiv:2606. 31700v1 Announce Type: new Abstract: Biological neural circuits obey Dale's principle: each neuron's synapses are uniformly excitatory or inhibitory.

By Yutaro Yamada, Luca Grillotti, Rujikorn Charakorn, Sebastian Risi, David Ha, Robert Tjarko Lange

diffusion reinforcement-learning benchmarks safety

arXiv AI

Jul 1

GR2 Technical Report

arXiv:2606. 31984v1 Announce Type: cross Abstract: Industrial recommendation systems serve billions of users through a multi-stage funnel -- retrieval, early-stage ranking, and re-ranking -- where the final re-ranking step disproportionately shapes user engagement and downstream performance, particularly for carousel and grid display formats.

By Yufei Li (Yongkang), Zaiwei Zhang (Yongkang), Mingfu Liang (Yongkang), Kavosh Asadi (Yongkang), Jay Xu (Yongkang), Jimmy Kim (Yongkang), Chongyang Bai (Yongkang), Jieyi Zhang (Yongkang), Hongye Xie (Yongkang), Prachi Agrawal (Yongkang), Dian Yu (Yongkang), Tianyi Chen (Yongkang), Jean-Pascal Billaud (Yongkang), Garret Buell (Yongkang), YK (Yongkang), Zhu (Yang), Sachin Patil (Yang), Brooke Bian (Yang), Zhou Fang (Yang), Kevin Huang (Yang), Shiva Sudanagunta (Yang), Yuzhen Huang (Yang), Emma Lu (Yang), Chris O'Brien (Yang), Yang Song (Yang), Lihong Li (Yang), Jacob Tao (Yang), Zhicheng Zhu (Yang), Chao Li (Yang), Gaoxiang Liu (Yang), Neil Wu (Yang), Zhongyin Hu (Yang), Li Han (Yang), Loki Chen (Yang), Ming Lei (Yang), Greg Rehm (Yang), Siyuan Song (Yang), Tianwei Zhang (Yang), Li Li (Yang), Ketan Singh (Yang), Yavuz Yetim (Yang), Ilyas Atishev (Yang), Satendra Gera (Yang), Ashkan Sadeghi (Yang), Rachel Yan (Yang), Nikko Mizutani (Yang), Shuaiwen Wang (Yang), Song Yang (Yang), Zhijing Li (Yang), Jiang Liu (Yang), Mengying Sun (Yang), Fei Tian (Yang), Xiaohan Wei (Yang), Chonglin Sun (Yang), Parish Aggarwal (Yang), Kaushik Rangadurai (Yang), Zhi Hua (Yang), Frank Shyu (Yang), Ruchit Sharma (Yang), Liyuan Li (Yang), Shike Mei (Yang), Wenlin Chen (Yang), Santanu Kolay (Yang), Ben Schulte (Yang), Deepak Chandra (Yang), Adam (Yang), Song, Sandeep Pandey, Xi Liu, Hamed Firooz, Luke Simon

llms nlp reinforcement-learning fine-tuning efficiency safety

arXiv AI

Jul 1

UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation

arXiv:2606. 31451v1 Announce Type: cross Abstract: Unified multimodal models (UMMs) have shown great promise in integrating understanding and generation across diverse modalities.

By Jiahang Tu, Fengyu Yang, Chenyang Ma, Xihang Yu, Ziyao Zeng, Shaokai Wu, Hanbin Zhao, Zhi Tao, Chao Zhang, Hui Qian, Alex Wong

multimodal benchmarks safety

arXiv AI

Jul 1

Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

arXiv:2603. 12893v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a standard technique for post-training diffusion-based image synthesis models, as it enables learning from reward signals to explicitly improve desirable aspects such as image quality and prompt alignment.

By David McAllister, Miika Aittala, Tero Karras, Janne Hellsten, Angjoo Kanazawa, Timo Aila, Samuli Laine

llms diffusion reinforcement-learning safety

arXiv AI

Jul 1

Corruption Robust Offline Reinforcement Learning with Human Feedback

arXiv:2402. 06734v2 Announce Type: replace-cross Abstract: We study data corruption robustness for reinforcement learning with human feedback (RLHF) in an offline setting.

By Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban, Adish Singla, Goran Radanovi\'c

reinforcement-learning safety

arXiv AI

Jul 1

DDIAgents: Mechanism-Conditioned Context Flow for Drug-Drug Interaction Prediction

arXiv:2606. 31085v1 Announce Type: new Abstract: Drug-drug interaction (DDI) prediction is essential for medication safety, yet it requires reasoning over heterogeneous biomedical evidence whose relevance changes across interaction mechanisms.

By Zhenqian Shen, Yu Liu, Xiaoyi Fu, Quanming Yao

llms agents benchmarks safety

arXiv AI

Jul 1

Amplifying Membership Signal Through Chained Regeneration

arXiv:2606. 31991v1 Announce Type: cross Abstract: The tendency of large generative models to memorize training data makes sample verification critical for privacy auditing and copyright enforcement.

By Wojciech {\L}apacz, Stanis{\l}aw Pawlak

llms diffusion safety

arXiv AI

Jul 1

Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens

arXiv:2606. 30755v1 Announce Type: cross Abstract: Claw-like AI agents (e.

By Peizhi Niu, Wenjie Qu, Shangding Gu, Tianneng Shi, Yuankai Li, Ahmad Tawaha, Hend Alzahrani, Vincent Siu, Boyi Li, Chenguang Wang, Jiaheng Zhang, Basel Alomair, Ming Jin, Muhao Chen, Chi Wang, Costas Spanos, Dawn Song

llms agents benchmarks safety

arXiv Machine Learning

Jul 1

From Failure to Alignment: A Requirements Engineering Framework for Machine Learning Systems

arXiv:2606. 31589v1 Announce Type: cross Abstract: Organisations designing, developing, and deploying machine learning systems (MLS) need to be able to check that these systems are trustworthy, and communicate this clearly to their stakeholders, be they different categories of users, engineers, or wider society.

By Amel Bennaceur, Gopi Krishnan Rajbahadur, Prince Mercy, Bashar Nuseibeh, Faeq Alrimawi

agents robotics safety

arXiv AI

Jul 1

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

arXiv:2606. 32032v1 Announce Type: cross Abstract: Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes.

By Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 1

Structure-Regularized Interpretable TCR-Epitope Prediction

arXiv:2606. 30902v1 Announce Type: cross Abstract: T cell receptor (TCR)-epitope binding prediction is essential for understanding adaptive immunity and developing immunotherapies.

By Jiarui Li, Zixiang Yin, Yunbei Zhang, Janet Wang, Samuel J. Landry, Zhengming Ding, Ramgopal R. Mettu

llms rag benchmarks safety

arXiv Machine Learning

Jul 1

Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?

arXiv:2606. 32008v1 Announce Type: new Abstract: Mechanistic interpretability (MI) requires full access to model internals, yet the APIs for most widely deployed language models at best expose log-probabilities over output tokens.

By Philippe Chlenski, Zachariah Carmichael, Ayush Warikoo, Chia-Tse Shao, Yingxiao Ye, Aobo Yang, Vivek Miglani, Nehal Bandi

llms safety

arXiv Machine Learning

Jul 1

Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again

arXiv:2606. 30995v1 Announce Type: new Abstract: Recent work has shown that well-optimized individual decision trees can match complex black box models in some settings, primarily in noisy domains.

By Zakk Heile, Hayden McTavish, Margo Seltzer, Cynthia Rudin

benchmarks safety

Get the daily briefing