AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

11,407 stories · RSS feed

arXiv AI

Jun 4

A Unified Framework for Locality in Scalable MARL

arXiv:2602. 16966v2 Announce Type: replace-cross Abstract: Scalable methods for networked multi-agent reinforcement learning let each agent plan using only a small neighborhood of the agent graph.

By Sourav Chakraborty, Amit Kiran Rege, Claire Monteleoni, Lijun Chen

agents reinforcement-learning safety

arXiv AI

Jun 4

MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs

arXiv:2511. 07107v3 Announce Type: replace Abstract: Ensuring the safety of Large Language Models (LLMs) is critical for real-world deployment.

By Liang Shan, Kaicheng Shen, Wen Wu, Zhenyu Ying, Chaochao Lu, Yan Teng, Jingqi Huang, Qingshan Liu, Guangze Ye, Guoqing Wang, Jie Zhou, Liang He

llms safety

arXiv Machine Learning

Jun 4

Geospatial Foundation Models to Enable Progress on Sustainable Development Goals

arXiv:2505. 24528v3 Announce Type: replace-cross Abstract: Foundation Models (FMs) are large-scale, pre-trained artificial intelligence (AI) systems that have revolutionized natural language processing and computer vision, and are now advancing geospatial analysis and Earth Observation (EO).

By Pedram Ghamisi, Weikang Yu, Xiaokang Zhang, Aldino Rizaldy, Jian Wang, Chufeng Zhou, Richard Gloaguen, Gustau Camps-Valls

computer-vision nlp benchmarks safety

arXiv AI

Jun 4

Activation Steering of Video Generation Models via Reduced-Order Linear Optimal Control

arXiv:2606. 04775v1 Announce Type: cross Abstract: Text-to-video (T2V) models trained on large-scale web data can generate undesired content, motivating interventions that reduce harmful outputs without sacrificing visual quality.

By Jihoon Hong, Alice Chan, Qiyue Dai, Julian Skifstad, Glen Chou

diffusion fine-tuning benchmarks safety

arXiv AI

Jun 4

From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models

arXiv:2512. 05277v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari

llms agents robotics multimodal benchmarks safety

arXiv AI

Jun 4

Time Series Forecasting as Reasoning: A Slow-Thinking Approach with Reinforced LLMs

arXiv:2506. 10630v3 Announce Type: replace-cross Abstract: To advance time series forecasting (TSF), various methods have been proposed to improve prediction accuracy, evolving from statistical techniques to data-driven deep learning architectures.

By Yitong Zhou, Yucong Luo, Mingyue Cheng, Qi Liu, Jiahao Wang, Daoyu Wang, Enhong Chen

llms reinforcement-learning fine-tuning safety

arXiv AI

Jun 4

LaVIDE: Language-Prompted Satellite Change Detection via Map-Image Alignment

arXiv:2411. 19758v2 Announce Type: replace-cross Abstract: Remote sensing change detection based on a map reference and an up-to-date image boosts timely observation of the Earth's surface when earlier images are lacking for comparison.

By Shuguo Jiang, Fang Xu, Chuandong Liu, Hong Tan, Shengyang Li, Lei Yu, Wen Yang, Sen Jia, Gui-Song Xia

rag computer-vision multimodal benchmarks safety

arXiv AI

Jun 4

BRAINCELL-AID: An Agentic AI Created Brain Cell Type Resource for Community Annotation

arXiv:2510. 17064v4 Announce Type: replace Abstract: Single-cell RNA sequencing has transformed our ability to identify diverse cell types and their transcriptomic signatures.

By Rongbin Li, Wenbo Chen, Zhao Li, Rodrigo Munoz-Castaneda, Jinbo Li, Neha S. Maurya, Arnav Solanki, Huan He, Hanwen Xing, Meaghan Ramlakhan, Zachary Wise, Nelson Johansen, Zhuhao Wu, Hua Xu, Michael Hawrylycz, W. Jim Zheng

llms rag agents safety

arXiv AI

Jun 4

Aligning Deep Implicit Preferences by Learning to Reason Defensively

arXiv:2510. 11194v3 Announce Type: replace Abstract: Personalized alignment is crucial for enabling Large Language Models (LLMs) to engage effectively in user-centric interactions.

By Peiming Li, Zhiyuan Hu, Yang Tang, Shiyu Li, Xi Chen

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 4

GeM-NR: Geometry-Aware Multi-View Editing for Nonrigid Scene Changes

arXiv:2606. 05142v1 Announce Type: cross Abstract: Recent developments in multi-view image editing with generative models have brought us a step closer toward general 3D content generation and customization.

By Josef Bengtson, Yaroslava Lochman, Fredrik Kahl

benchmarks safety

arXiv AI

Jun 4

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

arXiv:2605. 18879v3 Announce Type: replace-cross Abstract: Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety.

By Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song, Jinsong Su

llms fine-tuning safety

arXiv AI

Jun 4

Adaptive Patching Is Harder Than It Looks For Time-Series Forecasting

arXiv:2606. 04074v1 Announce Type: cross Abstract: Adaptive patching is a recent and compelling proposal for time-series Transformers: allocate finer patches where the sequence looks locally informative.

By Federico Zucchi, Yi Xie, Chao Zhang, Keyuan Luo, Thomas Lampert, Ziyue Li

llms benchmarks safety

arXiv AI

Jun 4

RUBAS: Rubric-Based Reinforcement Learning for Agent Safety

arXiv:2606. 04051v1 Announce Type: cross Abstract: The evolution of LLMs into tool-enabled agents creates a new class of safety challenges associated with real-world execution rather than simple text generation.

By Xian Qi Loye, Qinglin Su, Zhexin Zhang, Shiyao Cui, Qi Zhu, Fei Mi, Hongning Wang, Minlie Huang

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 4

R-APS: Compositional Reasoning and In-Context Meta-Learning for Constrained Design via Reflective Adversarial Pareto Search

arXiv:2606. 04823v1 Announce Type: new Abstract: Large language models (LLMs) are fluent on open-ended tasks, yet in agentic settings, where a system must plan, use tools, and act over extended horizons, fluency does not ensure reliable delivery.

By Jo\~ao Pedro Gandarela, Thiago Rios, Stefan Menzel, Andr\'e Freitas

llms agents robotics fine-tuning safety

arXiv AI

Jun 4

The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents

arXiv:2606. 04296v1 Announce Type: new Abstract: As autonomous AI agents move from conversational systems to long-horizon software execution, runtime safety layers that decide when to interrupt an agent have become essential.

By Manvendra Modgil

llms agents safety

arXiv AI

Jun 4

Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Space

arXiv:2604. 04944v2 Announce Type: replace-cross Abstract: Multiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs).

By Mohammad Reza Ghasemi Madani, Soyeon Caren Han, Shuo Yang, Jey Han Lau

llms benchmarks safety

arXiv AI

Jun 4

On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers

arXiv:2603. 28762v2 Announce Type: replace-cross Abstract: Modern Text-to-Image (T2I) diffusion models have achieved remarkable semantic alignment, yet they often suffer from a significant lack of variety, converging on a narrow set of visual solutions for any given prompt.

By Omer Dahary, Benaya Koren, Daniel Garibi, Daniel Cohen-Or

llms diffusion multimodal safety

arXiv AI

Jun 4

PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

arXiv:2603. 23841v2 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) are increasingly used as primary sources of information, their potential for political bias may impact their objectivity.

By Rohan Khetan, Ashna Khetan

llms benchmarks safety

arXiv AI

Jun 4

Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

arXiv:2603. 10044v2 Announce Type: replace-cross Abstract: A safety score earned on a benchmark need not predict how the same model behaves once it is wrapped in an agentic scaffold the benchmark never tested.

By David Gringras

llms agents benchmarks safety

arXiv AI

Jun 4

What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs? A Study with Tabula RASA

arXiv:2602. 02834v4 Announce Type: replace-cross Abstract: What structural inductive bias helps transformers reason over knowledge graphs?

By Jonas Petersen, Camilla Mazzoleni, Gian-Alessandro Lombardi, Federico Martelli, Riccardo Maggioni

llms safety

Get the daily briefing