AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

11,407 stories · RSS feed

arXiv Machine Learning

Jun 2

Looped Transformers with Layer Normalization Provably Learn the Power Method

arXiv:2606. 00605v1 Announce Type: new Abstract: Transformers have achieved remarkable success across a wide range of applications, and a growing body of work suggests that part of their strength comes from their ability to learn and execute algorithmic procedures.

By Lyumin Wu, Chenyang Zhang, Yuan Cao

llms safety

arXiv Machine Learning

Jun 2

How Neural Losses Shape VAE Latents

arXiv:2606. 00635v1 Announce Type: new Abstract: Modern VAEs are rarely trained with the pointwise likelihood implied by the standard $\beta$-VAE objective.

By Giorgio Strano, Luca Cerovaz, Michele Mancusi, Tommaso Mencattini, Emanuele Rodol\`a

diffusion safety

arXiv Machine Learning

Jun 2

Partial Fairness Awareness: Belief-Guided Strategic Mechanism for Strategic Agents

arXiv:2606. 00826v1 Announce Type: new Abstract: Strategic machine learning investigates scenarios where agents manipulate their features to receive favorable decisions from predictive models.

By Xinpeng Lv, Chunyuan Zheng, Yunxin Mao, Renzhe Xu, Hao Zou, Shanzhi Gu, Liyang Xu, Huan Chen, Yuanlong Chen, Wenjing Yang, Haotian Wang

agents robotics safety

arXiv Machine Learning

Jun 2

Enhancing LLM Metacognition via Cognitive Pairwise Training

arXiv:2606. 00869v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become central to LLM reasoning, but its outcome-level rewards can make models more willing to give confident answers when evidence or reasoning is unreliable.

By Weitao Li, Hao Zhou, Xuanyu Lei, Fandong Meng, Yuanhang Liu, Jingyi Ren, Ante Wang, Xiaolong Wang, Yuanchi Zhang, Fuwen Luo, Guangwen Yang, Lin Gan, Weizhi Ma, Yang Liu

llms reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jun 2

Longitudinal Multimodal Sensing of Physical Activity and Well-Being in Older Adults

arXiv:2606. 00345v1 Announce Type: new Abstract: Wearable and mobile sensing technologies enable continuous monitoring of human behavior and health in real-world settings.

By Flavio Di Martino, Mattia G. Campana, Marcello Magno, Lorenza Pratali, Franca Delmastro

multimodal safety

arXiv Machine Learning

Jun 2

Multi-Objective Reference-Aligned Machine Unlearning

arXiv:2606. 00399v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of specific training samples while preserving the model's utility.

By Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman, Shahryar Rahnamayan

safety

arXiv Machine Learning

Jun 2

KG-Guard: Graph-Based Hallucination Detection for Knowledge Base Question Answering

arXiv:2606. 00328v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for knowledge base question answering (KBQA), where answering requires selecting entities from a question-specific knowledge-graph subgraph.

By Albert Sawczyn, Piotr Bielak, Tomasz Kajdanowicz

llms rag nlp benchmarks safety

arXiv Machine Learning

Jun 2

Unlocking the Black Box of Latent Reasoning: An Interpretability-Guided Approach to Intervention

arXiv:2606. 01243v1 Announce Type: cross Abstract: Latent reasoning enables Large Language Models (LLMs) to perform multi-step inference within continuous hidden states, offering efficiency gains over explicit Chain-of-Thought (CoT).

By Shuochen Chang, Tong Bai, Xiaofeng Zhang, Qianli Ma, Qingyang Liu, Zhaohe Liao, Yibo Miao, Li Niu

llms safety

arXiv Machine Learning

Jun 2

World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications

arXiv:2606. 00133v1 Announce Type: new Abstract: World models, internal simulators that learn the structure and dynamics of an environment, have emerged as a central paradigm in the pursuit of artificial general intelligence, enabling agents to predict, plan, and reason within learned representations.

By Arif Hassan Zidan, Yi Pan, Hanqi Jiang, Ruiyu Yan, Wei Ruan, Zihao Wu, Lifeng Chen, Weihang You, Xinliang Li, Bowen Chen, Huawen Hu, Peilong Wang, Sizhuang Liu, Jing Zhang, Siyuan Li, Zhengliang Liu, Yu Bao, Lin Zhao, Lichao Sun, Dajiang Zhu, Xiang Li, Jinglei Lv, Quanzheng Li, Wei Liu, Tianming Liu, Wei Zhang

llms agents diffusion reinforcement-learning robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Quality-Diversity Evolution for Discovering Diverse Vulnerabilities in LLM Safety

arXiv:2606. 00801v1 Announce Type: cross Abstract: Current approaches to LLM adversarial testing suffer from coverage gaps: manual red-teaming does not scale, LLM-as-attacker methods exhibit mode collapse, and gradient-based approaches produce uninterpretable gibberish.

By Subhadip Mitra

llms safety

arXiv Machine Learning

Jun 2

IstGPT: LLM-based Anomaly Detection for Spatial-Temporal Graph in Industrial Systems

arXiv:2606. 01691v1 Announce Type: cross Abstract: Industrial Internet systems face increasing threats from sophisticated industrial control system (ICS) attacks, resulting in critical safety incidents.

By Yuchen Zhang, Ning Xi, Pengbin Feng, Shigang Liu, Jianfeng Ma, Yulong Shen, Yanan Sun, Xiaolin Zhou

llms robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

arXiv:2606. 01710v1 Announce Type: cross Abstract: Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification.

By Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani

llms rag fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Step-Level Sparse Autoencoder for Reasoning Process Interpretation

arXiv:2603. 03031v2 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved strong complex reasoning capabilities through Chain-of-Thought (CoT) reasoning.

By Xuan Yang, Jiayu Liu, Yuhang Lai, Hao Xu, Zhenya Huang, Ning Miao

llms efficiency safety

arXiv AI

Jun 2

AtomEval: Validity-Aware Atomic Evaluation of Adversarial Claim Rewriting in Fact Verification

arXiv:2604. 07967v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can rewrite refuted claims to evade evidence-based fact verifiers, but conventional attack success rate (ASR) can be inflated when rewrites change, weaken, or correct the false proposition they are supposed to preserve.

By Hongyi Cen, Mingxin Wang, Yule Liu, Jingyi Zheng, Hanze Jia, Tan Tang

llms multimodal safety

arXiv AI

Jun 2

You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

arXiv:2603. 00133v2 Announce Type: replace-cross Abstract: Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement.

By Kairan Zhao, Eleni Triantafillou, Peter Triantafillou

diffusion benchmarks safety

arXiv AI

Jun 2

Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding

arXiv:2603. 03312v3 Announce Type: replace-cross Abstract: Decoding natural language from non-invasive EEG signals is a promising yet challenging task.

By Yuchen Wang, Haonan Wang, Yu Guo, Honglong Yang, Xiaomeng Li

llms rag benchmarks safety

arXiv AI

Jun 2

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

arXiv:2603. 05308v3 Announce Type: replace-cross Abstract: Assessing whether an article supports an assertion is essential for hallucination detection and claim verification.

By Qiao Jin, Yin Fang, Lauren He, Yifan Yang, Guangzhi Xiong, Zhizheng Wang, Nicholas Wan, Joey Chan, Donald C. Comeau, Robert Leaman, Charalampos S. Floudas, Aidong Zhang, Michael F. Chiang, Yifan Peng, Zhiyong Lu

llms nlp benchmarks safety

arXiv AI

Jun 2

Beyond Independent Manipulation: Individual Fairness-aware Strategic Classification with Peer Imitation

arXiv:2606. 00827v1 Announce Type: cross Abstract: Strategic classification (SC) investigates scenarios where agents manipulate their features to obtain favorable decisions from predictive models.

By Xinpeng Lv, Chunyuan Zheng, Yunxin Mao, Renzhe Xu, Jinxuan Yang, Yuanlong Chen, Wangrong Huang, Shaowu Yang, Wenjing Yang, Xinwang Liu, Peng Cui, Haotian Wang

agents robotics safety

arXiv Machine Learning

Jun 2

Investigating and Alleviating Harm Amplification in LLM Interactions

arXiv:2606. 02423v1 Announce Type: cross Abstract: Large language models (LLMs) can serve as helpful assistants, yet they can equally function as harm amplifiers that enable malicious users to achieve harmful outcomes beyond their capabilities through extended interactions.

By Ruohao Guo, Wei Xu, Alan Ritter

llms benchmarks safety

arXiv AI

Jun 2

Multi-Objective Reinforcement Learning for Tactical Decision Making for Trucks in Highway Traffic

arXiv:2601. 18783v2 Announce Type: replace-cross Abstract: Balancing safety, efficiency, and operational costs in highway driving poses a challenging decision-making problem for heavy-duty vehicles.

By Deepthi Pathare, Leo Laine, Morteza Haghir Chehreghani

agents reinforcement-learning safety

Get the daily briefing