AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

10,632 stories · RSS feed

arXiv Machine Learning

Jun 26

Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search

arXiv:2606. 27291v1 Announce Type: new Abstract: Job-search platforms rely on low-bandwidth query interfaces that often fail to capture the high-dimensional complexity of candidate profiles.

By Ping Liu, Qianqi Shen, Jianqiang Shen, Wenqiong Liu, Rajat Arora, Yunxiang Ren, Chunnan Yao, Dan Xu, Baofen Zheng, Wanjun Jiang, Andrii Soviak, Kevin Kao, Jingwei Wu, Wenjing Zhang

llms reinforcement-learning safety

arXiv Machine Learning

Jun 26

Forecasting With LLMs: Improved Generalization Through Feature Steering

arXiv:2606. 27199v1 Announce Type: cross Abstract: Successful forecasting involves identifying patterns between historical and future states of the world which generalize to future observations.

By Humzah Merchant, Bradford Levy

llms safety

arXiv AI

Jun 26

Symbolic Reasoning Frameworks Trigger Memory-Mediated Ecosystem Dynamics in Multi-Agent LLM Systems

arXiv:2606. 07552v2 Announce Type: replace-cross Abstract: Large language models exhibit a risk-averse "turtle" bias as strategic agents.

By Augustin Chan

llms agents benchmarks safety

arXiv Machine Learning

Jun 26

At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization

arXiv:2606. 26396v1 Announce Type: new Abstract: Pre-trained transformers have demonstrated remarkable generalization abilities, at times extending beyond the scope of their training data.

By Praneet Suresh, Jack Stanley, Sonia Joseph, Luca Scimeca, Danilo Bzdok

llms fine-tuning safety

arXiv Machine Learning

Jun 26

MORL-A2C: Multi-Objective Reinforcement Learning Reranker for Optimizing Healthiness in MOPI-HFRS

arXiv:2606. 23603v2 Announce Type: replace Abstract: Unhealthy dietary behavior continues to be a persistent public health issue in the United States, exacerbated by recommendation systems that prioritize user preference without considering nutritional health.

By Aarya Vasantlal, Joshua Zolla, Chuxu Zhang

rag reinforcement-learning benchmarks safety

arXiv AI

Jun 26

Vulnerability of Natural Language Classifiers to Evolutionary Generated Adversarial Text

arXiv:2606. 27215v1 Announce Type: new Abstract: Deep learning models have achieved impressive performance across various fields but remain vulnerable to adversarial inputs, particularly in NLP, where such attacks can have significant real-world consequences.

By Manjinder Singh, Alexander E. I. Brownlee, Mohamed Elawady

llms rag nlp benchmarks safety

arXiv AI

Jun 26

Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline)

arXiv:2606. 27163v1 Announce Type: cross Abstract: I describe my solution to the LeHome Challenge 2026, an ICRA 2026 competition on bimanual garment folding.

By Ilia Larchenko

reinforcement-learning robotics multimodal safety

arXiv AI

Jun 26

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

arXiv:2606. 27161v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead.

By Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 26

Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations

arXiv:2606. 26185v1 Announce Type: new Abstract: LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions.

By Hiroki Tamba

llms safety

arXiv Machine Learning

Jun 26

Jailbreaking for the Average Jane: Choosing Optimal Jailbreaks via Bandit Algorithms for Automatically Enhanced Queries

arXiv:2606. 26936v1 Announce Type: cross Abstract: With a profusion of jailbreaks for LLMs now widely known, a growing concern is that non-expert malicious actors ("the average Jane") could elicit actionable responses to malicious requests.

By Prarabdh Shukla, Ritik, Suhas Rao, Arpit Agarwal, Arjun Bhagoji

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 26

LAMP: Lane-Aligned Motion Primitives for Feasible Trajectory Prediction

arXiv:2606. 26661v1 Announce Type: cross Abstract: Motion forecasting is essential for autonomous driving systems to enable safe decision-making and planning in complex driving scenarios.

By Sangjin Han, Hoseong Jung, Jeongtae Her, Changhyun Choi, H. Jin Kim

agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 26

Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

arXiv:2606. 26387v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text.

By Xi Xiao, Chen Liu, Chih-Ting Liao, Yunbei Zhang, Qizhen Lan, Yuxiang Wei, Lin Zhao, Janet Wang, Jianyang Gu, Muchao Ye, Tianyang Wang, Hao Xu

llms reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 26

Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention

arXiv:2603. 14717v2 Announce Type: replace Abstract: Generating novel protein sequences that respect a family's statistical constraints typically requires training deep generative models on thousands to millions of examples.

By Jeffrey D. Varner

llms safety

arXiv Machine Learning

Jun 26

Neural Speaker Diarization via Multilingual Training: Evaluation on Low-Resource Nepali-Hindi Speech

arXiv:2606. 26144v1 Announce Type: cross Abstract: Speaker diarization, the task of determining "who spoke when" in a multi-speaker recording, is a critical component in applications such as meeting transcription, accessibility tools, and multilingual information retrieval.

By Samip Neupane, Sandesh Pokhrel, Sandesh Pyakurel, Basanta Joshi

safety

arXiv AI

Jun 26

Detecting and Controlling Sycophancy with Cascading Linear Features

arXiv:2606. 26155v1 Announce Type: new Abstract: Interpreting and controlling model behaviors through activation steering methods requires many pairs of contrastive samples that clearly exhibit desired or undesired behavior.

By Maty Bohacek, Rishub Jain, Nicholas Dufour, Thomas Leung, Chris Bregler, Roma Patel

llms safety

arXiv Machine Learning

Jun 26

Quantization in Federated Learning: Methods, Challenges and Future Directions

arXiv:2606. 26822v1 Announce Type: new Abstract: Federated Learning (FL) has become a foundational paradigm for privacy-preserving distributed intelligence, yet its scalability remains fundamentally constrained by communication bottlenecks, device heterogeneity, and the challenges of training under statistically non-IID data.

By Farwa Ikram, Dipanwita Thakur, Antonella Guzzo, Giancarlo Fortino

efficiency safety

arXiv Machine Learning

Jun 26

Implementation of reinforcement learning in chemical reaction networks: application to phototaxis as curiosity-driven exploration

arXiv:2606. 26168v1 Announce Type: new Abstract: Living systems navigate environments using noisy and incomplete sensory signals.

By Ruyi Tang (LCQB-AG), Gr\'egoire Sergeant-Perthuis (LCQB-AG), David Colliaux

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 26

High-Probability PL-SGD with Markovian Noise: Optimal Mixing and Tail Dependence

arXiv:2606. 26316v1 Announce Type: new Abstract: We study first-order methods for smooth objectives satisfying the Polyak-\L{}ojasiewicz (PL) condition when gradient samples are generated by an exogenous Markov chain.

By Dhruv Sarkar, Aprameyo Chakrabartty, Vaneet Aggarwal

safety

arXiv Machine Learning

Jun 26

Scoring Is Not Enough: Addressing Gaps in Utility-fairness Trade-offs for Ranking

arXiv:2606. 26369v1 Announce Type: cross Abstract: Scoring functions are used to represent the relevance of individual documents.

By Shubham Singh, Ian A. Kash, Mesrob I. Ohannessian

safety

arXiv Machine Learning

Jun 26

Learning Probabilistic Filters with Strictly Proper Scoring Rules

arXiv:2606. 26497v1 Announce Type: new Abstract: Bayesian filtering of partially and noisily observed dynamical systems seeks to infer the evolving conditional distribution of the state of a dynamical system, given observations, in an online fashion.

By Eviatar Bach, Ricardo Baptista, Jochen Br\"ocker, Bohan Chen, Andrew Stuart

llms multimodal safety

Get the daily briefing