AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Machine Learning

Sep 30

RolloutFaith: Auditing Persistent Internal Interventions in Visual World Model

arXiv:2609.36843v1 Announce Type: new Abstract: Interpretability methods such as probes, activation patches and learned editors are designed to reveal or modify a model's current computation. World m...

By Junchi Yao, Ziyi Wang, Youling Huang, Lijie Hu

agents safety

arXiv Machine Learning

Sep 30

Physical Muon: Orthogonalization as an Equilibrium Computation

arXiv:2609.37525v1 Announce Type: new Abstract: Physical neural networks and analog in-memory computing could reduce the energy cost of neural network training. Realizing this potential, however, req...

By Yuren Hao

llms safety

arXiv Machine Learning

Sep 30

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

arXiv:2609.37616v1 Announce Type: new Abstract: Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on th...

By Abhinav Rajeev Kumar (Lossfunk), Paras Chopra (Lossfunk)

llms benchmarks safety

arXiv Machine Learning

Sep 30

Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space

arXiv:2609.37731v1 Announce Type: new Abstract: Activation space and parameter space provide complementary views of model computation. Activations represent information, while weights read, transform...

By Tue M. Cao, Lisiane Pruinelli, My T. Thai

llms safety

arXiv Machine Learning

Sep 30

Feedback-Calibrated Protein Optimization with Batch-Aligned Tail Arbitration

arXiv:2609.37808v1 Announce Type: new Abstract: Protein optimization aims to discover high-fitness sequences under a limited experimental budget. Existing machine-learning methods use task-specific p...

By Zefeng Lin, Xianyong Fang, Tianfan Fu, Xiaohua Xu

safety

arXiv Machine Learning

Sep 30

Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Sampling

arXiv:2609.38104v1 Announce Type: new Abstract: Power-sharpened sampling is an inference-time alternative to reinforcement-learning (RL) post-training for enhancing reasoning in large language models...

By Panagiotis Theodoropoulos, Nan Jiang, Xintong Duan, Ali Hasan, Yuriy Nevmyvaka, Evangelos A. Theodorou, Wei Deng

llms reinforcement-learning safety

arXiv Machine Learning

Sep 30

Privacy-Friendly Cohort Determination: Sealed, CSP-Independent In-Browser ML Inference of Professional Segments for Identity-Less Advertising

arXiv:2609.36153v1 Announce Type: cross Abstract: B2B advertising targets a viewer's professional attributes (employer size and industry, function, seniority) and has obtained them by matching identi...

By Om Shankar Tiwari, Navnit Shukla, Guanyu Wang, Akshay Jain

safety

arXiv Machine Learning

Sep 30

Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech

arXiv:2609.36324v1 Announce Type: cross Abstract: Flow-matching text-to-speech (TTS) models achieve high synthesis quality but require many neural function evaluations (NFEs) to integrate their gener...

By Yentl Collin, Evan Dufraisse, Amr Mohamed, Amine Khelif Khelif, Dani Bouch, Guokan Shang

efficiency multimodal safety

arXiv Machine Learning

Sep 30

FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance

arXiv:2609.36474v1 Announce Type: cross Abstract: In regulated industries like consumer finance, seemingly harmless user queries can exploit large language model vulnerabilities, triggering safety fa...

By Rikhiya Ghosh, Himanshu Kumar, Sriram Venkatapathy, Sahil Wadhwa, Alexandre G. R. Day, Pranab Mohanty

llms safety

arXiv Machine Learning

Sep 30

Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry

arXiv:2609.37230v1 Announce Type: cross Abstract: Visual distinctions are often finer than those reflected in linguistic conceptualization. Vision-language models exhibit a similar asymmetry: a disti...

By Woosang Jeon, Jiwon Yang, Soo Chung, Taehyeong Kim

llms diffusion multimodal safety

arXiv Machine Learning

Sep 30

FORUM: Frozen Outputs Reconciled Using Model Agreement for Visual Grounding

arXiv:2609.37488v1 Announce Type: cross Abstract: Frozen multimodal large language models (MLLMs) now solve standard referring expression comprehension with a single prompted call, yet on adversarial...

By Taiyo Sato, Takamasa Sanda, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama, Shunya Nagashima

llms multimodal benchmarks safety

arXiv Machine Learning

Sep 30

Foundation Neural-Network Quantum States for Molecular Potential Energy Surfaces in Second Quantization

arXiv:2609.37733v1 Announce Type: cross Abstract: Second-quantized neural-network quantum states have achieved accurate molecular energies, but extending them across molecular geometries requires a s...

By Lizhong Fu, Jianan Wei, Wenguan Wang, Honghui Shang

efficiency safety

arXiv Machine Learning

Sep 30

Post-Anomaly Detection Inference for Deep SVDD

arXiv:2609.37935v1 Announce Type: cross Abstract: Deep Support Vector Data Description (Deep SVDD) has become a prominent framework for unsupervised anomaly detection by learning latent representatio...

By Cao Le Cong Thanh, Dang Quang Vinh, Vo Nguyen Le Duy

benchmarks safety

arXiv Machine Learning

Sep 30

Boosting Adversarial Robustness and Generalization with Dictionary Structure

arXiv:2502.00834v2 Announce Type: replace Abstract: This work investigates a novel approach to boost adversarial robustness and generalization by incorporating structural prior into the design of dee...

By Zhichao Hou, Weizhi Gao, Hamid Krim, Runze Li, Xiaorui Liu

safety

arXiv Machine Learning

Sep 30

MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers

arXiv:2508.14925v2 Announce Type: replace-cross Abstract: By providing a standardized interface for LLM agents to interact with external tools, the Model Context Protocol (MCP) is quickly becoming a...

By Zhiqiang Wang, Yichao Gao, Yanting Wang, Suyuan Liu, Haifeng Sun, Haoran Cheng, Guanquan Shi, Haohua Du, Xiangyang Li

llms agents benchmarks safety

arXiv Machine Learning

Sep 30

Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

arXiv:2602.15382v3 Announce Type: replace-cross Abstract: Heterogeneous multi-agent systems combine models with different capabilities through a common communication interface. Exchanging internal st...

By Xiaoze Liu, Ruowang Zhang, Weichen Yu, Siheng Xiong, Liu He, Feijie Wu, Hoin Jung, Matt Fredrikson, Xiaoqian Wang, Jing Gao

llms agents efficiency multimodal benchmarks safety

arXiv Machine Learning

Sep 30

Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent

arXiv:2609.03221v3 Announce Type: replace-cross Abstract: Counterfactual fairness audits of clinical language-model agents report a flip rate: how often an action changes when only the patient's demo...

By Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar, Rahul Joshi

llms agents safety

arXiv Machine Learning

Sep 30

OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

arXiv:2609.32810v2 Announce Type: replace-cross Abstract: Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions,...

By Anqi Li, Zhixuan Ge, Yixuan Duan, Jiarong Qian, Chi-Yu Chen, MingYu Lu, Huan-Yu Hsu, Yu Gu, Yue Guo, Sheng Wang, Wei Qiu, Hanwen Xu

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Computation and Language

Sep 30

Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants

arXiv:2609.35812v1 Announce Type: new Abstract: In-car conversational assistants (ICAs) are increasingly integrated into vehicles to support route planning, vehicle control, and information access. E...

By Vaishnav Negi, Lev Sorokin, Soroosh Tayebi Arasteh, Andrea Stocco

llms safety

arXiv Computation and Language

Sep 30

Resolving the Missing Financial Data Crisis: A Generative AI Pipeline for SEC 10-K Extraction

arXiv:2609.35864v1 Announce Type: new Abstract: SEC 10-K filings contain substantial financial information that is not consistently captured in structured datasets, creating a missing-data problem af...

By Prisha Nair, Roee Shraga

llms safety

Get the daily briefing