AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 23

What Lies Beyond Perception: A Partition Theory of the Limits of AI Evaluation and Oversight

The paper introduces a partition theory of AI evaluation, arguing that an evaluator’s ability to certify a model’s capabilities is limited by its perception. It shows that if a model’s input partition refines the evaluator’s, it can pass all tests the evaluator can grade, but only a stricter evaluator can raise the discrimination floor. The authors define an echo test that is complete for the evaluator’s tests, proving that passing it only guarantees a minimum discrimination level and cannot detect exact imitation of weaker models.

By Ahmed M. Mahran
arXiv Computation and Language
Oct 23

Climbing the Ladder: Automating Data Pipelines and Reproducing Baselines in Low-Resource Settings

The paper describes the redevelopment of a small neural machine translation system for Sicilian, using parallel text extracted from the bilingual journal Arba Sicula and a language‑agnostic sentence encoder for alignment. It compares training small Transformers from scratch with a pretrained NLLB model fine‑tuned via low‑rank adapters, and applies a staged reverse‑training recipe that improves translation into Sicilian more than from it. Results show the pretrained model outperforms the small Transformers, but a multilingual adapter can degrade performance on untrained language directions.

By Donato Meoli, Eryk Wdowiak
arXiv Computer Vision
Oct 23

ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder

ProCLIP introduces a curriculum‑learning framework that progressively aligns a CLIP image encoder with an LLM‑based text embedder. It first distills CLIP’s text encoder knowledge into the LLM embedder, then fine‑tunes the image encoder via contrastive learning with self‑distillation, instance semantic alignment, and embedding structure alignment losses. The approach addresses CLIP’s token limit, lack of multilingual support, and preserves pre‑trained vision‑language alignment.

By Xiaoxing Hu, Kaicheng Yang, Ziyang Gong, Qi Ming, Zonghao Guo, Yu Tian, Xiang An, Ziyong Feng, Xue Yang
arXiv Machine Learning
Oct 23

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

The paper introduces Sensorimotor Alignment (S²), a training‑free offline metric that evaluates how well a model’s sensory representations match the geometry of physical motion. It demonstrates that higher S² scores correlate with better downstream robot task performance across various embodied model families, providing a low‑cost proxy for model selection before expensive closed‑loop testing. Additionally, S² can identify promising vision‑language model backbones early and offers representation‑level explanations for observed behaviors in vision‑language‑action models and world action models.

By Yunfei Xie, Howard H. Qian, Shiyi Lan, Kaiyu Hang, Chen Wei
arXiv Computation and Language
Oct 23

Domain-Specific Constitutional AI: Enhancing Safety in LLM-Powered Mental Health Chatbots

The paper introduces a domain‑specific Constitutional AI framework designed to improve safety in large‑language‑model‑powered mental health chatbots. It highlights the unique risks of mental‑health applications—such as misdiagnosis, symptom exacerbation, and self‑harm—and argues that general AI safeguards are insufficient for these sensitive contexts. The authors propose training models with mental‑health‑specific principles to ensure accurate crisis intervention, adherence to therapeutic guidelines, and robust handling of nuanced, vulnerable dialogues.

By Chenhan Lyu, Yutong Song, Pengfei Zhang, Amir M. Rahmani
arXiv Computer Vision
Oct 23

LoRA Superposition across Pretrained Models

LoRA Superposition across Pretrained Models explores whether low‑rank adaptations (LoRAs) trained independently on different pretrained backbones can be summed and applied unchanged to all contributing backbones. Across seven ViT‑B/16 CLIP backbones and eight image‑classification tasks, the exact sum of LoRAs improves all 56 backbone‑task pairs, achieving a mean accuracy gain of 30.21 percentage points over the unadapted backbone and only 0.28 percentage points below individually trained solo LoRAs. In contrast, a leave‑one‑model‑out sum yields a negligible gain of 0.06 percentage points, and the results suggest that backbone‑specific LoRAs are nearly orthogonal while cross‑backbone responses are concentrated on the diagonal, supporting a component‑wise non‑interference hypothesis.

By Keigo Watanabe, Hiroshi Kera, Kazuhiko Kawamoto
arXiv Computer Vision
Oct 23

Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes

The paper proposes a method to improve open‑vocabulary semantic segmentation by refining the CLIP visual encoder. It introduces a selective attention mechanism that preserves regional semantics and adds visual prototypes—auxiliary memory‑based classifiers built from confidently predicted patches—to bridge the modality gap. An optimizable feature extractor with exponential moving average updates further enhances prediction quality, and experiments on five datasets confirm the approach’s effectiveness.

By Aijing Yu, Zhengbo Wang, Lijun Sheng, Jian Liang, Xiao-Yu Zhang
arXiv Computer Vision
Oct 23

Interpreting vision transformers via residual replacement model

The paper investigates how vision transformers (ViTs) represent and process visual information by conducting a systematic analysis of 6.6K features across all layers using sparse autoencoders. It introduces the residual replacement model, which substitutes ViT computations with interpretable features in the residual stream, revealing a progression from low-level patterns to high-level semantics and specialized encoding of curves and spatial positions. This model provides a scalable, faithful, and parsimonious circuit that simplifies ViT computations, enabling intuitive understanding of ViT mechanisms and demonstrating its usefulness for debiasing spurious correlations.

By Jinyeong Kim, Junhyeok Kim, Yumin Shim, Joohyeok Kim, Sunyoung Jung, Seong Jae Hwang
arXiv Machine Learning
Oct 23

Harmful Responses Are Jailbreak Templates: Transferring Autoregressive Jailbreaks to Diffusion Language Models

The paper introduces RMT-Jailbreak, a method that transfers autoregressive jailbreak attacks to diffusion large language models (dLLMs). By masking tokens in a harmful response generated by an autoregressive model, the approach creates a high‑mask‑ratio template that, when fed to a dLLM, prompts it to reconstruct the harmful content. Experiments show that this technique effectively transfers jailbreak attacks across different models, highlighting a new vulnerability in diffusion-based language models.

By Jinwei Gan, Zifeng Cheng, Zhiwei Jiang, Ran Sun, Shiping Ge, Qing Gu
arXiv Machine Learning
Oct 23

Scaling Laws for Safety Probes

The paper introduces two scaling laws that predict the performance of safety probes—tools used to monitor language models for harmful behavior—based on model size and the number of labeled examples. By creating six large safety datasets and training probes across multiple model families, the authors show that probe error decreases as a power law with model size and further improves with more examples, with the improvement rate increasing for larger models. These laws enable researchers to forecast probe accuracy and decide whether to invest in more data or a larger model to achieve a desired error threshold.

By Yuchen Wu (University of Washington), Jerry Wei (Anthropic)
arXiv Machine Learning
Oct 23

What Comes Next and Why: Interpretable Next-Event Prediction with Neuro-Symbolic Rules

The paper introduces RULES, a neuro-symbolic model that learns sequential rules directly from data, and RUNES, which combines these rules with a residual network for next-event prediction. RULES can accurately recover ground truth rules in synthetic experiments and uncover compact, interpretable rule sets on real datasets. RUNES achieves competitive accuracy while offering transparent access to the active rules and the residual adjustments that refine predictions.

By Tim Nico Bauerschmidt, Isabel Valera, Jilles Vreeken
arXiv Machine Learning
Oct 23

PriMa-Causa: Causal Prescriptive Maintenance

PriMa-Causa is a new framework for causal prescriptive maintenance that ranks maintenance actions by their estimated interventional effect. It combines a causal reasoning layer, a parametric simulator that generates interventional ground truth, and a graph-conditioned tabular causal foundation model trained once on synthetic production‑line data. The approach achieves lower prescription error than associational baselines, especially under weak overlap and sparse context, and shows further improvement when a partially disclosed causal graph is used.

By Felix Saretzky
arXiv Machine Learning
Oct 23

PYTHIA, a transferable machine learning framework for simultaneous defect and adsorption site optimization

The paper introduces PYTHIA, a machine‑learning framework that combines foundational models and transfer learning to optimize defects and adsorption sites in materials. By training a large foundational model on chemically similar molecules and then fine‑tuning with a transformer layer and Gaussian processes, PYTHIA can efficiently predict optimal sites using limited data. The framework is demonstrated on two distinct systems, successfully identifying optimal adsorption sites even for complex materials.

By Ioannis Kouroudis, Zhaosu Gu, Poonam, Leon Muller, Alessio Gagliardi
arXiv Machine Learning
Oct 23

Shift-aware score transport for privacy auditing without retraining

The paper introduces a shift‑aware score transport method for privacy auditing that does not require retraining the target model. It estimates the in‑distribution non‑member score distribution from out‑of‑distribution data, transports this relationship to the target covariate distribution, and applies a conservative correction to account for potential extrapolation errors. Experiments on multiple datasets with distribution shifts show that the method provides more stable privacy estimates while maintaining tighter lower bounds than existing baselines.

By Yaohong Yang, Gauri Pradhan, Qi Chen, Samuel Kaski, Antti Honkela
arXiv AI
Oct 23

LaVGuard: Selective Decoding for Contextual Value Drift

LaVGuard introduces a selective decoding method to address contextual value drift in AI assistants. It separates an authorized ten‑dimensional profile from a frozen estimate of values in the visible prefix, using a profile‑relative density gate to decide when to evaluate candidate continuations. The approach constrains logit tilts by changed‑position and cumulative‑KL budgets, achieving higher true‑positive rates at low false‑positive rates and demonstrating improved risk‑safety scores compared to position‑matched edits.

By Jiale Dai, Liuxian Ma, Hongcan Deng, Xiaoke Niu, Wenjing Zhang, Zhaoxiang Liu, Shiguo Lian, Guojie Song
arXiv AI
Oct 23

RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization

RubricOpt is a framework that allows a policy to self‑revise the semantic rubric guiding its training. The policy diagnoses its own rubric‑free outputs, keeps clauses that are already satisfied, and edits those that fail to address recurring issues. This revised rubric is then used to steer Direct Preference Optimization (DPO) during training, and the rubric is removed at evaluation, with experiments showing mixed but sometimes positive effects on capability and open‑ended generation benchmarks.

By Wei Chen, Yubing Wu, Shigui Li, Delu Zeng, Jingfeng Liu, Junmei Yang, Zhiheng Zhou, Bo Li, Min Chen, Zhou Wang
arXiv AI
Oct 23

COMET: A coordinated multi-layer runtime monitoring framework for human-supervised AI-enabled safety systems

COMET is a coordinated multi-layer runtime monitoring framework designed for human-supervised AI-enabled safety systems. It employs three monitoring layers—signal integrity, physics consistency, and the AI system’s competence envelope—whose outputs are combined and displayed via a traffic-light interface. Evaluated on a compressed‑air leak dataset, COMET eliminates the AI’s silent failures under covariate shift and most sensor faults, while maintaining a low false‑alarm rate.

By Niclas Flehmig, Mary Ann Lundteigen, Shen Yin
arXiv AI
Oct 23

AI-Based End-to-End Automotive Design and Software Development for SDVs

The paper investigates how Artificial Intelligence can support end‑to‑end automotive design and software development for Software‑Defined Vehicles. It presents an AI‑driven toolchain that combines Large Language Models with Model‑Driven Engineering to convert natural‑language requirements into system architectures, executable software, and validation artifacts, while maintaining traceability, consistency, and safety compliance. Validation was performed on a centralized vehicle computing platform for adaptive cruise control and emergency braking using Hardware‑in‑the‑Loop and Software‑in‑the‑Loop tests, and the authors discuss current capabilities, limitations, and future directions toward more automated, reliable, and quality‑assured automotive software development.

By Nenad Petrovic, Alois Knoll
arXiv Machine Learning
Oct 23

CTRL-FM: Inference-Time Structural Control for Flow-Based Virtual Sensing under Long-Horizon Node Outages

CTRL‑FM is a flow‑matching framework that preserves observed sensor data while inferring missing values during long‑horizon node outages. It treats structural guidance as an inference‑time decision, selecting among uncontrolled, graph‑dominant, temporal‑dominant, and balanced policies based on self‑supervised proxy outages and local reliability estimates. Experiments on PeMS08, METR‑LA, Air‑36, and AQI show that CTRL‑FM outperforms existing graph, diffusion, and flow‑based methods, especially under 30% and 60% node outages, and degrades gracefully up to 75% sensor loss.

By Boshra Khalili, Andrew Smyth
arXiv Machine Learning
Oct 23

Escaping the Phylogenetic Trap: Persistent Homology Enables Cross-Species Ion Channel Classification via Multimodal Graph Neural Networks

The paper investigates how phylogenetic leakage can inflate performance estimates for protein classifiers, particularly in venom ion‑channel target prediction. By treating cross‑species prediction as a structured distribution shift, the authors compare five structural inductive biases using a multimodal graph neural network that fuses ESM‑2 embeddings with a disulfide‑aware attention mechanism. Persistent‑homology representations, which capture backbone geometry through topological descriptors, outperform other biases under a leave‑one‑taxon‑out protocol, achieving 63.4% accuracy versus 54‑55% for other geometric priors and 30% for a sequence‑only baseline.

By Arezoo Bybordi, Favour Achimba, Mariam Gelashvili, Jessy Ramirez, Weigang Qiu, Mand\"e Holford, Anita Raja