AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv Computation and Language
Sep 7

Measuring the Novelty of Biomedical Papers Using the Latent Distances between Knowledge Units

The paper proposes a new method for measuring the novelty of biomedical research by calculating latent distances between knowledge units—specifically MeSH terms—using three types of relationships: network, semantic, and hierarchical. It demonstrates that each relationship captures distinct distances and that combining all three yields a more accurate novelty assessment than existing metrics. Validation on a large PLoS ONE dataset and a H1 Connect dataset shows stronger alignment with peer judgments compared to prior indicators.

By Yi Zhao, Heng Zhang, Yuzhuo Wang, Wenqing Wu, Tong Bao, Chengzhi Zhang
arXiv Computer Vision
Sep 7

Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching

The paper introduces DM-Align, a single-stage optimization framework that jointly performs distribution matching for distillation and aligns video generative models with human preferences. By deriving complementary gradient directions—one minimizing the gap between real and fake models and another guiding the model toward preferred samples—the method eliminates the need for separate reinforcement learning and distillation stages. Experiments on multiple foundational video models show that this sample-guided approach consistently outperforms both standalone variants and traditional two-stage pipelines.

By Jiuzhou Lin, Junlong Wu, Fei Zuo, Huan Ouyang, Dewen Fan, Boheng Zhang, Huaiqing Wang, Jia Sun, Fan Yang, Houde Liu, Kehai Chen, Min Zhang, Tingting Gao, Han Li
arXiv Computer Vision
Sep 7

MEOX: Compact Multimodal Mixture-of-Experts for Earth Observation

MEOX is a compact multimodal masked autoencoder designed for Earth Observation that uses a 2.939 million‑parameter encoder and 3.115 million total parameters. It incorporates sensor‑specific adapters, explicit validity signals, and a shared sparse‑expert block to maintain modality‑dependent processing before a learned patch‑wise fusion, followed by fourteen encoder blocks that process a single spatial sequence with four metadata tokens. Pretrained on 1.228 million MMEarth64 samples, MEOX achieves strong performance on GEO‑Bench tasks, surpassing prior CSMoE results, and demonstrates effective sensor‑flexible representation learning with a modest parameter budget.

By Mohanad Albughdadi
arXiv Computer Vision
Sep 7

Collaborative On-Sensor Array Cameras

The paper presents a collaborative on‑sensor array camera that uses a distributed meta‑optics learning method to jointly optimize a 100‑million‑nanopost metasurface array for broadband visible imaging. By training the array end‑to‑end with a learned meta‑atom proxy and a parallax‑aware, noise‑aware reconstruction algorithm, the design overcomes the wavelength‑dependent limitations of traditional metalenses. Experimental results show that the camera delivers consistent image quality across varying scene illumination spectra without relying on generative reconstruction.

By Jipeng Sun, Kaixuan Wei, Thomas Eboli, Congli Wang, Cheng Zheng, Zhihao Zhou, Arka Majumdar, Wolfgang Heidrich, Felix Heide
arXiv Computer Vision
Sep 7

FSPGD: Rethinking Black-box Attacks on Semantic Segmentation

FSPGD introduces a feature-space black-box attack for semantic segmentation that targets intermediate representations rather than just output logits. The method uses a dual loss: an external loss to disrupt cross-model feature alignment and an internal loss to reduce consistency among same-class instances. Experiments on Pascal VOC 2012 and Cityscapes show that FSPGD outperforms existing logit-level and segmentation-specific attacks across CNN and Transformer backbones, and its adversarial examples improve robustness when used for training.

By Eun-Sol Park, MiSo Park, Yong-Goo Shin
arXiv Computer Vision
Sep 7

Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation

The paper introduces a hardware‑aware framework that uses genetic programming to evolve layer‑specific scalar functions for Vision Transformers, replacing traditional LayerNorm with efficient, heterogeneous approximations. By applying a post‑training re‑alignment strategy, the method eliminates the need for full model retraining while achieving 90‑93% variance capture and recovering over 84% of ImageNet‑1K Top‑1 accuracy for ViT‑B and ViT‑L. The resulting architecture removes the global reduction bottleneck, reducing arithmetic complexity and off‑chip memory traffic, thereby enabling efficient deployment of ViTs on edge accelerators.

By Kieran Carrigg, Sigur de Vries, Amirhossein Sadough, Marcel van Gerven
arXiv AI
Sep 7

AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems

AutoLR is an autonomous harness designed to streamline the iterative research‑and‑engineering cycle for industrial recommender systems, exemplified by NetEase’s gaming‑community app DASHEN. It integrates a multi‑expert council for adversarial review, a deterministic evidence‑weighted selector to allocate trial budgets, and a layered knowledge system that fuses external research with domain‑specific insights and empirical evidence. Large language model agents handle semantic reasoning and code generation, while deterministic controllers maintain control over execution, metrics, guardrails, and state management.

By Qi Zhang, Yanlin Chen, Wenchao Xiao
arXiv Computer Vision
Sep 7

Cross-dataset transportability of pediatric chest X-ray deep learning across three countries: discrimination, calibration, operating-point failure, and limited-label recovery

The study evaluates a deep‑learning model for pediatric pneumonia detection across chest X‑ray datasets from three countries, assessing discrimination, calibration, operating‑point transport, shortcut signals, and limited‑label recovery. Using a frozen DenseNet121 ensemble trained on Guangzhou data, the model achieved high internal AUROC (0.976) but performance dropped when applied zero‑shot to Bangladesh (AUROC 0.798) and Vietnam (AUROC 0.742). Limited‑label adaptation with Platt recalibration restored sensitivity but introduced significant specificity variability, highlighting the need to evaluate multiple performance dimensions in cross‑dataset transport studies.

By Nazim-E-Alam
arXiv Machine Learning
Sep 7

FedDRAW: Federated Dual Reputation Annealing Weighting for Heterogeneous Multi-Institutional Chest Radiograph Classification

FedDRAW introduces a new server‑side aggregation technique for federated learning called Federated Dual Reputation Annealing Weighting. It blends a data‑size prior with cosine similarity between client and global model parameters, using two annealing schedules to shift weight from size to similarity and to gradually relax selectivity. Experiments on 12 simulated client partitions of CheXpert and ChestMNIST datasets show FedDRAW outperforms seven federated baselines in AUC and geometric mean of sensitivity and specificity, with statistically significant results.

By Maryam Moradpour, Anne-Christin Hauschild
arXiv Computation and Language
Sep 7

Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time

The paper introduces an influence score that measures how much each attention head contributes to classification decisions in Transformer models, specifically for prompt injection detection. The score blends directional effects on logits with structural impact within the residual stream, allowing analysis at head, layer, and network scales. When applied to a DeBERTa model, the framework uncovers different decision patterns for correct versus incorrect predictions, offering a balanced approach between detailed circuit analysis and global output methods.

By Lisa Bouger, Yannick Teglia, Philippe Loubet Moundi
arXiv Computation and Language
Sep 7

Auditing Bias and Safety in Voice AI Customer Care

The paper introduces a validation‑gated audit framework for voice AI customer‑care systems, treating them as stateful, multi‑turn, tool‑mediated interactions where bias and safety can manifest as added burdens before a final decision. The framework distinguishes between native speech‑to‑speech, cascaded ASR‑to‑LM‑to‑TTS, and hybrid architectures, and applies matched service facts across controlled caller presentation conditions to validate fact invariance, presentation cues, artifacts, and acoustic measurements. It outlines seven validation gates, a six‑family metric set, and demonstrates the approach with a synthetic refund‑dispute audit example, while noting that production results are withheld until the protocol is satisfied.

By Vignesh Ethiraj, Ashwath David
arXiv Computation and Language
Sep 7

ConfRAG: Confidence-Guided Retrieval-Augmenting Generation

ConfRAG introduces a confidence-guided approach to reduce hallucinations in large language models and selectively trigger Retrieval-Augmented Generation (RAG) only when the model is uncertain. The ConfQA fine‑tuning strategy trains the model to answer correctly or respond with "I am unsure," achieving a drop in hallucination rates from 20‑40% to below 5% across factuality benchmarks. Building on ConfQA, ConfRAG limits external retrievals by more than 30% while maintaining over 95% accuracy in ideal scenarios.

By Yin Huang, Yifan Ethan Xu, Kai Sun, Vera Yan, Alicia Sun, Haidar Khan, Jimmy Nguyen, Jingxiang Chen, Mohammad Kachuee, Zhaojiang Lin, Yue Liu, Aaron Colak, Anuj Kumar, Wen-tau Yih, Xin Luna Dong
arXiv Computer Vision
Sep 7

Few-Shot Video Recognition via Hierarchical Metric Learning

The paper introduces HML-FSAR, a hierarchical metric learning framework for few-shot action recognition. It incorporates a spatial‑enhanced module, temporal MHA, heterogeneous alignment, spatial‑temporal fusion, and dictionary learning to build a comprehensive feature pipeline. Progressive constraints—center, alignment, contrastive, dictionary, and prototype metrics—are applied from frame‑level representations to final prototypes, improving feature compactness, alignment, discriminability, and robustness.

By Jiaxin Zhang, Haoran Gao, Xizhan Gao, Zihao Dong, Tingwei Wang, Sijie Niu
arXiv Computer Vision
Sep 7

Training-Free Speech-Centric Omni Understanding with Frozen VLMs

The paper introduces Training-Free Omni (TFO), a plug‑and‑play framework that transforms a frozen vision‑language model (VLM) into a speech‑centric omni model without modifying its architecture or requiring multimodal re‑alignment. TFO leverages Whisper to generate confidence‑filtered, timestamped transcripts and routes them through the VLM’s existing language interface, leaving the visual pathway untouched. Evaluations on 56 benchmarks across 21 languages show that TFO matches or surpasses native omni models on audio‑visual tasks, improves audio‑only performance, and preserves strong visual and reasoning capabilities.

By Ankan Deria, Hanoona Rasheed, Xilin He, Fahad Shahbaz Khan, Salman Khan
arXiv AI
Sep 7

The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMs

The paper investigates a continuation-triggered jailbreak in large language models, showing that moving an instruction suffix can markedly boost jailbreak success. By performing mechanistic interpretability at the attention‑head level, the authors reveal that the jailbreak arises from a competition between the model’s natural continuation drive and safety defenses learned during alignment. They introduce Head Competition Steering (HCS), an inference‑time technique that exploits this competition to suppress harmful outputs and distill the approach into a student model for efficient safety improvements.

By Yonghong Deng, Zhen Yang, Ping Jian, Xinyue Zhang, Zhongbin Guo, Chengzhi Li, Junxi Yin
arXiv Machine Learning
Sep 7

Locating and Steering Refusal Beyond Attention

The paper investigates where the ‘refusal’ behavior of language models resides across different architectures. It finds that a single direction in the residual stream governs refusal in transformers, and that the same direction—after a rigid rotation—also governs refusal in state‑space models (SSMs). By aligning these directions and applying a detector‑triggered gate, the authors demonstrate that refusal can be effectively transferred across transformer, SSM, recurrent, and hybrid architectures, showing that safety tooling can be ported by re‑estimating the direction at each architecture’s write site rather than rebuilding it from scratch.

By Preethi Carmel Bosco, Gopalakrishnan Srinivasan
arXiv Machine Learning
Sep 7

How Faithful Is Attribution for Sales Forecasting? A Counterfactual Study

The paper introduces a counterfactual interpretability layer for multi‑series WaveNet forecasters used in sales forecasting. It decomposes each forecast into contributions that exactly sum to the predicted value, avoiding allocation artifacts seen with SHAP‑style methods. The authors evaluate faithfulness via deletion/insertion tests, showing significant gaps that confirm the attributions reflect genuine model behavior, and they analyze where attribution is informative and where it is not.

By Glib Kechyn
arXiv Computer Vision
Sep 7

AngelFingerprint: A Traceable, Explainable, and White-Box Stealthy Watermark for Text-Guided Image Editing

AngelFingerprint introduces a watermarking framework for text-guided image editing that embeds the editing prompt’s CLIP text embedding directly into the diffusion model’s weights via a LoRA module. The watermark is recoverable from image pixels alone, providing an explainable trace of the edit while remaining stealthy even under full white-box access. Experiments on the MagicBrush dataset show the extractor achieves 86% top‑1 accuracy in 200‑way prompt retrieval, outperforming prompt inversion methods.

By Bo-Han Kung, Futa Waseda, Ching-Chun Chang, Isao Echizen, Shang-Tse Chen
arXiv Computer Vision
Sep 7

An Attention-Guided Global and Local Fusion Framework for Lesion-Focused Image Classification

The paper introduces an attention‑guided fusion framework that combines global and lesion‑focused local information for image classification. Using a three‑branch architecture built on DenseNet‑121, the model generates attention maps with Grad‑CAM, refines local features with CBAM, and adaptively fuses the two representations. Experiments on synthetic and real datasets, including skin, guava leaf, and grape leaf images, show that the fusion branch outperforms individual branches, achieving up to 97.75% accuracy on skin lesions and 99.64% on guava leaves.

By Mst Shafia Tasnima, Md Samaun Elaheea, Tanjim Taharat Aurpab, Md Musfique Anwar