AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Computation and Language
Sep 28

Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment

The study investigates how prompt template choices during Knowledge Distillation (KD) affect safety alignment in language models. It finds that using chat templates during KD degrades safety alignment, making models more compliant with harmful queries, while non-chat templates better preserve the base model’s internal representations. These effects are observed across LLaMA, Gemma, and Qwen families on multiple safety benchmarks.

By Anjila Budathoki, Manish Dhakal, Benjamin M. Ampel, Yi Ding
arXiv Computation and Language
Sep 28

Do we need to answer that question? Salience and Answerability of Potential Questions in Naturalistic Dialogue

The study examines whether the salience of automatically generated potential questions in naturalistic dialogue predicts whether they are answered. Using a dataset of 7,124 questions from the British National Corpus, the authors find a weak but significant positive correlation between salience and answerability, weaker than in monologic text. They also note that more structured interactions show stronger agreement among annotators than less organized dialogues.

By Amandine Decker (LORIA, UL, CNRS, SEMAGRAMME, GU), Maxime Amblard (SEMAGRAMME, LORIA), Ellen Breitholtz (GU)
arXiv Computer Vision
Sep 28

CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models

CCRV-Bench is a constraint‑driven benchmark designed to evaluate visual causal reasoning in vision‑language models on single‑image physical scenarios. It assesses four causal task dimensions—causal relation discovery, state prediction, causal diagnosis, and intervention—while applying constraints such as entity symbolization, spatial grounding, factual adversarial constraints, and minimalist output constraints to reduce shortcut learning. Experiments on 15 multimodal models reveal that constraint sensitivity varies by task and model, with intervention and spatial grounding having the largest impact and factual adversarial constraints improving causal diagnosis across models.

By Linyuan Gao, Yuan Wu, Yi Chang
arXiv Computer Vision
Sep 28

The Shape of Events: Edge-Based Inductive Biases via Cross-Domain Distillation

The paper investigates how knowledge distillation from event cameras to RGB images can alter the inductive biases of convolutional neural networks. By transferring learning from the event domain, the authors find that models gain color invariance, a shape bias, and improved robustness to high‑frequency noise, largely due to reduced reliance on texture and increased emphasis on edge‑based object shape. These changes are evidenced by early‑layer processing differences and a spectral trade‑off between robustness to missing high‑frequency content and vulnerability to its contamination or geometric disruption.

By Soshun Kihara, Shunsuke Yasuki, Masato Taki
arXiv Computer Vision
Sep 28

WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving

WALT introduces a method to align latent trajectories with pretrained driving world models, creating a compact generative trajectory space that preserves action-relevant semantics without altering the original model. The approach uses a dual-branch autoencoder to map raw waypoints into this latent space and transfers visual world knowledge into trajectory representations. Experiments on NAVSIM benchmarks show modest performance gains and a 30.5% reduction in planner FLOPs, indicating that maintaining world representations while extracting action-relevant information can improve trajectory planning efficiency.

By Mingkai Jia, Jiaxin Guo, Zhijian Shu, Jiawei Xu, Mingxiao Li, Jintao Cheng, Ping Tan, Wei Yin
arXiv Computer Vision
Sep 28

FeatMark: Feature-level Watermark Protection against Mimicry Attacks with Diffusion Models

FeatMark is a watermarking framework that protects images from text‑to‑image diffusion model mimicry attacks by embedding small, scene‑consistent micro‑features instead of pixel‑level perturbations. It constructs domain‑specific feature banks, selects executable features, and injects them via mask‑guided concept editing to create highly localized, natural edits. Experiments on VGGFace2, CelebA‑HQ, and WikiArt show FeatMark remains robust against ten strong watermark removal attacks and several adaptive attacks, with minimal impact on perceptual quality and extending to video mimicry scenarios.

By Haoyang Li, Ruoxi Sun, Qingqing Ye, Benjamin Zi Hao Zhao, Yaxin Xiao, Jason Xue, Haibo Hu
arXiv AI
Sep 28

Financial Fragility in Societies of LLM Agents: Coordination Failures and Stabilizing Mechanisms

The paper "Financial Fragility in Societies of LLM Agents: Coordination Failures and Stabilizing Mechanisms" investigates how large language model agents can collectively cause financial failures when making individual protective decisions. Using the FRAIL framework, the authors simulate bank runs, debt rollovers, and reward crowdfunding, finding that 77% of bank-run and 83% of debt-rollover episodes fail even without malicious agents. They test three interaction mechanisms—compensated commitments, centralized agreements, and participant-led coalitions—each improving outcomes but none dominating across all scenarios, noting that early broad commitments are key to successful stabilization.

By Zhenhao Fu, Ruipeng Xu, Qibing Ren
arXiv AI
Sep 28

CARGO: Context-Aware Retrieval-Gated Evaluation of Agentic AI in Production

CARGO is a framework for evaluating agentic AI systems in production that addresses the problem of reference-instance divergence (RID), where reference-based judges penalize correct answers that involve different entity identifiers. It treats retrieved references as procedural exemplars, grounds judgments in the live instance’s context, assigns a three-way status to claims, and gates evaluation by retrieval confidence. Using the CARGO-Bench diagnostic suite, CARGO eliminates false penalties and improves discrimination while revealing a limitation in detecting procedural corruptions.

By Mukul Chhabra, Shail Patel, Luigi Medrano
arXiv AI
Sep 28

From Shortcut Learning to Discrete Neural Insertion Sort

The paper investigates how neural networks can learn to execute the insertion‑sort algorithm faithfully. Analysis of the CLRS30 baseline reveals that the model often learns shortcuts, producing correct final outputs without following the intended intermediate steps. To address this, the authors propose Discrete Neural Insertion Sort, which represents the sequence as a chain, separates exchanges from control transitions, and discretizes node states after each step. Trained on sequences of length 16, the model attains perfect accuracy on lengths 64 and 128, but ablation studies show that discretization and graph structure alone are insufficient without additional supervision of the global inner‑loop state.

By Konstantinos Mylonas, Thrasyvoulos Spyropoulos
arXiv AI
Sep 28

FedHisto-PAST: Parameter-Efficient Stain-Aware Federated Learning for Cross-Site Lung Histopathology Classification

FedHisto-PAST v2 is a parameter‑efficient, stain‑aware federated learning framework for cross‑site lung histopathology classification, combining a frozen HIBOU‑B foundation model with techniques such as paired‑view prediction, feature consistency, prototype learning, and adaptive aggregation. In a five‑client, non‑IID simulation and an exploratory LungHist700 cohort, the method achieved a Macro‑F1 of 0.7286 and a balanced accuracy of 0.7305, with the prediction‑level consistency component providing the most clear independent benefit. The framework updated only about 1.25% of the model parameters, demonstrating efficient adaptation while acknowledging limitations in privacy guarantees and clinical validation.

By Muhammad Muhtasim Shahriar, M. M. Golam Hafiz, Saad Aloteibi, Mohammad Ali Moni
arXiv AI
Sep 28

ViSTA: A Simple Bridge Extends Visual Alignment to Clinical Time-Series Understanding in Multimodal LLMs

ViSTA is a lightweight adapter that adds irregular numerical measurements to a pretrained vision‑language model’s chart representations, enabling accurate clinical time‑series prediction without altering the base model’s parameters. On the MIMIC‑IV dataset, ViSTA outperforms other adaptations across four metrics for acute kidney injury and mortality prediction, achieving an AUC of 0.7376 with only 0.516 million trainable parameters. It also delivers strong temporal question‑answering performance, reaching 69.27% accuracy with significantly fewer trainable parameters than low‑rank adaptation methods.

By Junyi Gao, Yu Shi, Pingzhao Hu, Ewen M Harrison
arXiv AI
Sep 28

Attribution Bias in Large Language Models

The paper introduces AttriBench, a benchmark dataset that balances author fame and demographics to study quote attribution in large language models (LLMs). Using AttriBench, the authors evaluate 11 popular LLMs and find that accurate attribution remains difficult, with significant disparities across race, gender, and intersectional groups. They also identify a new failure mode—suppression—where models omit attribution entirely, which is unevenly distributed across demographics and not reflected by standard accuracy metrics.

By Eliza Berman, Bella Chang, Daniel B. Neill, Emily Black
arXiv AI
Sep 28

Smooth Piecewise Cutting for Neural Operator to Handle Discontinuities and Sharp Transitions

The paper introduces Cut-DeepONet, a two‑stage training framework that explicitly models discontinuities in partial differential equations by partitioning the domain into smooth subregions and representing discontinuities as boundaries in a higher‑dimensional space. This lifting strategy aligns the operator learning task with neural network inductive biases, allowing the model to avoid directly approximating discontinuities. Experiments on benchmark PDEs demonstrate that Cut‑DeepONet outperforms state‑of‑the‑art methods, even with low‑resolution data, while using fewer trainable parameters.

By Ha Dang, Sebastian Schmidt, Juergen Hesser
arXiv Computation and Language
Sep 28

Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline

The paper evaluates a production text‑to‑SQL pipeline that uses an LLM as a judge, finding that the deployed gpt‑4o‑mini judge agrees with human annotators only weakly (Cohen’s kappa 0.04 on a disagreement‑enriched set and 0.42 on a random spot‑check). The authors identify a specific failure mode, GRADE‑HALLUCINATION, responsible for most over‑flags, and demonstrate that a self‑hosted Qwen3.6‑27B model achieves substantially higher agreement (kappa 0.72) at a much lower cost. They also show that ensembling judges does not improve performance, and that their audit method flags a significant portion of out‑of‑domain SQLs as potential issues.

By Haowei Liu, Hsin-Tai Wu, Yi Fang
arXiv Computer Vision
Sep 28

Amplify What You Gaze At: Target Saliency Boosting in Text-to-Image Generation

The paper introduces Target Saliency Boosting, a new task that enhances the visual prominence of a specific object in text-to-image generation without visual priors. It proposes GazeME, a lightweight framework that inserts learnable marker tokens around object descriptions to indicate which objects to emphasize or suppress. By building a saliency-semantics dataset and using Saliency Prior Marker Activation, GazeME learns to adjust markers during training and automatically applies them at inference, effectively boosting target saliency while maintaining semantic alignment and image quality.

By Shengqi Dang, Zhengxi Yu, Feilin Han, Xingyu Lan, Nan Cao
arXiv Computer Vision
Sep 28

MDSkin-Net: Multi-Task Skin Lesion Analysis Driven by Pattern Analysis Priors and Spatial Alignment Regularization

MDSkin-Net is a multi‑task skin lesion analysis framework that integrates Pattern Analysis priors into a hybrid CNN‑Transformer architecture. It introduces a Pattern Analysis‑Guided Attention Module (PAGAM) with improved Efficient Channel Attention, Multi‑Scale Spatial Attention, and Biased Asymmetry Attention, along with a multi‑scale spatial alignment regularization that uses segmentation masks as soft supervision. Trained only on the ISIC 2017 training split, the model achieves high segmentation and classification performance on multiple datasets, demonstrating strong zero‑shot generalization across different cohorts.

By Yijian Li, Saad Bedros, Paul Bigliardi, Mei Bigliardi Qi, Vassilios Morellas, Nikolaos Papanikolopoulos
arXiv Computer Vision
Sep 28

HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

HyperErase introduces a hypernetwork-based framework for concept erasure in text-to-image models, replacing static adapters with prompt-conditioned parameter synthesis. The method maps textual descriptions to LoRA updates, eliminating per-prompt gradient optimization and manual merging. A decoupled rectification strategy further stabilizes and refines the synthesized adapters, yielding improved erasure effectiveness, image quality, and semantic alignment across diverse concepts.

By Yi Sun, Xinhao Zhong, Zhiqi Zhang, Yimin Zhou, Junhao Li, Yuxia Qiao
arXiv Computer Vision
Sep 28

Structured Reasoning Agentic Framework for Interpretable Critical View of Safety Assessment

The paper introduces ReasonCVS, a structured reasoning framework for assessing the Critical View of Safety in laparoscopic cholecystectomy. It uses a Vision‑Language Model to build an Anatomical Scene Graph Abstraction and a Large Language Model–based Rationale‑Aware Reasoning Agent to verify sub‑criteria, producing a final verdict with traceable clinical rationale. Experiments on the Endoscapes‑CVS201 benchmark show ReasonCVS outperforms existing methods with a 68.1% mAP while offering interpretable, criterion‑level explanations.

By Qing Xu, Yuxiang Luo, Zhen Chen
arXiv Statistics ML
Sep 28

Brenier Meets Adversarial Training: Optimal Transport Geometry for Robust Learning

The paper introduces a penalized distributionally robust optimization framework that allows an adversary to choose any distribution while incurring a Wasserstein penalty for deviating from the empirical distribution. It shows that the adversary’s problem can be reformulated as optimizing transport maps that push empirical samples to adversarial ones, proving that optimal maps are cyclically monotone. The authors argue that standard per-sample adversarial training violates this property and propose two remedies—multi-start particle ascent and input-convex neural network parameterization—to enforce cyclical monotonicity, demonstrating improved robustness and generalization in experiments on regression, image classification, and control tasks.

By Alireza Abdollahpoorrostam, Ehsan Sharifian, Buse \c{S}en, Marco Cuturi, Daniel Kuhn
arXiv AI
Sep 28

FLIP: Final Layer Inference-Time Probing for Vision-Language Models

FLIP is a final‑layer inference‑time probe designed to test whether a logit‑facing intervention site in an open‑weight vision‑language model (VLM) supports structured, task‑linked computation rather than generic perturbation. The probe applies elementwise flooring to the final normalized hidden state before logit computation, leaving other model components unchanged. By sweeping intervention strength on a controlled detection/counting task, FLIP identifies three regimes—negligible change, a bounded interior regime with improved detection recall and reduced counting error, and over‑suppression—while a four‑criterion protocol ensures the observed effects are mechanistically interpretable.

By Drandreb Earl O. Juanico, Rowel O. Atienza