AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 28

Vib2ECG: A Co-Located Chest-Lead SCG-ECG Dataset and Benchmark for ECG Reconstruction

Vib2ECG introduces a new dataset of synchronized twelve‑lead ECG and chest‑wall vibration recordings collected at the six standard precordial sites (V1–V6) from 17 subjects, using inertial measurement units. The accompanying benchmark demonstrates that a lightweight 364K‑parameter U‑Net can reconstruct electrical cardiac signals at various locations from these low‑cost vibrational signals, though a hallucination effect—where ECG waveforms appear without corresponding electrical activity—was observed and analyzed. This work expands the use of cardiac vibrational signals for mobile ECG monitoring and offers insights into the spatial relationship between cardiac electrical and mechanical activities.

By Guorui Lu, Xiaohui Cai, Todor Stefanov, Qinyu Chen
arXiv Machine Learning
Oct 28

Theory and interpretability of Quantum Extreme Learning Machines: a Pauli transfer matrix approach

The paper develops a theoretical framework for quantum extreme learning machines (QELMs) using the Pauli transfer matrix (PTM) formalism. It shows how encoding, reservoir dynamics, and measurement observables jointly determine the nonlinear information processing capacity of a QELM, and provides interpretable diagnostics that can guide model design. The authors apply their analysis to forecasting nonlinear dynamical systems, demonstrating that a trained QELM’s readout closely reproduces the underlying flow map at leading orders.

By Markus Gross, Hans-Martin Rieser
arXiv Machine Learning
Oct 28

CrystalREPA: Transferring Physical Priors from Universal MLIPs to Crystal Generative Models

CrystalREPA is a plug‑and‑play framework that aligns the atom‑wise hidden states of crystal generative encoders with frozen universal machine‑learning interatomic potential (MLIP) representations using an element‑aware contrastive objective. By doing so, it transfers stability‑aware atomistic priors to generative models with minimal training overhead and no extra inference cost, consistently improving thermodynamic stability, structural validity, and fidelity across multiple generative frameworks and benchmark datasets. The study also shows that an MLIP’s suitability as a teacher is better predicted by the distinguishability of its atom‑wise representation space than by its leaderboard accuracy, providing a practical, accuracy‑independent selection criterion. whyItMatters":"The approach demonstrates a practical method to enhance crystal generative models’ ability to produce stable, realistic structures by leveraging existing MLIP knowledge without incurring additional inference costs."

By Chengqian Zhang, Yucheng Jin, Duo Zhang, Tiejun Li, Han Wang
arXiv AI
Oct 28

A Lightweight Attention-Based CNN with Saliency-Map Interpretability for Skin Cancer Detection

The paper introduces a lightweight attention-based CNN (CNN-Attention) for binary skin cancer classification that uses channel attention to highlight informative dermoscopic regions while keeping computational demands low. Trained on a dataset of 2,637 images, the model achieved 87 % accuracy, 0.9475 ROC AUC, and 0.9430 PR AUC, matching MobileNet’s performance with roughly half the parameters and storage. Saliency maps are employed to provide explainability by visualizing the image areas that influence predictions.

By Sanjana Islam Kasfia, Md. Ismiel Hossen Abir, AKM Bahalul Haque, Abir Mahmud Shahariar
arXiv Computation and Language
Oct 28

LLM Hallucination Detection via Low-rank Multilinear Decomposition

The paper introduces LOTUS, a low‑rank multilinear decomposition method for detecting hallucinations in large language models. LOTUS compresses variable‑length token representations into compact features using statistical descriptors of peak activations, distributions, and cross‑layer changes, then applies low‑rank decomposition to capture the underlying structure of hidden states. Experiments on several LLMs and QA benchmarks show that LOTUS outperforms recent state‑of‑the‑art hallucination detectors while requiring less labeled data.

By Devansh Sharma, Faizul Rakib Sayem, Shahana Ibrahim
arXiv Machine Learning
Oct 28

Amplify, Locate, Estimate (ALE) on Rare-Failure Rates of Fine-Tuned LLMs

The paper introduces Amplify, Locate, Estimate (ALE), a technique for efficiently estimating rare failure probabilities in safety‑fine‑tuned large language models. ALE amplifies failure rates by exploiting changes in token log‑probabilities, then uses a score‑function estimator and importance sampling to extrapolate back to the original rare rates. Experiments on 41 targets show ALE’s estimates closely match reference values, achieving accurate predictions for failure rates as low as 10⁻⁵ with far fewer labeled rollouts than direct sampling.

By Patrick Reichherzer, Jonathan Evertz, Thorsten Holz
arXiv Computation and Language
Oct 28

What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

The paper investigates how knowledge distillation affects the internal mechanisms of language models by comparing GPT‑2 with its six‑layer student, DistilGPT‑2. Using techniques such as linear CKA, logit lens analysis, and attention pattern decomposition, the authors examine two circuits: Indirect Object Identification (IOI) and Greater‑Than (GT). They find that the IOI circuit collapses completely in the distilled model, whereas the GT circuit largely survives, with the student’s layer‑4 head effectively remapping the teacher’s layer‑9 comparison head, achieving 87.7% of the original task performance.

By Nabeelah Faumi, Deshan Sumanathilaka
arXiv Computation and Language
Oct 28

Cross-modal Steering Vectors Exist Across Both Image and Text Generation

The paper investigates whether steering vectors—specific vectors that influence generation—can transfer across modalities in unified multimodal models. By demonstrating that vectors that steer text generation can also steer image generation (and vice versa) for concepts like emotion and object size, the authors show that these vectors have a consistent, interpretable effect even for modality‑specific concepts. The study reveals a geometric signature: highly transferable steering vectors point in similar directions across the modality gap, suggesting shared underlying mechanisms.

By Donghao Zeng (Mila, McGill University), Nishant Subramani (Carnegie Mellon University), Tamar Rott Shaham (MIT CSAIL), Benno Krojer (Mila, McGill University, Northeastern University)
arXiv Computation and Language
Oct 28

PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy

PRISM-BENCH is a new benchmark comprising 1,000 expert‑curated vision‑language questions across five under‑represented reasoning categories, each paired with an idealized reasoning trace validated by a second annotator. It introduces diagnostics beyond final‑answer accuracy: a perception probe that attributes about 85% of frontier‑model failures to misperception, and trace alignment that predicts answer correctness with a macro AUROC of 0.66. The benchmark also presents CORE, an efficiency‑aware metric that reorders 24.2% of model pairs relative to accuracy while remaining uncorrelated with response length, revealing that recent gains are driven by recognition rather than spatial grounding.

By Vansh Bhatia, Mehdi Rezagholizadeh, Parsa Ashrafi Fashi, Igor Gilitschenski, Vikram Appia, Emad Barsoum
arXiv Computation and Language
Oct 28

Watermark Basins: Black-Box Detection of Distilled LLMs

The paper introduces Watermark Basin Alignment (WM‑Basin), a black‑box method for detecting distilled large language models (LLMs). It exploits a persistent watermark basin—a key‑specific geometric signal embedded in the teacher’s parameter space during fine‑tuning—that remains detectable even when token‑level signals vanish. WM‑Basin maps suspect model outputs to gradient representations, removes ordinary fine‑tuning directions, and tests alignment with the watermark basin, achieving significantly higher detection rates than token‑level baselines across a range of teacher and student models.

By Hongyan Chang, Hady Elsahar, Tom Sander, Pierre Fernandez, Saeed Mahloujifar, Sylvestre-Alvise Rebuffi, Tom\'a\v{s} Sou\v{c}ek, Tuan Tran, Valeriu Lacatusu, Surya Parimi, Rashel Moritz, Vanessa Stark, Alexandre Mourachko
arXiv Computation and Language
Oct 28

Game-Theoretic Solvers with Bayesian Opponent Modeling and Moral Rhetoric Framing for GLEE

The paper presents an agent designed for the GLEE Competition that engages in bargaining, negotiation, and persuasion games against human and AI opponents. The agent combines a classical game‑theoretic solver, an online Bayesian layer for estimating opponent archetypes, and a language model for generating messages. Experiments include 13,200 Monte Carlo games against 11 synthetic archetypes, an ablation study of the opponent‑modeling layer, and an analysis of 9,337 live‑game messages showing a strong correlation between moral rhetoric framing and buyer purchase decisions.

By Hayden Arnold
arXiv Computation and Language
Oct 28

DYNASHIELD: A Black-Box Moving Target Defense for LLMs via Dynamic Decoding Customization

The paper introduces DYNASHIELD, a black-box moving target defense for large language models that dynamically customizes decoding hyperparameters and system prompts during inference. It first identifies decoding configurations that lower the probability of successful jailbreak attacks, then samples from a weighted pool of these configurations to add controlled variability to the model’s behavior. Experiments on seven open-source LLMs and four state-of-the-art jailbreak attacks show that DYNASHIELD significantly reduces attack success rates while preserving response quality and adding minimal inference overhead.

By Xiaoqun Liu, Weiming Qi, Qiben Yan
arXiv Computer Vision
Oct 28

ORBITAL: Optimization of Robust Brain--Computer Inference Through Architecture-Aware Low-Precision Compression

ORBITAL is a framework that optimizes robust brain–computer interface (BCI) decoders by combining structured low‑precision compression, recovery‑aware operating‑point selection, multi‑attack evaluation, and native edge validation. In experiments on nine subjects using the BCI Competition IV‑2a dataset, the S25‑TRADES variant achieved the highest mean true‑union robustness among compressed EEGNet models (23.06 % vs 21.27 % for the standard full‑precision model) while reducing parameters by 26.27 % and MACs by 25.10 %. On a Jetson Orin Nano, FP16 compression lowered board energy by up to 16.20 % and latency by 8.00 %, whereas native INT8 quantization roughly doubled the latency and energy, highlighting the importance of joint robustness and hardware‑aware optimization.

By Saim Rehman, Muhammad Shafique
arXiv Computer Vision
Oct 28

Efficient Remote Sensing Image-Text Retrieval via Interaction-Free Recall and Text-Guided Reranking

Efficient Remote Sensing Image-Text Retrieval via Interaction-Free Recall and Text-Guided Reranking proposes a fast-then-fine (FTF) retrieval framework that first performs interaction-free multi-granular recall to quickly match query embeddings with pre-computed gallery embeddings, then refines the top‑K candidates using a parameter‑free text‑guided cross‑modal reranking module called the Balanced Text‑guided Interaction Block (BTIB). The method introduces an inter‑ and intra‑modal loss to jointly optimize cross‑modal alignment across multi‑granular representations. Experiments on public benchmarks show that FTF achieves competitive retrieval accuracy while significantly improving retrieval efficiency compared with existing methods.

By Xi Chen, Xu Chen, Xiamgyang Jia, Wei Wang, Xu Zhang, Zhenyuan Shun
arXiv Computer Vision
Oct 28

RePAIR: Inverting Backdoor Attacks in Semantic Segmentation

RePAIR is a post‑hoc, label‑free defense for semantic segmentation models that counters backdoor attacks by inferring victim‑target relationships from unlabeled images and correcting predictions in feature space and output space. It localizes suspicious pixels by comparing classifier head outputs with internal semantic evidence, applying conservative constraints to avoid harming clean predictions. Experiments demonstrate that RePAIR significantly reduces attack success across various segmentation architectures while largely preserving clean performance.

By Arnold Brosch, Kira Maag
arXiv Computer Vision
Oct 28

When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

The paper investigates how multimodal large language models decide to refuse answering visually unanswerable questions. It examines three control mechanisms—instruction phrasing and placement, post‑training methods, and internal activation interventions—across three models on the MoHoBench benchmark. The study finds that refusal behavior is highly sensitive to instruction details, can be amplified by post‑training (sometimes leading to mixed refusal and answer attempts), and can be modestly altered by activation interventions without causing indiscriminate refusals.

By Ananth K Suresh, Aveen Dayal, Gouthaman KV, Aditya Pola, Vineeth N Balasubramanian
arXiv Computer Vision
Oct 28

Single-Step Bidirectional Unpaired Image Translation Using Implicit Bridge Consistency Distillation

The paper introduces Implicit Bridge Consistency Distillation (IBCD), a method that enables single-step, bidirectional unpaired image translation without adversarial loss. IBCD uses a diffusion implicit bridge model that connects PF-ODE trajectories between distributions, parametrized on a signed time axis to select the target domain, allowing one network to translate in both directions in a single step. The authors also propose distribution matching for consistency distillation and an adaptive weighting scheme based on distillation difficulty, achieving state‑of‑the‑art performance on benchmark datasets and extending to brain MRI contrast translation.

By Suhyeon Lee, Kwanyoung Kim, Jong Chul Ye
arXiv Computer Vision
Oct 28

PGID: Progressive Guided Inversion and Denoising for Robust Watermark Detection

PGID: Progressive Guided Inversion and Denoising is a training‑free, plug‑and‑play framework that protects semantic watermarking in diffusion models from imprint removal and forgery attacks. It restores detection reliability by projecting perturbed latents back to their original regions through progressive inversion‑denoising cycles, eliminating intermediate deflections and mitigating adversarial perturbations. Extensive evaluations show that PGID successfully recovers removed watermarks and identifies forged instances across multiple schemes.

By Minh Quoc Duong, Chun Tong Lei, Chun Pong Lau
arXiv Computer Vision
Oct 28

Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation

DreamTacVLA is a Vision‑Language‑Action framework that incorporates high‑resolution tactile images alongside wrist‑camera and third‑person vision to enable robots to anticipate and react to contact dynamics. It uses a Hierarchical Spatial Alignment loss to align tactile tokens with visual counterparts and a tactile world model to predict future tactile signals, trained on a hybrid dataset from digital twins and real experiments. The approach achieves up to 95% success on contact‑rich manipulation tasks, outperforming existing VLA baselines.

By Guo Ye, Zexi Zhang, Xu Zhao, Shang Wu, Haoran Lu, Shihan Lu, Han Liu