AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,884 stories · RSS feed

arXiv Computer Vision
Aug 28

TempJail: Temporal Jailbreak Attacks against Image-to-Video Generation Models

The paper introduces TempJail, a temporal jailbreak framework targeting image‑to‑video generation models. It exploits a newly identified vulnerability where unsafe semantics arise from the composition of frames over time, rather than from single‑frame violations. By decomposing malicious captions into visual conditions and temporal instructions, and by employing controlled latent perturbations and template rewriting, TempJail achieves a 23.3 % higher attack success rate than prior methods on several commercial models.

By Qi Lu, Zehui Guo, David Yuanda Gan, Zijing Li, Hengda Zhang, Weijun Xu, Qiankun Zhang
arXiv AI
Aug 28

Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents

The paper demonstrates that safety mechanisms for autonomous large language model agents fail to compose across iterative loops, as trajectory‑scoped monitors cannot detect attacks whose evidence is spread over multiple iterations. It introduces LoopHarness, a system that maintains a persistent, non‑decaying safety state across loops, bounding unauthorized actions with a constant that does not grow with the number of iterations. The authors provide a comprehensive evaluation protocol, including attacks that require cross‑iteration evidence, module ablations, and adaptive white‑box red‑team testing.

By Chenhao Wu, Haoxuan Jia, Yang Liu, Yingguang Yang, Yuhan Lin, Chongyang Zhang, Hao Zheng, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Shang Luo, Kefu Xu, Jifeng Zhu, Bin Chong
arXiv Computation and Language
Aug 28

SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning

SPEAR (Symbolic Process Evaluation and Alignment Reward) is a training‑free, plug‑and‑play reward method for on‑policy distillation in reinforcement learning. It converts natural‑language reasoning traces into domain‑adaptive symbolic milestones and uses the longest common subsequence to align student exploration with teacher milestones, producing a dense, order‑aware reward that enforces logical consistency without an external neural verifier. Experiments on math, science, and commonsense reasoning tasks show that SPEAR effectively bridges the reasoning gap between student and teacher models through sequence‑level distillation with efficient dense process rewards.

By Zhuochun Li, Yuelyu Ji, Yiming Zeng, Daqing He
arXiv AI
Aug 28

Beyond Factual QA: Mentorship-Oriented Question Answering over Long-Form Multilingual Content

The paper introduces MentorQA, a multilingual dataset and evaluation framework for mentorship-oriented question answering derived from long‑form videos. It contains nearly 9,000 QA pairs across four languages and defines evaluation dimensions such as clarity, alignment, and learning value that extend beyond factual accuracy. Experiments show that Multi‑Agent QA pipelines outperform other architectures, especially on complex topics and low‑resource languages, while automated LLM‑based evaluation shows variable alignment with human judgments.

By Parth Bhalerao, Diola Dsouza, Ruiwen Guan, Oana Ignat
arXiv Machine Learning
Aug 28

MODIS: Multi-Omics Data Integration for Small and unpaired datasets

MODIS is a semi‑supervised framework for integrating multi‑omics data that are often unpaired, partially labeled, and scarce, such as in rare disease studies. It trains on a large reference database and a small target dataset simultaneously, using diagonal integration and class‑label alignment to handle class imbalance. The architecture combines variational auto‑encoders, a class classifier, and an adversarially trained modality classifier, with a regularized relativistic GAN loss for stable training, and demonstrates high accuracy on synthetic data and the TCGA cancer dataset.

By Daniel Lepe-Soltero, Thierry Arti\`eres, Ana\"is Baudot, Paul Villoutreix
arXiv AI
Aug 28

Why did My Robot Just Change Personality? Prompting Guidelines for a Grounded Robot Persona in LLM-Based HRI

The paper "Why did My Robot Just Change Personality? Prompting Guidelines for a Grounded Robot Persona in LLM-Based HRI" addresses the lack of clear prompt design in human‑robot interaction using large language models. It proposes a framework and a structured prompt template with eight functional components to specify, bound, and adapt robot behavior. The authors base their guidelines on a review of prior work and on survey data from 27 HRI experts, highlighting issues such as unclear robot personality, the need for user adaptation, and ethical concerns around safety, deception, and governance.

By Ashita Ashok, Franziska Babel, Patrick Holthaus, Rucha Khot, Karla Bransky, Fethiye Irmak Dogan, Karsten Berns, Silvia Rossi, Minha Lee, Guy Laban
arXiv AI
Aug 28

EduRiskX: A Neuro-Symbolic Framework with F-Logic Reasoning for Early Academic Risk Prediction

EduRiskX is a neuro-symbolic framework that combines a temporal Transformer-based predictor with an F-Logic symbolic reasoning module to forecast students’ academic risk early in online courses. The neural part models longitudinal activity sequences, while the F-Logic rule base, grounded in Engagement Theory and the Student Integration Model, offers interpretable, rule-based explanations. On the Open University Learning Analytics Dataset, EduRiskX achieves an accuracy of 0.900 and an F1-score of 0.894 by week 38, detecting risk on average by week 9.32 with a 94.30% detection rate, outperforming state‑of‑the‑art time‑series and deep learning baselines.

By Yu Fu, Yongqi Kang, Yong Zhao, Rongfang Bie
arXiv AI
Aug 28

DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?

DEEPCHART is a new benchmark that evaluates large language models (LLMs) on faithful data‑science chart generation. It contains 1,482 expert‑annotated instances from scientific papers, financial filings, and ecosystem reports, and assesses chart creation through an Extract–Reason–Visualize pipeline. Experiments show that while LLMs can produce visually plausible charts, they frequently hallucinate data at the extraction and reasoning stages, especially in long, noisy, and multimodal contexts.

By Jiahui tang, Kuicai Dong, Dexun Li, Hongchao Gu, Haocheng Yu, Wei Han, Chen Zhang, Yong Liu, Hao Wang, Enhong Chen
arXiv AI
Aug 28

VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation

The paper introduces Vision-Free Adaptation (VFA), a method that separates multilingual language enhancement from visual alignment in multimodal large language models. VFA fine‑tunes a base LLM on multilingual text to create a multilingual task vector, which is then merged with the vision‑aligned task vector of an existing MLLM. Experiments on five MLLMs and six multilingual benchmarks show consistent gains while preserving multimodal and text‑only performance, and using less than 2% of text data narrows the performance gap to fully multimodal‑trained models.

By Yixia Li, Yaqing Shi, Zhiwen Ruan, Dongdong Zhang, Lingjie Jiang, Shaohan Huang, Yun Chen, Guanhua Chen, Furu Wei
arXiv AI
Aug 28

Self-Generated Text Recognition: Quality Heuristics, Cross-Task Transfer, and Downstream Bias in LLM Evaluation

The paper investigates Self‑Generated Text Recognition (SGTR), the ability of large language models (LLMs) to identify their own outputs. By evaluating 13–21 models across 6 experimental designs, it shows that SGTR accuracy varies with evaluation format, conversation structure, and task domain, and that a quality‑heuristic bias dominates results. The study also finds that fine‑tuning for SGTR in one setting can generalize to others and may cause models to prefer their own outputs when judging, highlighting potential safety concerns.

By Jesse St. Amand, Callum Canavan, Sohaib Imran, Joseph Hewson, Aaron Lutz, Shi Feng, Puria Radmard, Lennie Wells
arXiv AI
Aug 28

NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluation

NeuronFuzz is a white‑box fuzzing framework that uses internal safety neurons of large language models as continuous feedback for safety evaluation, eliminating the need to generate full model responses during testing. It constructs a SafetyOracle that converts neuron activations into a differentiable safety alarm score, enabling gradient‑based identification of sensitive template positions and fluent, context‑compatible prompt mutations. Evaluated on 21 text and multimodal models, NeuronFuzz achieves a 76‑100% jailbreak discovery rate on five white‑box source models and demonstrates strong zero‑shot transfer to open‑weight and proprietary targets.

By Zhiyuan Xu, Muhammad Firhard Roslan, Joseph Gardiner, Sana Belguith, Lichao Wu
arXiv AI
Aug 28

How Unlikely Is "Unlikely"? Assessing Verbal Probability Perception Across Large Language Models

The study evaluates how large language models (LLMs) interpret verbal probability expressions by mapping words to numbers and testing consistency across 19 models. Results show that LLMs largely mirror human benchmarks—preserving word order, recovering key anchor points, and reflecting the high variance of the term "possible"—but they exhibit a systematic upward bias for negative expressions like "unlikely" and "improbable." Explanation elicitation reduces within‑model variance but increases divergence between models, while a bidirectional roundtrip test reveals that leading models maintain coherent internal representations.

By Christos Petridis, Konstantinos Pelechrinis, Zoran Obradovic
arXiv AI
Aug 28

Diff Mining: Logit Differences Reveal Finetuning Objectives

Diff Mining is a framework that identifies what a finetuned language model has learned by comparing its logits to those of its base model. It extracts per-context logit differences on a reference corpus and aggregates them into an interpretable token set using either a Top‑K frequency method or Non‑negative Matrix Factorization. The approach outperforms existing model‑diffing methods in domain detection and bias identification, and it requires only access to output logits, making it scalable to large models.

By Greg Kocher, Robert West, Cl\'ement Dumas, Julian Minder
arXiv AI
Aug 28

MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA

MedFG-VQA is a lightweight medical visual question answering framework that uses a memory bank to enhance low‑frequency DCT features and graph‑enhanced cross‑attention for visual‑textual alignment. It introduces Frequency‑Memory Fusion to retrieve and fuse low‑frequency information from a learnable memory bank, and Graph‑Aware Cross‑Attention to refine cross‑modal features via graph convolution. The authors also create SynMed‑VQA, a synthetic dataset of over 2 million QA pairs across nine imaging modalities, and show that MedFG‑VQA matches or outperforms larger models on several biomedical VQA benchmarks while keeping computational costs low.

By Haowen Gu, Gensheng Pei, Zeren Sun, Mingwu Ren, Xiangbo Shu, Yazhou Yao, Fumin Shen
arXiv AI
Aug 28

AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air

AirLLM introduces a hierarchical diffusion policy framework that adapts Low‑Rank Adaptation (LoRA) rank configurations for remote fine‑tuning of large language models over wireless channels. The system uses a Proximal Policy Optimization agent to make coarse decisions based on wireless and linguistic cues, then refines these decisions with Denoising Diffusion Implicit Models to produce task‑ and channel‑specific rank vectors. Experiments across different signal‑to‑noise ratios show that AirLLM improves fine‑tuning performance while substantially lowering transmission costs.

By Shiyi Yang, Xiaoxue Yu, Rongpeng Li, Jianhang Zhu, Zhifeng Zhao, Honggang Zhang
arXiv AI
Aug 28

CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models

CounterVid introduces a scalable counterfactual video generation framework that creates videos differing only in actions or temporal structure while keeping scene context intact. The approach uses multimodal LLMs for action proposals and diffusion models for editing, producing a synthetic dataset of ~26k preference pairs for action recognition and sequence ordering. With the MixDPO optimization method, the authors demonstrate significant improvements in action recognition and temporal ordering on Qwen2.5‑VL and InternVL3 backbones, while maintaining overall video understanding.

By Tobia Poppi, Burak Uzkent, Amanmeet Garg, Lucas Porto, Garin Kessler, Yezhou Yang, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara, Florian Schiffers
arXiv Machine Learning
Aug 28

A Single Suffix to Break Them All: Basin-Aware Jailbreaks for Merged Model Families

The paper investigates safety risks in model merging, showing that even when all constituent models are individually safety‑aligned, merging can expose a jailbreak vulnerability rooted in the pretrained foundation model. It introduces Basin‑Aware Jailbreak (BAJ), a min–max optimization method that generates adversarial suffixes transferable across merged models sharing the same backbone, without needing the exact merging coefficients or checkpoints. Experiments demonstrate BAJ’s high transfer success rates across diverse backbones and merging settings, and its resilience against existing defenses.

By Yu Zhe, Yixin Tan, Junhao Wei, Wang Chen
arXiv Machine Learning
Aug 28

Adversarial Training Without Input Gradients via Low-Rank Householder Expansions

The paper introduces a method for adversarial training that avoids computing input gradients by using a low‑rank Householder expansion (LRHE) to directly generate small‑norm adversarial examples from a network’s parameters. This approach requires only forward passes and standard back‑propagation, eliminating the inner maximization loop and reducing computational cost to roughly 2.8 PGD steps per epoch. The resulting models achieve comparable robustness to multi‑step PGD training for small relative ε budgets, demonstrating the feasibility of gradient‑free adversarial training.

By Tiana C. Johnson, Donsub Rim
arXiv Machine Learning
Aug 28

Profit based evaluation of machine learning for nitrogen recommendations in winter wheat

The study evaluates machine learning for nitrogen recommendations in winter wheat by directly scoring profit loss on 892 yield response curves, rather than relying on prediction accuracy. Results show that ML models alone fail to recover the best nitrogen rate within farm tolerance and underperform standard UK advice across price scenarios. However, a simple post‑model correction step significantly reduces profit losses, and a hybrid approach combining standard advice with a damped correction eliminates bias and large losses.

By Xulong Wang, Po Yang