AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv Machine Learning
Sep 17

Gradient Descent with Stochastic Subspaces via Persistence of Memory

The paper introduces a novel technique called "persistence of memory" to enhance stochastic subspace methods for large‑scale optimisation. By using a weakly correlated guidance vector that is refreshed only at wide intervals, the method provides a structured direction for random subspace descent. The authors demonstrate that this guidance can be efficiently computed in sparse or minibatch settings and present the first theoretical analysis of classical SSD methods for sparse functions, showing alignment with low‑lying Hessian eigenvectors near the optimum.

By Subhroshekhar Ghosh, Clement Z. Q. Ng, Pierre-Louis Poirion, Akiko Takeda
arXiv Machine Learning
Sep 17

TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

TTM-Bench is a framework designed to benchmark text-to-music systems by establishing a common protocol for reproducible evaluation. It measures performance along two axes: musical-content alignment—assessed through semantic, genre, and musical-descriptor agreement scores against a shared musical specification—and computational efficiency, which includes generation latency, real-time factor, resource usage for local models, and cost for hosted services. A preliminary case study using TTM-Bench shows that higher alignment does not necessarily mean lower computational demands, underscoring the need for distinct, interpretable metrics.

By Giorgia Adorni, Michela Papandrea, Battista Rimoldi, Tiziano Leidi
arXiv Machine Learning
Sep 17

Topology-enhanced machine learning for speech signal processing

The paper introduces TopCap, a topology‑enhanced method for extracting features from speech time series. TopCap captures fine structural details, such as vibrations, that traditional spectral analysis may miss. When applied to classifying voiced versus voiceless consonants, TopCap matches neural network accuracy, and when combined with neural networks it improves robustness to noise, accuracy, stability, convergence, and interpretability.

By Pingyao Feng, Qingrui Qu, Haiyu Zhang, Siheng Yi, Zhiwang Yu, Zeyang Ding, Yifei Zhu
arXiv Computation and Language
Sep 17

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings

The paper introduces a benchmark for evaluating open‑source instruction‑tuned large language models (LLMs) on key‑value pair extraction from documents under both clean‑text and noisy OCR conditions. It tests decoder‑only models such as Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek on FUNSD, CORD, and SROIE datasets, using OCR outputs from PaddleOCR, EasyOCR, and Tesseract. Results show that while modern LLMs perform well on high‑quality text, their performance drops sharply with OCR noise, and the main determinants of success are semantic reasoning and textual fidelity, with larger models offering diminishing returns under noisy inputs.

By Zahra Anvari, Vassilis Athitsos
arXiv Machine Learning
Sep 17

Interpretable Retinal Disease Prediction Using Biology-Informed Heterogeneous Graph Representations

The paper introduces a biology-informed heterogeneous graph representation that models retinal vessel segments, intercapillary areas, and the foveal avascular zone to predict diabetic retinopathy stages from OCTA images. This graph-based approach reframes staging as a graph-level classification task solved with a graph neural network, achieving AUC-ROC values up to 84% and outperforming biomarker-based classifiers, CNNs, and vision transformers. The method also provides detailed, interpretable explanations by precisely localizing abnormal vessels and non-perfusion areas.

By Laurin Lux, Alexander H. Berger, Maria Romeo Tricas, Richard Rosen, Alaa E. Fayed, Sobha Sivaprasada, Linus Kreitner, Jonas Weidner, Martin J. Menten, Daniel Rueckert, Johannes C. Paetzold
arXiv AI
Sep 17

From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation

The paper introduces a 3D-CLIP encoder trained with structured hard negatives to improve vision‑language alignment for text‑to‑CT generation. This encoder drives a latent diffusion model that operates directly in 3D latent space, eliminating spatial artifacts from super‑resolution pipelines. Experiments on the CT‑RATE dataset show state‑of‑the‑art image fidelity and factual correctness across 18 pathological conditions, with lower inference time and GPU memory usage than competing methods.

By Daniele Molino, Camillo Maria Caruso, Filippo Ruffini, Paolo Soda, Valerio Guarrasi
arXiv AI
Sep 17

Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results

The paper proposes Domain-aware Fourier Features (DaFFs) for Physics-Informed Neural Networks (PINNs), embedding domain-specific geometry and boundary conditions into the positional encoding. DaFFs eliminate the need for explicit boundary loss terms, simplify optimization, and reduce training cost, leading to orders-of-magnitude lower errors and faster convergence compared to vanilla PINNs and RFF-based PINNs. An LRP-based explainability framework further shows that DaFFs produce more physically consistent relevance attributions, improving interpretability.

By Alberto Mi\~no Calero, Luis Salamanca, Konstantinos E. Tatsis
arXiv AI
Sep 17

HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark

HINTBench is a new benchmark for evaluating agents’ intrinsic risk, comprising 596 trajectories (400 synthetic risky, 136 synthetic safe, 30 real risky, 30 real safe) with an average length of 24 steps. It supports three tasks—risk detection, risk-step localization, and intrinsic failure-type identification—using a unified five-constraint taxonomy. Experiments show a large performance gap: while large language models can detect risky trajectories, they score below 37 on strict-F1 for risk-step localization, and existing guard models transfer poorly to this setting.

By Jiacheng Wang, Jinchang Hou, Fabian Wang, Ping Jian, Chenfu Bao, Zhonghou Lv
arXiv Machine Learning
Sep 17

Attention Dispersion as a Diagnostic Signal for Hallucination in Large Language Models

The paper proposes using the temporal volatility of internal attention mechanisms—measured by an unsupervised attention dispersion metric—as a diagnostic signal for hallucinations in large language models. It demonstrates that spikes in attention entropy within intermediate layers correlate with reasoning breakdowns, and shows statistically significant AUC improvements of up to +0.076 over output-based baselines on GSM8K and MATH-500 benchmarks using the Qwen2.5 model family.

By Shardul P. More, Tanuja S. Pawar
arXiv Computation and Language
Sep 17

How Calibration Content Shapes Attention-Based Reranking

Attention-based rerankers use a null-query calibration step to reduce positional and structural bias, but this step can inadvertently remove useful signal when prompts contain detailed instructions, constraints, personas, or demonstrations. The study demonstrates that such calibration harms performance on instruction-heavy tasks and introduces an interpolated null calibration that selectively limits instruction influence on the null baseline. This modification restores or surpasses the effectiveness of standard rerankers on instruction-heavy tasks while maintaining benefits when the null pass remains relevance-agnostic, and shows that in-context demonstrations can improve reranking with minimal calibration interference.

By Petros Karypis, Hossein Rajaby Faghihi, Peter Chen, Rui Zhu, Noveen Sachdeva, Yan Zhu, Julian McAuley
arXiv Computer Vision
Sep 17

Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery

The paper introduces Aligned Consensus Teacher (ACT), a semi‑supervised framework for visible‑infrared object detection that operates under an image‑pair‑level setting with only a few labeled pairs. ACT combines Cycle‑Consistent Region Alignment, Cross‑Modal Consensus Mean‑Teacher, and Text‑Guided Cross‑Modal Instance Augmentation to address limited supervision, pseudo‑label errors, and scarce tail‑class annotations. Experiments on DroneVehicle and VEDAI demonstrate that with just 10% labeled pairs, ACT achieves 94.3% of the fully supervised mAP.

By Qi Ming, Xiaxin Yuan, Jiahuan Zhou, Jiangmeng Li, Xudong Zhao, Zhanchao Huang, Juan Fang, Shaoguang Huang, Aleksandra Pizurica
arXiv Computer Vision
Sep 17

Think Before You Move: Latent Motion Reasoning for Text-to-Motion Generation

The paper introduces Latent Motion Reasoning (LMR), a two‑stage approach that separates text‑to‑motion generation into a planning phase and an execution phase. LMR uses a Dual‑Granularity Tokenizer to create a compressed, semantically rich reasoning latent for global trajectory planning and a high‑frequency execution latent for detailed motion fidelity. Experiments on T2M‑GPT and MotionStreamer show that this architecture improves both semantic alignment and physical plausibility compared to direct translation methods.

By Yijie Qian, Juncheng Wang, Yuxiang Feng, Chao Xu, Wang Lu, Yang Liu, Baigui Sun, Yiqiang Chen, Yong Liu, Shujun Wang