AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Computation and Language
Oct 7

Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention

The study introduces MedQADE, a German open‑response clinical benchmark with 3,800 question‑answer pairs and physician reference annotations. It evaluates large language models (LLMs) as judges, finding that while some LLMs (e.g., Gemini 3 Flash) achieve physician‑level agreement on correctness, they exhibit self‑bias and low abstention rates. Physicians showed moderate agreement on correctness but limited agreement on difficulty, and their abstention increased with perceived difficulty.

By William Philipp, Finn Fassbender, Daniel Fister, Thorsten Langer, Martje G. Pauly, Rebecca Herzog, Markus A. Hobert, Theresa Paulus, Alexander Baumann, Chi Wang Ip, Lukas L. Goede, Johanna Reimer, Sebastian L\"ons, Ronald B\"ock, Sebastian Fudickar
arXiv Computer Vision
Oct 7

The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Models

The paper investigates how diffusion models, even after fine‑tuning to unlearn harmful concepts, still retain a linear residual of those concepts in token embeddings. It introduces SubAttack, a jailbreak method that extracts this residual by learning orthogonal attack tokens composed of interpretable textual elements, and SubDefense, a plug‑and‑play defense that projects out the same subspace to suppress the residual. Experiments show SubAttack outperforms prior attacks in power and transferability, while SubDefense offers stronger robustness and better preservation of safe generation quality.

By Siyi Chen, Yimeng Zhang, Sijia Liu, Qing Qu
arXiv Computer Vision
Oct 7

Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition

The paper introduces Latent-Action-Guided Video-Language Feature Learning (LAG-VLFL) for recognizing surgical instrument–tissue interactions. By compressing frame-to-frame feature changes into latent actions and predicting next‑frame features, the method aligns video and textual action descriptions without requiring extra spatial or motion annotations. Experiments show that LAG-VLFL improves interaction grounding, temporal‑direction sensitivity, and achieves competitive recognition with faster inference and lower INT4 accuracy loss compared to V‑JEPA2/2.1.

By Jiajun Cheng, Sainan Liu, Subarna Tripathi, Xiaofan Yu, Shan Lin
arXiv Computer Vision
Oct 7

Deformable CT-US Registration via Anatomy-Aware Implicit Neural Representations

arXiv:2610.08419v1 Announce Type: new Abstract: Slice-to-volume registration between ultrasound (US) and preoperative computed tomography (CT) imaging would enhance many minimally invasive interventi...

By Agnieszka Lach, Magdalena Wysocki, Feng Li, Mohammad Farid Azampour, Benjamin D. Killeen, Felix Ginzinger, Mathias Braun, Philipp Steininger, Heinz Deutschmann, Nassir Navab
arXiv Machine Learning
Oct 7

Constant-Curvature Sliced Gromov-Wasserstein for Heterogeneous Cross-Curvature Alignment

The paper introduces Constant‑Curvature Sliced Gromov‑Wasserstein (CCSGW), a new divergence for aligning probability distributions on heterogeneous constant‑curvature spaces such as hyperbolic and spherical manifolds. It extends sliced Gromov‑Wasserstein by adding geodesic‑based one‑dimensional projections for spherical spaces, enabling efficient and principled comparison across manifolds with different curvatures while preserving intrinsic geometric relationships. The authors provide theoretical analysis showing that CCSGW controls intrinsic geometric discrepancy and demonstrate consistent performance gains when integrated into mixed‑curvature learning tasks like graph anomaly detection, node classification, and multimodal learning.

By Shanglin Li, Wenjing Lu, Muyang Li, Nicu Sebe, Ziheng Chen
arXiv Machine Learning
Oct 7

Neural Algorithmic Reasoning for Graph Saddle Point Problems

The paper introduces “GraphPDHG,” a message‑passing neural network that aligns with the Chambolle‑Pock Primal–Dual Hybrid Gradient algorithm to solve general graph saddle‑point problems. The authors prove that “GraphPDHG” can efficiently simulate PDHG and can learn an accelerated version of the algorithm. Experiments demonstrate that the learned warm start improves second‑order optimizers and that alignment with PDHG yields better size generalization compared to standard GNN baselines.

By Samantha Chen, Jesse He, Coleman Clougherty, Gal Mishne, Chester Holtz
arXiv AI
Oct 7

When better traffic forecasts fail to improve signal control: a layered diagnostic study of forecast-to-decision value

The study examines why improved traffic forecasts do not always lead to better signal‑control outcomes. Using 29 days of reconstructed traffic data from Xuancheng, China, the authors evaluate various forecasting methods—including point forecasts, conformal intervals, and dependence‑aware scenarios—within a closed‑loop control framework. While entry‑level and movement‑level forecasts reduce mean absolute error by about 4%, the resulting signal‑control decisions sometimes worsen queue vehicle‑seconds and spillback exposure, highlighting gaps between forecast accuracy and operational benefit.

By Jianing Long, Xiaobin Li, Wuming Lei, Weiguang Wang
arXiv AI
Oct 7

Evaluating Escalation Signals for LLM Routing: Targets, Controls, and Five Ways to Fool Yourself

The paper investigates whether semantic entropy—a measure of disagreement among a language model’s sampled answers—can serve as a cheap signal for deciding when to route a query from a small to a larger language model. Experiments on GSM8K and other benchmarks show that semantic entropy can distinguish small‑model mistakes and improve routed accuracy, but the authors also reveal that a simple question‑difficulty rule can mimic its performance and that other factors (definition of success, benchmark design, live sampling cost) can undermine its effectiveness. They propose a checklist of checks to validate escalation signals and demonstrate how to predict when a cached‑outcome approach will fail. "whyItMatters":"The study highlights the importance of rigorous evaluation of escalation signals, showing that seemingly promising metrics can be misleading without proper controls and that practical routing decisions must account for cost and benchmark design."

By Ramin Pishehvar, Andrea Morandi, Mahesh Viswanathan