AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv AI
Sep 11

Non-Stationarity Breaks Permutation Surrogates in Multi-Agent Reinforcement Learning: Diagnosis and Remedies

The paper evaluates information‑theoretic measures for detecting directed influence in multi‑agent reinforcement learning by testing a guardrail in two games—a social dilemma and a coordination race—over 100 seeds. It shows that omitting the non‑stationary training transient leads to near‑perfect false‑positive rates, while simply excluding the transient is insufficient. The authors propose a block‑wise permutation null model that maintains the data structure and achieves low false‑positive rates (≈5%) while remaining highly sensitive to injected links.

By Nikolaos Al. Papadopoulos, Konstantinos E. Psannis
Hugging Face Trending Papers
Sep 10

Domain-Specific Hallucination Detection in Large Language Models

The paper introduces a multi‑signal pipeline for detecting hallucinations in large language model outputs, combining fine‑tuned DeBERTa‑v3 classification, Monte Carlo Dropout uncertainty, and temperature‑scaled calibration. On the HaluEval benchmark it achieves strong performance (F1 = 0.915, AUROC = 0.977) and further improves accuracy to 93.2% with MC Dropout. The authors also demonstrate that applying Direct Preference Optimization to a Qwen2.5‑0.5B generator reduces hallucination rates from 85.5% to 37.7%, and show that domain‑specific fine‑tuning (PubMedBERT on SciFact) yields better results than general‑domain training.

Hugging Face Trending Papers
Sep 10

RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety

RAG-Safety-Bench is a benchmark designed to evaluate how retrieval-augmented generation (RAG) affects the safety of large language models (LLMs). It isolates safety impacts by testing four conditions: non-RAG, RAG with an oracle document, RAG with related but non-answer documents, and RAG with random safe documents. Results on five open-source LLMs reveal an inverse relationship between benign and unsafe capabilities, show that standard safety guardrails do not guarantee safety in RAG, and confirm that even benign documents can trigger unsafe outputs.

Hugging Face Trending Papers
Sep 10

LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation

LOCUS is a post‑training technique that selects a task‑aware low‑rank adaptation subspace to reduce token‑cost while preserving utility. By updating only a tiny fraction of parameters (0.24–0.28 %) on 3 B‑parameter backbones, LOCUS cuts continuation length by up to 39.84 % on Pythia‑2.8B and 14.87–17.58 % on Qwen2.5‑3B, without materially affecting the internal preference diagnostic.

Hugging Face Trending Papers
Sep 10

The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge

The Eloquence team presents three strategies for the Interspeech 2026 MLC‑SLM Task 2, a multilingual MCQA challenge across 21 languages. They fine‑tune Voxtral‑Mini‑3B with LoRA and cross‑lingual augmentations, achieving 0.72 macro‑accuracy; they use multimodal in‑context learning on the frozen Voxtral‑24B to correct label bias, reaching 0.81; and they deploy a training‑free retrieval system with a voice‑anchored memory, scoring 0.68. All approaches surpass the official baseline.

Hugging Face Trending Papers
Sep 10

FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation

FreeFlow is a hierarchical transformer for optical flow estimation that eliminates traditional flow-specific inductive biases such as correlation volumes and feature warping. It relies on a single feed-forward encoder–decoder architecture that incorporates window attention, shifted-window attention, and reduced-resolution global attention. This design allows the model to scale with capacity while achieving state‑of‑the‑art accuracy on Sintel, KITTI‑2015, and Spring benchmarks, all while remaining memory efficient at 1080p inference.

Hugging Face Trending Papers
Sep 10

Improving the Sensitivity of Gravitational Wave Detection with Weighted Conformal Prediction

The paper proposes a weighted conformal prediction framework to combine outputs from multiple gravitational‑wave detection pipelines, providing statistically rigorous confidence estimates for candidate events. By incorporating likelihood‑ratio reweighting, the method corrects for covariate shift between simulated training data and real observations, restoring well‑calibrated coverage. Experiments on mock datasets show that this approach improves sensitivity, especially for events near the detection threshold, enabling recovery of true signals that would otherwise be missed.

Hugging Face Trending Papers
Sep 10

Portable Semantics, Private Dialects: Reuse and Negative Transfer in Latent Communication Between Language-Model Cells

The study investigates whether independently trained language‑model societies share a common packet language and how inherited interface states affect learning. A comprehensive audit of 30 pairwise interactions among six restricted societies shows that most cross‑initialization pairs fail to interoperate, with only one pair achieving full bidirectional compatibility. Further experiments reveal that reinitializing only the packet reader, writer, and mouth dramatically improves accuracy, while inherited interfaces never outperform fresh ones by the preregistered margin.

Hugging Face Trending Papers
Sep 10

Order-Aware 2.5D Multiple Instance Learning for Preoperative MRI-Based Perineural Invasion Risk Assessment in Intrahepatic Cholangiocarcinoma

The paper introduces Order-Aware Slab Multiple Instance Learning (OAS‑MIL) for predicting perineural invasion (PNI) in intrahepatic cholangiocarcinoma using preoperative T2‑weighted MRI. It treats each tumor‑centered MRI crop as an ordered sequence of overlapping 2.5D slabs, extracting slab‑level features with a shared encoder and aggregating them through both permutation‑invariant set‑attention and bidirectional sequence‑attention branches. In five‑fold patient‑level cross‑validation, OAS‑MIL achieved a mean AUROC of 0.770, outperforming volumetric and MIL baselines, indicating that axial order provides a useful inductive bias for weakly supervised PNI prediction.

Hugging Face Trending Papers
Sep 10

Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment

The paper introduces LogiMed‑RoB, a benchmark that tests large language models (LLMs) on hierarchical logical consistency in medical risk‑of‑bias assessments, using 860 randomized controlled trials and 14,820 queries based on Cochrane RoB 2.0 expert logic. It evaluates models across four dimensions—Atomic Consistency, Domain Consistency, Aggregation Consistency, and Evidential Faithfulness—revealing a severe Error Compounding Effect where high atomic accuracy does not translate to end‑to‑end consistency. Experiments on ten state‑of‑the‑art LLMs show that even top performers can collapse to 45.13% overall consistency, with some models nearly failing entirely, and that many models struggle to deduce correct outcomes from retrieved evidence.

Hugging Face Trending Papers
Sep 10

When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text

Large language models used as judges for social bias are affected by noisy text, such as typos and broken punctuation. In experiments with 3,822 stereotype-related responses, noise more often turns neutral judgments into biased ones than the reverse, with up to a 120‑fold difference. The effect is strongest at mild realistic noise levels and leads to systematic overestimation of bias, especially in fairness‑critical categories.

arXiv AI
Sep 10

Evidence-Aligned Entity Verification for Hallucination Detection in Retrieval-Augmented Generation

The paper introduces Evidence-Aligned Entity Verification (EAEV), a method for detecting entity-level hallucinations in retrieval-augmented generation (RAG). EAEV aligns generated entities with retrieved evidence across three dimensions and uses counterfactual stability analysis to maintain robust alignments when evidence changes. Experiments on multiple RAG benchmarks show that EAEV consistently outperforms existing hallucination detection methods and generalizes well.

By Runsong Jia, Zhen Fang, Mengjia Wu, Jie Lu, Yi Zhang
arXiv AI
Sep 10

CodeTD: Topology of Attention Detects Hallucinations in Code LLMs

The paper introduces CodeTD, a novel method that uses topological data analysis of attention maps from code language models to pre‑execution assess code correctness and detect hallucinations. It quantifies prompt‑generation mismatch through topological patterns and is evaluated on multiple benchmarks (HumanEval, MBPP, BigCodeBench, MultiPL‑E) across five programming languages and ten Code LLMs up to 34B parameters. Results show CodeTD outperforms recent baselines and transfers well between coding benchmarks.

By Daria Voronkova, Ilya Trofimov, Anton Dmitriev, Eduard Tulchinskii, Evgeny Burnaev, Serguei Barannikov
arXiv AI
Sep 10

How to Verify Probabilistic Consistency of Predictive Models

The paper presents an interactive probabilistically checkable proof (PCP) protocol that allows a polynomial‑time verifier to check the approximate consistency of a probabilistic predictor defined by two circuits, P and Q. By evaluating these circuits at a few points and querying a proof oracle that encodes a witnessing probability distribution, the verifier can confirm that the predictor’s many conditional‑probability claims are self‑consistent. The authors also establish that the problem of verifying l₂‑approximate consistency for explicit probabilistic claims lies in NP, with certificates of size O(mn + log B), and show how to eliminate dependence on the input bit‑precision B through a small additive gap.

By Orr Paradise, Oliver Richardson, Yoshua Bengio, Shafi Goldwasser
arXiv AI
Sep 10

Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Tasks

The paper examines how Denoising Diffusion Probabilistic Models (DDPMs) perform on globally constrained discrete tasks such as Sudoku and N-queens. It shows that standard diffusion sampling, which keeps updates close to the noisy state, often preserves early mistakes, whereas sampling directly from the model’s clean predictions dramatically improves validity (e.g., Sudoku from 31% to 95%). The authors further introduce self‑correction training, exposing the model to its own predictions to reduce inference errors, which enhances the performance of standard samplers across tasks.

By Mariia Drozdova, St\'ephane Liem Nguyen, Fran\c{c}ois Fleuret
arXiv AI
Sep 10

NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting

NOAH is a generative transformer that learns the full multimodal patient journey by integrating bidirectional time and a variational latent space. Trained on over 559 million clinical events from 431,000 hospital visits, it processes medical images, time‑series, numeric signals, categorical events, and both structured and unstructured records. The model supports autoregressive forecasting, zero‑shot classification, and counterfactual intervention simulation, yielding strong performance on clinical outcomes, ICD chapters, comorbidities, and time‑to‑event prediction.

By Tobias Susetzky, Raphael Rehms, Dmitrii Seletkov, \"Ozg\"un Turgut, Michelle Espranita Liman, Lisa Steinhelfer, Rickmer Braren, Daniel Rueckert
arXiv AI
Sep 10

Hypersolid: Emergent Vision Representations via Short-Range Repulsion

The paper introduces Hypersolid, a self‑supervised learning objective that uses short‑range repulsion to prevent representation collapse. It combines view alignment with local collision avoidance, creating a latent geometry of compact, semantically aligned neighborhoods with low anisotropy. This geometry improves unsupervised clustering and fine‑grained separation, though it reduces transferability.

By Esteban Rodr\'iguez-Betancourt, Edgar Casasola-Murillo
arXiv Machine Learning
Sep 10

CoRL: Co-Evolutionary Reinforcement Learning for Adaptive Indirect Prompt-Injection Attacks and Defenses

The paper introduces CoRL, a co-evolutionary reinforcement learning framework designed to defend against adaptive indirect prompt-injection attacks on tool-augmented language agents. CoRL operates in three stages—attacker fine‑tuning, bilateral Co‑PPO training, and defender fine‑tuning—using verifier‑grounded repairs to adapt to changing attack strategies. Experiments on 1,514 executions show that CoRL reduces attack success rates to 0% while improving task utility, demonstrating its effectiveness against adaptive adversaries.

By Boyang Zhang, Qingxin Xiao, Lingwei Dang, Qingyao Wu
arXiv Machine Learning
Sep 10

Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems

The paper reports an empirical scalability study of data‑parallel training for Kolmogorov‑Arnold Networks (KANs) on high‑performance computing systems. Using up to eight NVIDIA A100 GPUs across four nodes on the FinisTerrae III supercomputer, the authors evaluate strong and weak scaling, communication overhead, and model‑size scaling, finding a 74.7% parallel efficiency and a 5.97× speedup at eight GPUs. They observe non‑monotonic communication costs driven by All‑Reduce choices and inter‑node latency, and note that while the parameter‑to‑memory ratio improves with larger models, training time scales less favorably, leading to guidelines for GPU topology and model‑size selection.

By Guangneng Chen, David Garcia Selfa, Pablo Quesada Barriuso