AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv AI
Sep 30

Automated Screw Planning for Reduced Pelvic Fractures Based on Statistical Shape Models and Deep Learning

The article presents a fully automated pipeline for preoperative iliosacral screw planning in pelvic fracture patients, utilizing patient-specific 3D anatomy to identify safe screw corridors and generate individualized insertion trajectories. In a study of 200 clinical cases, the automated method increased the safety margin of safe insertion corridors by 2% across four screw types, reduced mean planning time by over 90%, and achieved a 95% clinical acceptance rate compared to conventional manual measurements.

By Yang Gao, Sutuke Yibulayimu, Yanzhen Liu, Zian Zhao, Yudi Sang
arXiv AI
Sep 30

ImbalancE: Inference-Time Latent Search Against Degree Imbalance in Link Prediction

The paper introduces ImbalancE, an inference‑time latent search method that mitigates degree imbalance bias in Knowledge Graph Embedding models. It targets the problematic prediction of target entities with much lower degrees than anchor entities, a common issue in recommender systems and other applications. Experiments on benchmark datasets show that ImbalancE improves predictions on the most imbalanced triples compared to conventional methods.

By Alberto Bernardi, Luca Costabello, Christophe Gueret
arXiv AI
Sep 30

Physics-Informed Multi-Agent Coordination for Hospital Patient Flow Optimization

The paper introduces a Physics‑Informed Multi‑Agent Coordination framework that embeds calibrated BCMP queueing topologies into a decentralized multi‑agent reinforcement learning system for hospital patient flow. It formulates the problem as a Decentralized Partially Observable Markov Decision Process with coupled resource constraints, enabling departmental agents to negotiate patient routing and service scaling while exchanging localized action fingerprints to handle non‑stationarity. Empirical tests on MIMIC‑IV data show the approach reduces cumulative system delay compared to static Markovian models, heuristic dispatching, and independent multi‑agent baselines, all while preserving clinical safety constraints.

By Guoqing Zhang, Rafik Hadfi, Takayuki Ito
arXiv AI
Sep 30

Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

The paper introduces two token-error supervision methods for sparse mixture-of-experts (MoE) large language models, aligning routing affinities with token-level cross-entropy loss. The first method predicts an error score per expert and uses it to adjust affinities before top‑K selection, while the second directly aligns router affinities to the model’s objective without extra heads or inference changes. Experiments on Granite and ARC‑Challenge show accuracy gains of about 2.3–2.94 percentage points over a parameter‑matched baseline, preserving the native sparse execution budget.

By Yury Nahshan, Nati Daniel, Jacob Goldberger, Yoli Shavit
arXiv AI
Sep 30

You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference

The paper demonstrates that in open‑weight LLM inference markets, selecting a model is insufficient; clients must also choose a provider, as the same model can differ markedly in quality, latency, availability, and price across providers. The authors propose a market‑aware routing approach, including a measured‑map policy and an online router called FACET, which certifies provider feasibility for each task and safely falls back to a reliable anchor. Experiments show that this strategy yields cost savings while maintaining quality and avoiding degraded endpoints.

By Liang He, Jingbo Wen, Yixiong Chen, Yue Yang, Qizhen Lan, Kangning Cui, Xilu Wang
arXiv AI
Sep 30

Agentic Commerce Bench: Measuring Fraud Detection for Agents That Spend Money

The paper introduces the Agentic Commerce Bench (ACB), a benchmark for measuring fraud in AI agents that autonomously spend money. It presents a taxonomy of agentic commerce fraud, a dataset of twenty fraud classes derived from real production data, and an open‑source detector stack called gordonguard for auditing and replaying hostile counterparties. The study shows that current reasoning layers and security scanners perform poorly on many classes, highlighting the need for better detection mechanisms.

By Ankit Srivastava, Debjyoti Paul
arXiv AI
Sep 30

PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions

The paper introduces PACT, a tuning method for single-token typed-decision models that leverages contrastive pair data to add four training terms—difference-in-differences margin, permutation-consistency, evidence-necessity, and ordinal transport cost—without requiring new annotations. PACT achieves comparable accuracy to existing recipes while reducing position bias and ordinal error, and it improves robustness and stability across seeds. The authors provide code, data splits, and trained adapters for reproducibility.

By Yida Lin
arXiv AI
Sep 30

PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

PADM'E is a method for synthesizing preference‑aligned data to meta‑evaluate language‑model (LM) evaluators of agentic behaviors. It reframes meta‑evaluation as a preference judgment problem, generating criterion‑based data with small LMs and no human involvement. In a prototype, PADM'E produced 1,000 samples across four domains and three criteria, and human validation showed agreement with human judgment rising from 73% to 85% compared to a naive baseline.

By Cheng Chang, Yining Mao, Peng Qi
arXiv AI
Sep 30

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

The paper investigates how the modality gap— the separation between image and text representations in contrastive vision‑language models—affects different downstream tasks. By showing that a single dominant direction accounts for most of the image‑text mean separation, the authors explain why reducing or removing this gap can improve zero‑shot classification, degrade retrieval, or restore performance depending on the task. The study provides a geometric framework that clarifies when and why gap interventions should be applied in vision‑language systems.

By Aditya Sharma, Divya Saxena
arXiv AI
Sep 30

Render Before Reading: Visual Rendering as a Prompt Injection Defense

The paper investigates how multimodal large language models are more susceptible to prompt injection when adversarial instructions are presented as text rather than as non-textual inputs like images. It proposes a training‑free defense that renders untrusted payloads into typographic images (or audio) before they reach the model, a method called Pictionary. Experiments on ten models and two benchmarks show that this approach significantly lowers attack success rates while maintaining normal functionality, and that fine‑tuning on image‑rendered instructions can further reduce the modality gap.

By Jie Zhang, Andrei Baroian, Jan N. van Rijn, Avital Shafran, Florian Tram\`{e}r
arXiv AI
Sep 30

Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models

The paper introduces MATE, a reinforcement‑learning‑based post‑training framework for unified multimodal models that lets the generation and understanding branches challenge each other instead of cooperating. In MATE, each branch proposes candidate outputs that the other must reproduce, and the solver is trained on the worst‑handled candidate, creating an evolving adversarial loop without a separate adversary. Experiments on Janus‑Pro‑1B show that MATE improves generation and understanding metrics, including GenEval (+2.4), DPG‑Bench (+1.7), and an average of nine understanding benchmarks (+0.7), while enhancing consistency across image‑text cycles.

By Wentao Zhou, Weijie Gan, Jiayun Wang
arXiv AI
Sep 30

Explainability from Training with Applications to TCR-Epitope Prediction

The paper introduces Explainability from Training (EFT), a model‑agnostic method that tracks how deep learning models learn and organize evidence during training. EFT is applied to four leading T cell receptor‑epitope prediction models, revealing distinct learning trajectories for CNNs and transformers, conflicts between TCR alpha and beta chain evidence, and differences in feature preferences when using real versus predicted structural data. The authors also present a new benchmark, TCR‑XAI2, comprising 388 experimentally resolved TCR‑epitope structures and several predicted models to evaluate these insights.

By Jiarui Li, Zixiang Yin, Samuel Landry, Zhengming Ding, Ramgopal Mettu
arXiv AI
Sep 30

Where Should Physics Enter a Molecular Crystal Generator?

The paper introduces CrystAF, an all‑atom crystal flow‑map generation model, and evaluates where physics should be incorporated into generative crystal structure models. By applying physics‑informed post‑training, the authors improve molecular validity and crystal packing without altering sampling speed, while inference‑time corrections further refine the structures. The study demonstrates that post‑training and inference‑time physics are complementary, and that the post‑training approach transfers to other generators such as Clari‑M and MolCrystalFlow.

By Haocheng Tang, Junmei Wang, Wengong Jin
arXiv AI
Sep 30

Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models

The study compares human and vision‑language model (VLM) responses to cross‑modal association tasks, using identical stimuli (a pseudo‑word and two images) and recording both choices and eye movements. While larger VLMs show some alignment with human choices, their attention patterns correlate poorly with human gaze, performing no better than a simple center‑bias baseline. Fine‑tuning VLMs on human choices improves choice alignment but not attention alignment, and training on human gaze improves attention correlation without affecting choice accuracy.

By Sumin Hong, Katsumi Ibaraki, Renee Shi, David Chiang, Toby Jia-Jun Li