AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv Computer Vision
Sep 7

Compositional Reward Models for Conditional Medical Image Generation

The paper introduces PRISM, a Compositional Reward Model framework that decomposes image quality into multiple verifier‑grounded stages for conditional medical image generation. By assigning distinct rewards for fine‑to‑coarse properties—such as intensity, texture, structural alignment, and semantic fidelity—and combining them via a Hierarchical Constrained Propagation mechanism, PRISM addresses shortcomings of single‑scalar reward approaches. Experiments on PanNuke, CeDeM, and ISIC datasets show that data generated with PRISM improves downstream model performance, achieving higher mDice, lower MRE, and increased F1 scores compared to baseline methods.

By Aayush Kumar Tyagi, Prathosh A. P., Mausam
arXiv Computer Vision
Sep 7

Reward-Aware Trajectory Shaping for Few-step Visual Generation

The paper introduces Reward-Aware Trajectory Shaping (RATS), a lightweight framework that aligns teacher and student latent trajectories at key denoising stages while adaptively regulating teacher guidance through a reward-aware gate. By allowing the student to optimize toward reward-preferred generation quality, RATS can surpass the teacher rather than merely imitate it. Experiments show that RATS improves the efficiency–quality trade-off in few-step visual generation, narrowing the gap between few-step students and stronger multi-step generators without extra test-time cost.

By Rui Li, Bingyu Li, Yuanzhi Liang, Haibin Huang, Chi Zhang, XueLong Li
arXiv Computer Vision
Sep 7

Towards patient-specific optimization for mandibular reconstruction planning based on predicted bone-union propensity

The paper introduces OsteoOpt++, an image‑to‑decision loop that uses pre‑operative CT scans to build a personalized digital twin of a patient’s mandible and then applies Bayesian optimization to adjust six surgical variables for improved bone‑union propensity at the donor‑host interface. In both generic defect models and patient‑specific cases, the optimized plans increased donor‑mandible apposition by up to 29 % and 26 % respectively compared to surgeon‑generated plans, and the predicted apposition closely matched year‑1 bone formation (Dice overlap 70–85 %). The study demonstrates the feasibility of using apposition‑derived predictions to evaluate and compare reconstruction options and provides open‑source code for further development.

By Hamidreza Aftabi, John E. Lloyd, Amanda Ding, Benedikt Sagl, Eitan Prisman, Antony Hodgson, Sidney Fels
arXiv Statistics ML
Sep 7

Optimal Stratified Allocation for Rare-Event Onset Forecasting in Dependent Sequences

The paper derives the exact finite‑population variance of a weighted risk estimator for rare‑event forecasting in dependent sequences and solves for the optimal stratified allocation of a small subsample. It shows that the optimal allocation is equal across strata, independent of the imbalance ratio, and provides a parameter‑free efficiency prediction A(π,f). The authors validate these theoretical predictions on a real‑world dataset of U.S. equities, demonstrating that the predicted ordering of sampling designs matches empirical results.

By Jaskaran Singh
arXiv AI
Sep 7

A Systematic Evaluation of Cross-Lingual Consistency Enhancement Methods in Multilingual Language Models

The paper presents a unified evaluation of cross‑lingual consistency (CLC) enhancement methods for multilingual language models, covering inference‑time interventions and post‑training approaches across three model families and three closed‑form benchmarks. Results indicate that post‑training methods, especially direct distribution alignment, consistently improve CLC across all model‑dataset combinations, while other methods are more sensitive to answer format and language coverage. The study also examines the impact of CLC enhancement on culturally diverse question answering, finding no systematic degradation in controlled settings but occasional accuracy drops in open‑ended generation, particularly for non‑English responses.

By Jirui Qi, Mingyang Wang, Hinrich Sch\"utze, Raquel Fern\'andez, Arianna Bisazza
arXiv AI
Sep 7

A Roadmap for MEG Foundation Models

The article outlines the emerging field of Magnetoencephalography (MEG) foundation models, explaining how these reusable, pretrained models can surpass traditional task‑specific decoding pipelines. It reviews current design choices—such as tokenization, sensor versus source representations, and self‑supervised objectives—and notes the limited number of existing MEG‑specific models and datasets. The authors propose a roadmap that includes native MEG pretraining, adaptation of EEG models, transfer from generic time‑series models, and multimodal integration with other neuroimaging and behavioral data, while emphasizing the need for coordinated infrastructure, rigorous evaluation, and responsible data‑sharing practices.

By Philipp Th\"olke, Hamza Abdelhedi, Yorguin Mantilla-Ramos, Fouad Lbakali, Oumayma Gharbi, Catherine Duclos, Annalisa Pascarella, Vanessa Hadid, Oiwi Parker Jones, Karim Jerbi
arXiv Machine Learning
Sep 7

SMILE: Bridging Continuous Optimization and Discrete Symbolic Recovery

SMILE (Sine, Multiplication, Identity, Logarithm, Exponential) is a hybrid framework that merges continuous gradient-based optimization with discrete symbolic recovery for symbolic regression. It operates in three stages: structural analysis to uncover the compositional hierarchy of the target expression, continuous optimization to learn parameters of a network using interpretable activations, and symbolic recovery via structured pruning, coefficient optimization, and rounding to produce a compact expression with exact symbolic constants. Evaluated on SRBench, SMILE achieves the highest symbolic solution rate under high noise, remains on the Pareto front of accuracy versus complexity, and recovers simpler expressions much faster than competing methods.

By Mansooreh Montazerin, Antonio Ortega, Ajitesh Srivastava
arXiv Machine Learning
Sep 7

GLASS: Graph-Language Alignment with Spherical Scoring for Transferable Graph-Level Anomaly Detection

GLASS is a graph‑level anomaly detection framework that aligns graph and language representations on a unit hypersphere to achieve cross‑domain transferability. It constructs a Graph Descriptor Prompt to encode local, global, and semantic graph properties, and uses a multi‑slice soft cosine objective to unify graph and text embeddings. Anomaly scoring is performed via spherical density estimation with von Mises‑Fisher kernels, enabling zero‑shot detection and few‑shot adaptation across twelve benchmarks and three meta‑domains, outperforming recent GLAD baselines.

By Xudong Wang, Chris Ding, Tongxin Li, Jicong Fan
Hugging Face Trending Papers
Sep 4

A Human-in-the-Loop Framework for AI-Assisted Scoring in Large-Scale Writing Assessment

The paper introduces a human‑in‑the‑loop framework for AI‑assisted scoring of short written responses in a large‑scale national assessment. Using data from two recent test editions with about 5,000 responses each, the study validates that AI-generated scores align moderately to highly with human raters across multiple rubric dimensions. The framework also identifies when human review is most needed, allowing more efficient allocation of expert effort while maintaining assessment quality.

Hugging Face Trending Papers
Sep 4

Cross-dataset transportability of pediatric chest X-ray deep learning across three countries: discrimination, calibration, operating-point failure, and limited-label recovery

The study evaluates a deep‑learning model for pediatric pneumonia detection across three countries, testing not only discrimination but also probability calibration, fixed operating‑point transport, shortcut signals, and limited‑label recoverability. Using a DenseNet121 ensemble trained on Guangzhou data, the model achieved high internal AUROC (0.976) but performance dropped to 0.798 and 0.742 on Bangladeshi and Vietnamese datasets, respectively. Limited‑label adaptation with Platt recalibration restored sensitivity but introduced significant specificity variability, highlighting the need for comprehensive cross‑dataset evaluation.

Hugging Face Trending Papers
Sep 4

CoLMIN: LLM-based Multi-Decision Path Negotiation for Cooperative Autonomous Driving

CoLMIN is an LLM-based framework for multi-decision path negotiation in cooperative autonomous driving. It introduces a Multi-Intent Negotiation module that generates multiple driving intentions, an Evaluation-based Shallow Reflection Module that provides feedback to accelerate consensus, and a Deep Reflection Module that mitigates cognitive fixation by reflecting on negotiation histories. Experiments in the CARLA simulation show that CoLMIN outperforms existing methods in challenging interactive driving scenarios.

arXiv AI
Sep 4

Representational alignment yields generalizable safety in language models

The paper argues that aligning large language models (LLMs) at the level of latent representations—specifically by matching their internal categorization of moral concepts to human prototype-based judgments—improves safety. Current alignment methods that focus on observable responses fail to preserve fine-grained moral categorization, leaving models vulnerable to adversarial rephrasings. By optimizing representational similarity, the authors demonstrate that LLMs can maintain more robust moral categorization and exhibit better adversarial robustness across multiple benchmarks and model sizes.

By Lingyu Li, Yan Teng, Yingchun Wang, Xia Hu
arXiv AI
Sep 4

HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning

The paper introduces Sand-Glass, a benchmark for evaluating translation under syllable-level duration constraints, and proposes Homura, a reinforcement learning framework that optimizes the trade-off between semantic preservation and temporal compliance. Homura uses a constrained reinforcement learning objective with a dynamic syllable-ratio reward to effectively control output length. Experimental results show that Homura outperforms strong baselines, achieving precise length control while maintaining semantic adequacy.

By Ziang Cui, Mengran Yu, Chenyu Shi, Yingxuan Shi, Tianjiao Li
arXiv Computation and Language
Sep 4

Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?

The paper evaluates uncertainty estimation (UE) methods for clinical vision‑language models (VLMs) on visual question answering (VQA). Across 8 UE techniques and 12 VLMs, UE quality tracks model accuracy, degrading where performance is weakest, and fails to signal uncertainty when models are stressed by hiding the correct answer (NOTA perturbations). However, UE on unperturbed inputs reliably predicts which predictions will collapse under NOTA, suggesting UE can diagnose model fragility.

By Arnisa Fazla, Alberto Testoni, Ameen Abu-Hanna, Barbara Plank, Iacer Calixto
arXiv Computer Vision
Sep 4

Auditing Patient Privacy in Medical Generative Models: Scalable Memorization Detection with DeepSSIM++

DeepSSIM++ is a self‑supervised similarity metric designed to audit memorization in medical generative models at scale. It aggregates multi‑scale features and uses anatomy‑preserving augmentations to create an embedding space where cosine similarity approximates SSIM, removing the need for exact pixel‑level registration. Compared to existing baselines, DeepSSIM++ improves Macro F1 by 33–46 percentage points and speeds up large‑scale similarity computation by several orders of magnitude.

By Antonio Scardace, Francesco Guarnera, Sebastiano Battiato, Daniele Rav\`i
arXiv AI
Sep 4

Value-Preserving Architectures for Agentic AI Systems

The paper "Value-Preserving Architectures for Agentic AI Systems" discusses how architectural choices in large language model-based multi‑agent systems (MAS) can promote human‑centered values such as privacy, fairness, and safety. It introduces three value‑preserving architectural patterns: a privacy‑aware federated topology, a distributed architecture that encourages pluralism and diversity, and a guard‑agent design to detect and mitigate unfairness. Representative use cases illustrate how these patterns can be applied in real‑world scenarios, aiming to provide guidelines for building trustworthy MAS.

By Alessandro Pesare, Tommaso Dolci, Katja Hose, Emanuel Sallinger
arXiv AI
Sep 4

InSituMeasure: Probing Situated Measurement Grounding in Industrial Scenes with Multimodal Large Language Models

InSituMeasure is a new benchmark that tests multimodal large language models on situated measurement tasks in industrial scenes. It includes 2,922 real monitoring images of eight types of engineering instruments, with detailed gauge-attribute labels and noise tags for failure diagnosis. The benchmark defines metrics for numerical accuracy, unit consistency, rejection of unanswerable queries, and alignment of model failures with annotated error factors, revealing that even the best models achieve only 25.7% joint value‑unit accuracy and 51.8% confidence‑diagnosis F1.

By Chao Shen, Xinyuan Li, Yunfan Zhou, Jianguo Yao, Haibing Guan, Zhihai Wang, Xijun Li
arXiv AI
Sep 4

WELD: The First Naturalistic Long-Period Small-Team Workplace Emotion Dataset for Ubiquitous Affective Computing

WELD is the first dataset that combines long‑term (30.1 months), naturalistic workplace recordings, a stable small‑team social structure, and a fully passive sensing protocol approved by institutional review boards. It contains 733,780 per‑frame seven‑class facial‑expression probability vectors from 49 employees of a Chinese software company, making it the longest in‑the‑wild emotion corpus that supports both within‑individual longitudinal and within‑team relational analyses. The authors validate the corpus by reproducing known affective phenomena and report four novel findings, including variance decomposition of daily valence, hidden Markov emotional regimes, turnover prediction metrics, and systematic over‑prediction of “angry” on neutral Asian faces by an off‑the‑shelf FER model.

By Xiao Sun