AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv AI
Oct 31

Agentic process evidence

The article discusses how AI agents are transforming software development by autonomously creating pull requests, selecting dependencies, refactoring code, interpreting test failures, and managing infrastructure across the SDLC. It highlights the lack of accountability for these transient processes, noting that traditional tools like git logs and governance frameworks assume human decision‑making, which no longer applies. To address this, the authors propose Agentic Process Evidence (APE), a signed, tamper‑evident, machine‑readable attestation that links AI agent sessions to their outcomes, enabling policy‑as‑code controls and human oversight only when necessary.

By Yuval Kainan, Carmit Hershman
arXiv AI
Oct 31

Training language models in simulation for psychiatric intake

The study demonstrates that multi‑turn reinforcement learning (RL) in simulation environments significantly improves a language model’s performance on structured, text‑based psychiatric intake tasks. In a randomized, double‑blind crossover trial involving 60 matched scenario pairs, the RL‑trained agent outperformed its base model in 76.7% of cases, showing higher scores on clinical examination, patient‑centered communication, and differential diagnosis appropriateness. Across 127 validated simulated scenarios, the trained agent nearly doubled the base model’s clinical rubric score and reduced under‑triage of acute safety risks by more than half, indicating that simulation provides a controlled testbed for training and stress‑testing psychiatric conversational agents.

By Samuel Schmidgall, Tim Strother, Alex Bijamov, Min Woo Sun, Anil Palepu, Akshay Goel, Ulysse Marteau-Ferey, Th\'eo Moutakanni, Yubin Kim, Joseph Breda, Christina Chen, Justin Chen, Paul Jhun, Andrew Marshall, Sarah Montgomery-Taylor, Eun Kyoung Hong, Shashank Ravi, Kameron C. Black, David J. H. Wu, Austin Schoeffler, John Lyon Havlik, Elizabeth C. Stade, Jared Moore, Johannes C. Eichstaedt, Logan Pierce, Jeff Jopling, Johannes Thrul, Thomas H. McCoy, Neguine Rezaii, Mark S. Chen, Ellen E. Fitzsimmons-Craft, Shirley Wang, Desmond C. Ong, Katharina Schultebraucks, Torrey Creed, Bhavna Daryani, Aleksandra Faust, S. Sara Mahdavi, Caroline Figueroa, Eric Lin, Kate H. Bentley, Ernest Lim, Zhihan Jiang, Xuhai "Orson" Xu, Daniel McDuff, John Torous, Yun Liu, Po-Hsuan Cameron Chen, Mike Schaekermann, Joelle Barral, Alvin Rajkomar, Tao Tu, Lin Yang, Isaac Galatzer-Levy, David Racz, Valentin Li\'evin
arXiv AI
Oct 31

FVH-Bench: Evaluating Image Generator Safety Beyond Overtly Harmful Imagery

The paper introduces FVH-Bench, a benchmark designed to evaluate image generators for functional visual harm (FVH)—the risk posed by images that enable dangerous actions or deception rather than merely depicting overtly harmful content. FVH-Bench contains 669 prompts across five harmful visual functions and ten risk domains, and tests 11 image generators, many of which rarely refuse these prompts and often produce artifacts that fulfill the targeted harmful functions. The study also shows that existing safety detectors provide limited coverage of FVH, highlighting gaps in current safety alignment assessments.

By Zihao Zhu, Yuhan Liu, Siwei Lyu, Baoyuan Wu
arXiv AI
Oct 31

Poisoned Skills: Propagating Jailbreaks Through Agent Markdown Files

The paper investigates whether Markdown skill files used to configure AI agents can serve as an injection channel for jailbreaks. By embedding a jailbreak into a LangChain-based question‑answering agent’s system prompt, the authors demonstrate that the attack success rate jumps from 0.5% under a benign prompt to an average of 45%, with some models reaching 68.9%. Two obfuscation techniques—zero‑width characters and phonetic transformations—do not outperform plain‑text injection, and the phonetic variant actually lowers success rates.

By Kashyap R Manjusha, Murali Emani
arXiv AI
Oct 31

Responsible Quantum Machine Learning for Medical Imaging: A Fairness Audit Across Sensitive Subgroups

The study audits fairness in quantum and classical machine‑learning classifiers for medical imaging, focusing on dermatological lesions (Fitzpatrick skin type) and chest X‑rays (age group). Using ResNet‑18 embeddings, the authors train SVM, MLP, QSVM, and QCNN under baseline and mitigation strategies (Equalized Odds regularisation, Gradient Reversal Learning, and both combined). Results show that while quantum models can match aggregate performance, they often exhibit subgroup disparities, and some mitigation approaches reduce fairness without maintaining clinical performance.

By Giovanni dos Santos Franco, Felipe Mahlow, Ellison Fernando Cardoso, Jos\'e Adenaldo Santos Bittencourt, Felipe Fanchini
arXiv AI
Oct 31

OPERA: OPamp Design by an Expert-Taught Reasoning Agent, from Specifications to Layout

OPERA is an expert‑taught reasoning agent that uses a large language model to design CMOS opamps from specifications to post‑layout sign‑off with open‑source tools. The agent first learns the gm/ID sizing method, then the gLayout framework, and writes a reusable layout generator that closes the layout‑extraction loop. In the SKY130 process, OPERA designed a shared constant‑gm bias and six different opamps that meet all target specifications across 15 PVT corners, with DRC‑clean and LVS‑matched layouts.

By Vishal Saxena, Soumyajit Mandal
arXiv AI
Oct 31

Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence

The paper introduces two algorithms for safe multi‑objective reinforcement learning that guarantee zero constraint violation during learning. The first algorithm learns a safe Pareto‑optimal policy for a given preference by combining optimistic reward estimates with pessimistic cost bounds, achieving an ×O(1/√T) suboptimality gap. The second, preference‑free algorithm safely explores first and then computes a safe policy for any preference, retaining the same safety guarantee and achieving an ×O(1/√T+1/√K) suboptimality gap.

By Chenxi Li, Yao Shu, Siyang Gao, Shuang Qiu, Guangxin Jiang
arXiv AI
Oct 31

Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

The paper introduces MedCBR, a two‑stage alignment framework for medical vision‑language models that improves diagnostic reasoning at the capability boundary. First, it uses Difficulty Weighted Boundary Preference Optimization (DBPO) to focus learning on counterfactual boundary cases, and second, it applies Entropy Gated Grounded Policy Optimization (EGPO) to reward anatomical grounding and reasoning fidelity while mitigating rubric noise. Experiments on six medical benchmarks demonstrate consistent accuracy gains, especially on challenging differential diagnosis tasks, with the combined stages delivering super‑additive improvements.

By Hongyu Bai, Jiaqi Zeng, Chunzheng Zhu
arXiv AI
Oct 31

Exploring the Potential of Symbolic Regression in Precision Livestock Farming

The study explores explainable AI for precision livestock farming by using acceleration sensor data to predict farrowing time in sows. It first detects nest‑building behavior in 166 of 180 sows, then applies symbolic regression to forecast the remaining time to farrowing, achieving a mean absolute error of 9.28 hours. Symbolic regression outperformed traditional closed‑box models such as support vector regression and neural networks in both accuracy and interpretability.

By Elisabeth Mayrhuber, Maciej Oczak, Michael Iwersen, David Brunner, Viktoria Dorfer, Stephan Winkler
arXiv AI
Oct 31

When to spend privacy: optimal allocation in private learning

The paper presents a framework for optimally allocating a fixed privacy budget across iterations of Differentially Private Stochastic Gradient Descent. By modeling the Gaussian noise-variance schedule as an open-loop control problem under a zero-Concentrated Differential Privacy constraint, the authors derive an exact-budget simplex formulation and introduce the Privacy‑Budget Adjoint Mirror Descent (PB‑AMD) algorithm. The method guarantees budget exhaustiveness, stability, and stationarity, and yields a square‑root allocation law that empirically outperforms existing allocation rules and reduces optimization time compared to projected gradient descent.

By Umberto Biccari, Yongcun Song
arXiv AI
Oct 31

Understanding and Mitigating Role Spoofing in LLMs

The paper investigates how large language models (LLMs) can be tricked into attributing text to the wrong role, a phenomenon called role spoofing. It shows that LLMs rely on both content cues (semantics and style) and structural cues (control tokens, formatting) to track who said what, making them vulnerable to attacks that combine fake structure and content. The authors demonstrate that adversarial training can mitigate content-based attacks but not structure-based ones, and propose a linear-probe method to detect role spoofing across settings.

By Ezra Edelman, Benjamin Edelman, Surbhi Goel, Eran Malach
arXiv AI
Oct 31

Agentic AI for Clustering and Relationship Discovery in Prediction Markets

The paper introduces a hierarchical agentic reduction (HAR) pipeline that constructs inspectable cross‑event relation graphs from prediction market contracts. By clustering markets and applying typed LLM reasoning, the method proposes signed relations with rationales between specific markets. Evaluation on eight monthly Kalshi catalogs, using 30.3 million trades, shows that the identified relations exhibit higher signed Pearson and Spearman price‑change correlations than matched within‑topic controls, especially for medium or high‑strength relations.

By Agostino Capponi, Alfio Gliozzo, Brian Zhu
arXiv AI
Oct 31

Trustworthy AI and Democracy: A Dual Taxonomy and Operational Framework of Democratic Risks and Contributions

The paper introduces the AIRD/AIPD framework, a dual taxonomy that categorises AI’s democratic risks (threats to autonomy, fairness, public discourse, institutions, and trust) and its positive contributions (enhancing efficiency, accessibility, security, informed citizenry, transparency, accountability, inclusive dialogue, and evidence‑based policymaking). Grounded in democratic principles and the European Commission’s Trustworthy AI requirements, the framework offers an operational process for identifying contexts, classifying effects, mapping to governance mechanisms, and defining responsible actors, illustrated with examples on electoral misinformation and public service allocation.

By Oier Mentxaka, Natalia D\'iaz-Rodr\'iguez, Mark Coeckelbergh, Marcos L\'opez de Prado, Emilia G\'omez, David Fern\'andez Llorca, Enrique Herrera-Viedma, Francisco Herrera
arXiv AI
Oct 31

Robust Taylor-Lagrange Control for Safety-Critical Systems with Guaranteed Feasibility

The paper introduces a robust Taylor‑Lagrange Control (rTLC) method that extends the safety function to a higher order using Taylor’s expansion with Lagrange remainder. This approach ensures that safety constraints remain satisfied throughout each discretization interval, thereby preserving feasibility in safety‑critical control problems. The authors demonstrate the effectiveness of rTLC on an adaptive cruise control scenario and compare it with existing methods.

By Wei Xiao, Christos Cassandras, Anni Li
arXiv AI
Oct 31

Latent Style-based Quantum Wasserstein GAN for Drug Design

The paper introduces a latent style-based quantum Wasserstein GAN (QGAN) for drug design, integrating a variational autoencoder with a hybrid quantum GAN to generate molecular structures. The QGAN employs style-based noise reuploading and a Wasserstein gradient penalty to enhance trainability and reduce mode collapse, achieving comparable performance to classical models while using far fewer trainable parameters. Experiments on noiseless simulators and an IBM Heron quantum computer demonstrate the model’s capability to sample thousands of new molecules and reveal quantum noise effects.

By Julien Baglio, Yacine Haddad, Richard Polifka
arXiv AI
Oct 31

Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey

The survey titled "Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey" reviews how Mixture-of-Experts (MoE) frameworks address multimodal learning problems. It categorizes MoE’s contributions into three roles: an efficient multimodal engine that scales without excessive computation, a representation learner that fuses diverse expert knowledge, and an adapter that handles imperfect data such as missing modalities. The paper also highlights research gaps like interpretable routing, expert communication, and lifelong multimodal learning.

By Liangwei Nathan Zheng, Wenhao Liang, Wei Emma Zhang, Lin Yue, Weitong Chen
arXiv AI
Oct 31

Can I Take Another Dose? DoseBench: Evaluating LLM Decision-Making in OTC Dosing QA

DoseBench is a new benchmark consisting of 230 consumer‑style questions about whether a person can safely take another dose of acetaminophen or ibuprofen. The study evaluates eight large language models, finding that overall accuracy ranges from 44.8 % to 62.2 %, but models with similar accuracy can make vastly different dosing decisions, with some incorrectly approving up to 60.5 % of prohibited doses. The authors show that providing more guidance or drug facts can improve accuracy but may also cause models to overlook missing information, underscoring the difficulty of applying safety rules correctly in health QA.

By Maroof Kousar, Jinsung Han, Mohit Chandra, Munmun De Choudhury, Yibo Hu
arXiv AI
Oct 31

Whom Do AI Agents Serve? Role-Induced Sponsorship Bias in Decision Models and Generative LLMs

The paper investigates how AI shopping agents—whether powered by generative large language models or non-generative decision models—evaluate sponsored product listings. It finds that when the agent is described as serving a booking platform rather than a traveler, the penalty for paid listings is reduced, revealing a sponsorship bias. The bias persists across different model types, prompts, and reasoning levels, but can be mitigated by naming the consumer as the beneficiary or by fine‑tuning models to align with consumer interests.

By Davood Wadi, Yu Ma
arXiv AI
Oct 31

SkillCT: A Multidimensional Benchmark and Execution-Free Baseline for Agent Skill Assessment

SkillCT is a new benchmark that expands agent skill assessment beyond downstream task execution to a multidimensional evaluation across nine dimensions grouped into Static, Dynamic, and Human-Assisted assessments. It measures specification quality, behavioral robustness, semantic alignment, and responsibility boundaries without requiring repeated executions. The benchmark also introduces an execution-free baseline that predicts skill quality profiles by comparing to similar reference skills, reducing token usage and evaluation time while enabling skill refinement and repository maintenance.

By Mengyu Sun, Zunlong Zhou, Ziyuan Yang, Yi Zhang
arXiv AI
Oct 31

Disparity Laundering: Consensus Without Fairness in Clinical Multi-Agent Systems

The paper investigates how clinical large language models, when deployed as multi‑agent teams, fail to eliminate demographic bias through deliberation. Across five model families and four aggregation topologies on 120 MIMIC‑IV emergency cases, every configuration amplified or relocated bias, giving Black patients more acute triage. The authors term this phenomenon "disparity laundering" and show that only by making counterfactual demographic versions visible and pooling diverse perspectives at every deliberation round can the bias be substantially reduced without retraining or architectural changes.

By Xiaoyang Wang, Tianrui Wang, Christopher C. Yang