AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv AI
Sep 18

MAGS: Multi-agent Auto-formalization Guarantees Safety for Agentic Outputs

The paper introduces MAGS, a multi-agent framework that automatically generates executable programs with formal safety guarantees. MAGS translates LLM-generated code into the verification-aware language Dafny, repairs any safety violations using verifier feedback, and then compiles the verified code back into executable form. Evaluations on 220 diverse examples—including CUDA kernels, terminal scripts, and robotic-arm tasks—show a 100% success rate in producing programs that meet frozen safety specifications, with additional safety and functional tests confirming strong performance across domains.

By Albert Wu, Nicholas Roberts, Tzu-Heng Huang, Haoran Lin, Gil Friedman, Sungjun Cho, Gabriel Orlanski, Frederic Sala
arXiv AI
Sep 18

Closed-World Resolution Against Tool Hallucination in LLM Agents

The paper investigates a new failure mode of tool‑augmented large language model agents: calling non‑existent tools with arguments that do not match any declared schema. It introduces a five‑class taxonomy of tool hallucination, presents a training‑free closed‑world resolver that checks registry membership and signatures, and demonstrates that hallucinations persist across ten hosted models and various invocation surfaces, including the Model Context Protocol. The authors release a Hallucinated‑Tools Benchmark to enable comparison of resolver methods.

By Laxmipriya Ganesh Iyer
arXiv AI
Sep 18

Neuro-Symbolic Agentic AI for Networked Low-Altitude UAVs

The paper proposes Neuro‑Symbolic Agentic AI (NSAAI) as a framework that blends neural grounding, symbolic reasoning, and closed‑loop interaction to enhance decision‑making for networked low‑altitude UAVs. It outlines NSAAI’s strengths in data efficiency, compositional generalization, continual learning, and zero‑shot transfer, and presents a reference architecture covering task management, planning, verification, skill execution, and network interaction. An urban fire‑inspection simulation demonstrates how a UAV can coordinate sensing, cloud access, and verified image‑delivery skills under intermittent connectivity, illustrating NSAAI’s potential for reusable skills, evidence‑grounded decisions, and adaptive mission execution.

By Yuqi Ping, Tianhao Liang, Nanchi Su, Guangyu Lei, Junwei Wu, Qinyu Zhang, Tingting Zhang
arXiv AI
Sep 18

Can Data Attribution Filter Out Subliminal Learning? Not Reliably

The paper investigates whether gradient‑based data attribution methods can filter out subliminal learning in language models. Three attribution techniques—GradCos, a contrastive variant, and EK‑FAC—are evaluated on three models and compared to divergence tokens, a strong baseline that requires counterfactual teacher models. Results show that EK‑FAC can mitigate some subliminal effects at the token level, but overall the methods are inconsistent and generally less effective than divergence tokens, especially when filtering entire samples.

By Moritz Weckbecker, Sweta Jena, Jonas M\"uller, Ponnurangam Kumaraguru, Sebastian Lapuschkin, Wojciech Samek, Louis Jaburi, Gon\c{c}alo Paulo
arXiv AI
Sep 18

NeuSOGA3D: A Neuro-Symbolic Framework for Explainable 3D Geometric Reconstruction

NeuSOGA3D is a hybrid neuro‑symbolic framework that reconstructs 3D geometry from unorganized point clouds by combining learned perceptual priors with explicit symbolic geometric reasoning. It projects point clouds onto orthographic planes, builds symbolic implicit spline representations, and fuses them via shape‑preserving constructive solid geometry to produce a coarse visual hull. Additional detail is added through cross‑sectional decomposition and volumetric reconstruction with Partial Shape‑Preserving Splines, yielding CAD‑compatible, structurally meaningful models across all 40 ModelNet40 categories.

By Qingde Li, Qingqi Hong, Zihan Li, Jie Tian
arXiv AI
Sep 18

Q&A on Any Spreadsheet Requires Interpreting Its Grid Structure

The paper proposes a new framework that improves spreadsheet chunking for large language model (LLM)-driven retrieval-augmented generation (RAG) systems by adding semantic cell annotations. This approach outperforms current state‑of‑the‑art methods but is limited by the inherent two‑dimensional, unstructured nature of spreadsheets, which cannot be fully captured by finite classification categories. The authors argue that future progress requires dimensionality‑reduction techniques to flatten spreadsheets into one‑dimensional text, simplifying downstream RAG interpretation and generation.

By Zofia Smole\'n
arXiv AI
Sep 18

Optimal Transport Metric Learning for Feature Alignment in Partially Supervised Segmentation

The paper proposes a two‑stage learning framework for multi‑organ segmentation that handles partially annotated datasets and domain shifts. First, the model learns accurate segmentations from available annotations to build robust feature representations. Second, it introduces learnable organ prototypes and a Sinkhorn‑triplet loss to enforce organ‑wise feature consistency across datasets, keeping embeddings of the same organ close while separating different organs, even when annotations are missing.

By Dakini Mallam Garba, Salim Abdou Daoura
arXiv AI
Sep 18

The AR Fairness Metamodel: A Structured Framework for Fairness Measures

The paper introduces the AR Fairness Metamodel, a structured framework for representing, analyzing, and comparing fairness scenarios. It incorporates key elements such as agents, resources, and their attributes, and supports both discrete and continuous fairness measures—including equality, equity, group fairness, individual fairness, the Gini index, the Theil index, Jain's fairness index, and a specific measure for Australia's Child Care Subsidy. The metamodel builds on the Tiles framework, offering modular components that can be connected to capture diverse fairness definitions, and includes formal proofs of relationships among group fairness, individual fairness, and envy‑freeness. An open‑source implementation of the Tiles framework is provided to facilitate practical fairness modeling and evaluation across various applications.

By Julian Alfredo Mendez, Timotheus Kampik
arXiv AI
Sep 18

From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning

The survey "From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning" reviews over 300 works on efficient multimodal learning (EML), proposing a structured taxonomy that spans model, algorithm, and system layers. It synthesizes how cross‑layer co‑design addresses the Efficiency‑Utility‑Privacy trade‑off and illustrates this through a case study of multimodal large language models. The paper also offers optimization blueprints for various domains, discusses a shift toward self‑regulating intelligence, and outlines open challenges for future EML research.

By Pan Wang, Siwei Song, Hui Ji, Siqi Cao, Heng Yu, Zhijian Liu, Huanrui Yang, Yingyan Celine Lin, Beidi Chen, Mohit Bansal, Xiaoming Liu, Pengfei Zhou, Ming-Hsuan Yang, Tianlong Chen, Jingtong Hu
arXiv AI
Sep 18

SoK: Trading Agents or Market Crashers? Dissecting Robustness and Security Failures in Academic Financial LLM Trading Schemes

The paper introduces FARSIGHT, a framework for evaluating the robustness and security of financial trading agents powered by large language models. It assesses agents on their resilience to market turbulence, such as flash crashes, and their vulnerability to three types of attacks: on information sources, on the agents themselves, and on agents acting as attackers. Applying FARSIGHT to 15 academic trading schemes reveals that 80% fail at least one robustness test and all exhibit security weaknesses, highlighting the risk of market-wide crashes from both accidental misjudgments and deliberate attacks.

By Mengxiao Wang, Nitesh Saxena
arXiv AI
Sep 18

Reproducing Transparent and Scrutable Recommendations: Exploring Open-Weight Models via Natural-Language User Profiles

This reproducibility study confirms that incorporating generated natural‑language user profiles into recommender systems enhances transparency and allows users to directly intervene by correcting preferences or addressing cold‑start issues. The authors replicated the original findings and extended the evaluation with context ablation, multi‑seed stability tests, and mechanistic interpretability analysis using nnsight. Their results show that while perturbing profiles shifts predicted ratings uniformly across genres, the overall rankings remain unchanged, attributing this to the rating‑regression objective rather than the profile interface.

By Noah Mami\'e, Laurin van den Bergh
arXiv AI
Sep 18

Governance-as-Code: Translating EU AI Act Technical Requirements into Executable Compliance Pipelines for Generative AI Systems

The paper introduces Governance-as-Code (GaC), a framework that translates the EU AI Act’s technical requirements into 43 machine‑checkable acceptance criteria across six compliance modules. GaC runs within a CI/CD pipeline, producing Article‑indexed audit evidence and providing actual Rego policy code. The authors validate GaC on two enterprise deployments, showing it reproduces manual audit findings—including three penalty‑triggering violations—while reducing audit labor by about 75%.

By Rudrendu Kumar Paul, Sourav Nandy
arXiv AI
Sep 18

Music Hallucination in Audio-Language Models: A Hierarchical Formulation and Empirical Study

The paper presents the first music‑specific, layer‑wise empirical study of hallucination in audio‑language models, framing it as a hierarchical perceptual grounding failure across five layers: sound events, temporal properties, tonal attributes, style, and emotion. It introduces MuseDiag, a diagnostic framework that evaluates nine models and finds universal vocal misperception, significant tonal perception differences, and identifies Audio‑Flamingo‑3 as the most stable model. The study also proposes two training‑free mitigation methods, ADD‑M and TPA, which reduce hallucination in probing but show variable effectiveness in free‑form generation, highlighting the need for multi‑paradigm evaluation.

By Yu Liu, Jiahui Liu, Zhilin Liu, Cong Cao, Fangfang Yuan, Yuling Yang, Pin Xu, Yanbing Liu
arXiv AI
Sep 18

Risk-Set Transported Synthetic Control with Difference-in-Differences Adjustment under Staggered Treatment Adoption

The paper introduces Risk-Set Transported Synthetic Control with Difference-in-Differences Adjustment (RT‑SC‑DiD), a method for staggered treatment‑adoption studies that keeps the donor pool fixed by reallocating weights from exiting donors to similar surviving donors while applying a DiD baseline correction. It analyzes distortion from horizon‑by‑horizon re‑optimization, derives bounds on error propagation, and proposes diagnostics and a donor‑only placebo for tuning the transport penalty. Empirical simulations show that intermediate transport regularization reduces average RMSE compared to independent horizon‑specific estimation and strong anchoring, supporting the method’s bias‑variance trade‑off. "whyItMatters":"The method offers a principled way to stabilize synthetic‑control weights over time in staggered designs, potentially improving causal inference when donor support contracts as treatments roll out."

By Mojtaba Eslami
arXiv AI
Sep 18

LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Reality

The paper introduces an automated pipeline that converts non‑critical driving scenes into safety‑critical scenarios by integrating computer vision, Large Language Models (LLMs), and Augmented Reality (AR). It detects and tracks road users, extracts safety features such as distance, velocity, motion direction, and Time‑to‑Collision (TTC), and evaluates scene criticality. Safe scenes are then modified by an LLM, which generates realistic collision‑inducing objects and behaviors that are overlaid onto the original scene using AR, achieving 97.52% safety classification accuracy on the nuScenes dataset and producing realistic scenarios like pedestrian crossings, rear overtaking vehicles, and sudden‑stop events.

By Noura Fady, Farah Khaled, Catherine M. Elias
arXiv AI
Sep 18

Stress-testing Alignment Midtraining

The paper "Stress-testing Alignment Midtraining" examines the effectiveness of alignment midtraining (AMT), a technique that continues pretraining on alignment-relevant data to improve generalisation beyond post‑training methods. Experiments on models up to 110 billion parameters and 1 billion midtraining tokens reveal that AMT can steer a model’s motivation in simple scenarios, but its effects are quickly overridden by even a tiny fraction of finetuning data with a competing motivation. The study also shows that rule-following requires demonstrations in either the midtraining or post‑training datasets to be robustly learned, leading the authors to conclude that current public evidence is insufficient to confirm that AMT resolves the core alignment challenges of powerful AI systems.

By Sid Baines, Jonathan Bostock, Maria Angelica Martinez, Andrew Draganov, David Africa, Daniel Tan
arXiv AI
Sep 18

Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation

The paper introduces SafeHarness, an obstacle‑aware framework that improves the safety of coding agents for robot manipulation. By decomposing tasks into route planning and contact execution, the harness enables the agent to prioritize collision avoidance, achieving 71.9% task success and 87.5% collision avoidance—significantly better than prior methods. The study demonstrates that safety constraints can be effectively integrated into language‑model‑driven robot controllers.

By Bingxin Xu, Yuzhang Shang, Zhen Dong, Emilio Ferrara
arXiv AI
Sep 18

Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance

The paper introduces a method for selecting the best vision‑language model for a downstream task by analyzing the internal dynamics of the visual encoder. It represents each task with layer‑wise conductance and uses an entropy‑regularized alignment to derive a target‑conditioned block importance distribution. The proposed Directional Conductance Divergence (DCD) metric captures asymmetric transferability, enabling accurate prediction of model rankings without direct inference, and achieves a 14.7% NDCG@5 improvement over SWAB on 48 VLMs across 21 datasets.

By Wei Yang, Hong Xie, Tao Tan, Xin Li, Defu Lian, Enhong Chen
arXiv AI
Sep 18

AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulation

AntiGrounding is a visual action-selection framework that turns short robot trajectories into both executable motion plans and rendered prompts for vision‑language model evaluation. After filtering for feasibility, each trajectory is scored on safety, task alignment, efficiency, and physical plausibility using structured multi‑view visual question answering, and the best trajectories are refined and validated by a digital twin before real‑world execution. In eight real‑world manipulation tasks, the system achieved a 71.25% success rate with a single GPT‑6 Astra evaluator, outperforming baseline methods.

By Wenbo Li, Yiteng Chen, Wenhao Li, Qingyao Wu
arXiv AI
Sep 18

Information-Geometric Inverse Distillation for Enhancing Adversarial Transferability

The paper introduces Inverse Knowledge Distillation (IKD), an attack‑agnostic technique that enhances adversarial transferability by maximizing the discrepancy between benign and adversarial prediction distributions on a surrogate model. IKD employs a CE/KL‑equivalent soft‑label objective to push adversarial predictions away from a fixed benign anchor, leveraging Fisher‑sensitive surrogate directions. The authors provide theoretical analysis showing CE and KL induce identical gradients, derive a lower bound on Fisher‑subspace overlap, and demonstrate through extensive ImageNet experiments that IKD consistently improves black‑box attack performance across CNN, ViT, and defended models.

By Wenyuan Wu, Yuan Sun, Yingke Chen, Chao Su, Xi Peng, Dezhong Peng, Xu Wang