AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv Computation and Language
Sep 11

LLMAR: A Tuning-Free Recommendation Framework for Sparse and Text-Rich Industrial Domains

LLMAR is a tuning‑free recommendation framework designed for sparse, text‑rich industrial B2B domains. It transforms user behavioral history into structured semantic motives using LLM inference, employs a reflection loop to self‑correct hallucinations, and operates cost‑effectively with asynchronous batch processing. Experiments on MovieLens‑1M, Amazon Prime Pantry, and a construction risk dataset show LLMAR surpasses state‑of‑the‑art learning models, achieving up to a 54.6% nDCG@10 improvement while keeping inference costs around $1 per 1,000 users.

By Ryogo Hishikawa, Ichiro Kataoka, Shinya Yuda
arXiv Machine Learning
Sep 11

DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis

DeFiFlowBench is a benchmark comprising 207 natural‑language prompts for synthesizing DeFi workflows, evaluating graph coverage, configuration completeness, and declared safety predicates, and testing trade configurations on a local EVM. The study shows that direct, constrained, and few‑shot prompting still yield unsafe executions, and that a slippage bound derived from a quote does not prevent price impact. The proposed Koan‑Safe system—combining a prompt‑only intent parser, a replaceable generator, and structural repair—achieves a higher static safety proxy score and records no unsafe executions on the benchmark, while ablation studies reveal the limits of default safety thresholds and the need for explicit trade protections. whyItMatters":"The results demonstrate that current prompting methods can still authorize costly trades and that explicit safety mechanisms like Koan‑Safe are necessary to prevent unsafe DeFi workflow executions."

By Abhinav Rajeev Kumar, Harshit Arora, Varun Singh, Manikandan Nanjappan
arXiv Machine Learning
Sep 11

Predicting Privacy Leakage from Weight Spectral Density

The paper investigates whether inexpensive spectral metrics from the heavy‑tailed self‑regularisation framework can predict membership inference attack (MIA) vulnerability, offering a scalable alternative to costly shadow‑model attacks. Experiments on image and tabular classification tasks show that stable rank correlates positively with overall MIA success, while Log alpha‑Norm correlates negatively with MIA risk in low false‑positive regimes, outperforming conventional generalisation gap measures. These findings suggest that neural network spectra contain privacy leakage signals not captured by traditional overfitting metrics, pointing to spectral analysis as a promising direction for privacy auditing.

By Richard J. Preen, Jim Smith
arXiv Machine Learning
Sep 11

Dynamic language model representations for multi-objective reaction optimisation

The paper introduces a method that learns dynamic reaction representations directly from textual descriptions using a fine‑tuned language model coupled with Gaussian process surrogates. This approach enables multi‑objective Bayesian optimisation for chemical reactions, achieving faster convergence than traditional descriptor libraries or one‑hot encodings across nickel‑, palladium‑, and iridium‑catalysed systems. Prospective experiments on a palladium‑catalysed cyanation and an asymmetric hydrogenation produced high‑yield, high‑enantiomeric‑excess conditions after only two rounds of high‑throughput testing, translating directly to gram‑scale synthesis.

By Joshua W. Sin, David Ming Segura, Bojana Rankovi\'c, Siu Lun Chau, Marius D. R. Lutz, Andrea Anelli, Ryan P. Burwood, Kurt P\"untener, Maximilian J. Notheis, Raphael Bigler, Philippe Schwaller
arXiv Machine Learning
Sep 11

From Protocols to Evidence: Bounded Claims for AI in Service of the Common Good

The paper argues that AI should be evaluated not only by principles but by concrete protocols that translate commitments into roles, requirements, records, oversight, and assessment. It introduces a rupture test linking institutional baselines to system evaluation, and distinguishes evidence‑bounded deployment from measurement‑bounded governance. The authors propose the RISE AI architecture to make bounded, evidence‑based claims about Responsibility, Inclusivity, Safety, and Empowerment, emphasizing the need for engineering, institutional repair, and ongoing moral judgment.

By Nitesh V. Chawla, Paulo Benanti
arXiv Machine Learning
Sep 11

Adaptive Diffusion Freezing: Privacy-preserving Diffusion Models Against Membership Inference Attacks

Adaptive Diffusion Freezing (ADF) is a new privacy‑preserving framework for diffusion models that protects against membership inference attacks (MIAs). It uses cross‑timestep adaptive freezing training, where a mask matrix controls which data subsets participate at each diffusion timestep, reducing over‑memorization and aligning model behavior for member and non‑member samples. A pretraining‑based risk‑aware freezing policy estimates MIA risk and suppresses high‑risk subset‑timestep pairs, achieving a superior privacy‑utility‑efficiency trade‑off across multiple datasets.

By Jialu Guo, Xiao Han, Junjie Wu
arXiv AI
Sep 11

An Autonomous GeoAI Agent for Arctic Eco-Navigation

An autonomous GeoAI agent for Arctic eco‑navigation is presented, integrating operational, physical, ecological, and community criteria into a unified routing framework. The system employs multiple specialized agents for data acquisition, multi‑objective route generation, and skyline‑based decision support, explicitly accounting for exposure to sensitive ecological areas such as Essential Fish Habitat and seal critical habitat. By keeping value judgments under human control, the framework aims to produce safer, more transparent, and socially responsible Arctic navigation routes.

By Samira Alkaee Taleghan, Younghyun Koo, Farnoush Banaei-Kashani
arXiv AI
Sep 11

Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery

The paper introduces a black-box framework for evaluating agentic AI systems, focusing on multi-step vulnerabilities that standard single-turn tests miss. It presents a seven-domain taxonomy linking observable behaviors to risk categories, an automated SAGE-RT red-teaming process generating 120 adversarial scenarios per domain, and a human-validated evaluation using LLM judges. Empirical tests on CrewAI and AutoGen agents show significant governance, privacy, and behavior risks, demonstrating the framework’s ability to uncover critical architectural weaknesses without privileged access.

By Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi
arXiv AI
Sep 11

Structural Process Supervision for Latent Chain-of-Thought Reasoning

The paper introduces Prototype-Mediated Process Supervision (PMPS), a method that uses learnable reasoning prototypes to provide structural supervision for latent chain-of-thought embeddings, addressing representation collapse and uneven information distribution. PMPS aligns latent and explicit CoT embeddings in a shared prototype space via many-to-many soft assignment and employs a Progressive Sequential Alignment module to guide training from positional priors to adaptive matching. Experiments show PMPS reduces output token length to under 50% of explicit CoT on GSM8K-Aug and improves accuracy by 2.08% over SIM-CoT, even surpassing CoT-SFT on GPT-2 and achieving the highest accuracy among latent reasoning methods on larger models and harder tasks.

By Yiqi Li, Xu Chen, Chen Ju, Jiangchao Yao, Zhaoyang Li, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Yu Wang
arXiv AI
Sep 11

Quantifying Logical Consistency in Transformers via Query-Key Alignment

The paper introduces a lightweight evaluation method for logical reasoning in transformer-based language models, using query-key alignments within attention heads to compute a QK-score. This single forward-pass technique identifies valid versus invalid inferences and is validated across multiple reasoning benchmarks, showing robustness to distractors and deeper reasoning. Experiments span models from 1.5B to 70B parameters, demonstrating scalability.

By Eduard Tulchinskii, Anastasia Voznyuk, Laida Kushnareva, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
arXiv AI
Sep 11

CS-Guard: Benchmarking LLM Guardrails for Code Generation Security

CS-Guard is a new benchmark that systematically evaluates guardrails for code generation security, covering 1,000 malware-generation prompts, 7 jailbreak attacks, and a novel fictional scenario attack (FSA) for text-to-code generation, as well as 331 code prompts for code-to-code generation. The study empirically tests nine guardrails across seven large language models, finding that many guardrails fail to prevent malicious code generation, with attack success rates reaching about 50% for text-to-code and up to nearly 100% for code-to-code and FSA scenarios. CS-Guard introduces a modular three-layer guardrail taxonomy and releases its benchmark and data to support future research.

By Jinyang Li, Mingyu Guo, Hung X. Nguyen
arXiv AI
Sep 11

A Trust-Network-Based Federated Learning Framework for Multi-Center Aging Clock Prediction

The paper introduces TNFL, a trust‑network‑based federated learning framework designed for multi‑center aging clock prediction. TNFL propagates models along directed trust relations without centralized aggregation, combining an age‑aware mixture‑of‑experts model with generative replay to mitigate forgetting and drift. Experiments on multiple molecular datasets demonstrate effective aging‑clock prediction with limited local data, interpretable age‑dependent patterns, and stable performance across interaction orders, while revealing coordinated higher‑order protein subnetworks linked to aging.

By Chunxu Zhang, Bo Li, Wenliang Wang, Yang Liu, Di Jiang, Yuan Huang, Yo-ichi Nabeshima, Akinori Yamamura, Bo Yang, Qiang Yang
arXiv AI
Sep 11

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

The paper introduces Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer‑selective unlearning framework for large language models. FOM-UL uses a forget‑to‑retain significance score to identify transformer layers that strongly influence the forget set while being insensitive to the retain set, allowing targeted updates that preserve most of the model. Experiments on TOFU, KnowUnDo, and MUSE-style benchmarks show that FOM-UL reduces residual memorization and maintains utility better than several baselines, even after 8‑bit and 4‑bit post‑training quantization, and it also limits recovery of forgotten content in adversarial prompt tests.

By Ravi Ranjan, Olivera Kotevska, Agoritsa Polyzou
arXiv AI
Sep 11

Influence-Oriented Personalized Federated Learning

Influence-Oriented Personalized Federated Learning (FedC^2I) introduces a framework that quantifies both client-level and class-level influence to enable adaptive parameter aggregation in federated learning. By modeling inter-client influence through influence vectors and matrices, FedC^2I allows clients to selectively acquire knowledge from similar peers and guides the aggregation of feature representations and classifiers. Experiments under non-IID settings show that FedC^2I outperforms existing federated learning methods in effectiveness, robustness, and interpretability.

By Yue Tan, Guodong Long, Jing Jiang, Chengqi Zhang
arXiv Machine Learning
Sep 11

Artificial Intelligence Algorithms for the Detection of Pathologies Related to Lung Cancer through Image Analysis using Convolutional Neural Networks and Data Augmentation: a systematic mapping of the literature

The article reviews recent applications of artificial intelligence and deep learning, particularly convolutional neural networks with transfer learning and data augmentation, for detecting lung cancer in medical images. It surveys 96 studies from 2015 onward, highlighting high sensitivity and specificity achieved by these methods. The authors also identify challenges such as data standardization, model explainability, patient privacy, and ethical concerns that must be addressed before clinical deployment.

By Pablo Ramirez Amador
arXiv Machine Learning
Sep 11

A Multi-Stage Rule-Chaining Framework for Compositional and Interpretable Cognitive Reasoning

The paper introduces a multi-stage rule‑chaining framework for the Abstraction and Reasoning Corpus (ARC), aiming to model cognitive generalization by inferring abstract rules from few examples. It combines three solvers—a deterministic rule discovery module, a pattern‑composition engine, and a structural abstraction layer—executed sequentially in a fallback hierarchy that reuses earlier reasoning traces to improve interpretability and generalization. The system achieved over 95% accuracy on ARC tasks, demonstrating strong performance across deterministic, compositional, and abstract categories.

By Deblina Kar
arXiv Machine Learning
Sep 11

The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystems

The paper investigates whether an oligopolistic concentration of generative AI models accelerates or steers the phenomenon of model collapse when models are recursively trained on each other’s outputs. Using controlled ecosystems of 13 open‑source models and an injected probe that pushes one model’s market share to 90%, the authors find that varying market concentration has little effect on the speed or final state of collapse. Instead, the pace of collapse is largely determined by which models supply the training pool and how susceptible those models are to being carried along, with human‑written text in the pool roughly halving the drift.

By Yangze Liu, Zhongyi Han
arXiv AI
Sep 11

CT-SAFR: Safe and Interpretable Chain-of-Thought Reasoning for Autonomous Robots: A Multi-Layered Verification Framework for Trustworthy AI-Driven Robotic Decision Making

CT‑SAFR is a multi‑layered verification framework designed to enhance the safety and faithfulness of Chain‑of‑Thought reasoning in autonomous robots. The framework achieves a 94.2% hallucination detection rate with sub‑500 ms latency, and a warehouse robot case study shows an 87% reduction in unsafe reasoning outputs. The study also offers recommendations for responsible deployment of reasoning‑capable autonomous robots.

By Cagri Temel
arXiv Computer Vision
Sep 11

TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents

TeleOCR is a unified framework for document parsing that tackles challenges in both decoupled and end-to-end Vision‑Language Models. It introduces deformation‑aware learning to handle geometric distortions, an adaptive sampling mechanism for complex layouts, and a content‑structure decoupled strategy to model formula grammars and table structures. The approach achieves state‑of‑the‑art results on multiple benchmarks, including top placement in the ICDAR 2026 Sci‑ImageMiner Challenge.

By Peng Cai, Zhaofan Zou, Shifa Liu, Yikun Wang, Jiawei Tang, Kaicheng Yang, Meng Tong, MingKun Jiang, Zhongjiang He, Hao Sun
arXiv AI
Sep 11

Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning

The paper introduces a multimodal in‑context learning framework that uses contrastive demonstration modeling to align large language models’ responses with the required reasoning paths. By contrasting suboptimal and better responses and incorporating a response‑conditioned retrieval mechanism, the method explicitly guides models beyond surface imitation. Experiments on various multimodal tasks, especially visual question answering, show consistent performance gains.

By Mingbo Yang, Wenqiang Wang, Zhaolu Kang, Peng Chen, Yannan Chen, Sunshang Wang, Yan Xiao