AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv AI
Sep 18

MAS-Shield: A Defense Framework for Secure and Efficient LLM MAS

MAS-Shield is a defense framework for Large Language Model–based Multi-Agent Systems that uses a coarse‑to‑fine filtering pipeline. It first selects critical agents, then applies lightweight auditing to most cases, and finally escalates only suspicious signals to a heavyweight committee. Experiments show a 92.5% recovery rate against adversarial attacks and a latency reduction of over 70% compared to existing methods.

By Kaixiang Wang, Zhaojiacheng Zhou, Bunyod Suvonov, Jiong Lou, Zihan Wang, Yuxiang Zheng, Yidan Lin, Wutong Zhang, Xianghan Kong, Chentao Wu, Jie Li
arXiv AI
Sep 18

VLM-CAD: VLM-Optimized Collaborative Agent Design Workflow for Analog Circuit Sizing

The paper introduces VLM-CAD, a workflow that uses Vision Language Models (VLMs) for analog circuit sizing while mitigating spatial blindness and logical hallucinations. It incorporates a neuro‑symbolic parsing module, Image2Net, to convert schematics into topological graphs and JSON, and an Explainable Trust Region Bayesian Optimization method, ExTuRBO, to guide design decisions with sensitivity evidence. Experiments on 12 sizing tasks across six circuits and four technology platforms show a Strict Pass@1 of 23.3% and a Relaxed Pass@1 of 91.7%.

By Guanyuan Pan, Shuai Wang, Yugui Lin, Tiansheng Zhou, Pietro Li\`o, Zhenxin Zhao, Yaqi Wang
arXiv AI
Sep 18

Perturbing the Phase: Analyzing Adversarial Robustness of Complex-Valued Neural Networks

The paper introduces Phase Attacks, a novel adversarial technique that targets the phase component of complex-valued inputs in complex-valued neural networks (CVNNs). It also extends traditional adversarial attacks to the complex domain and compares CVNNs with real-valued neural networks (RVNNs). The results show that CVNNs can be more robust in some cases, yet both architectures are highly vulnerable to phase perturbations, with Phase Attacks causing greater performance degradation than equivalent magnitude‑based attacks.

By Florian Eilers, Christof Duhme, Xiaoyi Jiang
arXiv AI
Sep 18

Modeling Human Behavior with Type Vectors Using AI

The paper presents an AI-based method that models human behavior by assigning a language model a vector of trait intensities—called a type vector—and asking it to predict actions in various settings. By adjusting traits such as Altruism, Risk Aversion, Fairness, and Trust, the authors fit the model to 119,147 decisions from 78,657 subjects across 35 countries and 10 economic games, finding that three dimensions (Risk Aversion, Strategic Sophistication, and Trust) closely match human choices. The resulting type vectors cluster into fewer than a dozen groups and can predict behavior in new games with different rules, demonstrating the method’s generalizability and interpretability.

By Matthew O. Jackson, Benjamin S. Manning, Yutong Xie, Walter Yuan, Qiaozhu Mei
arXiv Machine Learning
Sep 18

Enhanced Agriculture-informed Neural Network by Domain Knowledge

The paper introduces KAINN, a hybrid neural‑mechanistic model that augments the Agriculture‑informed Neural Network with domain knowledge on fertilizer diffusion, soil respiration, and water‑filled porosity to predict nitrous oxide emissions from agriculture. Experiments across CNN, LSTM, and Transformer architectures show that KAINN achieves lower root mean square error, lower mean absolute error, and higher R-squared values compared to purely data‑driven models and the original AINN. The learned interfaces exhibit smoother, more physically consistent parameter trajectories with reduced uncertainty.

By Ci Lin, Futong Li, Rose Chong-Wu, Tet Yeap, Iluju Kiringa
arXiv Machine Learning
Sep 18

Opinion Dynamics-based Coalition Formation for Federated Learning in Heterogeneous IoT Systems

The paper proposes an opinion‑dynamics approach to form client coalitions in federated learning for heterogeneous IoT systems, specifically targeting smart‑meter water‑consumption forecasting. By modeling coalition formation as a Hegselmann‑Krause bounded‑confidence process on local model weights, the method creates stable coalition structures within ten iterations without extra client computation or communication. Experiments on a real smart‑metering dataset show that this HK‑based coalition formation reduces mean absolute error by up to 54% compared to FedAvg and achieves the highest global accuracy (83‑85%).

By Mohammed El Hanjri, Anas Abouaomar, Hamidou Tembine, Abdellatif Kobbane
arXiv Machine Learning
Sep 18

AURA: Adaptive Uncertainty-Routed Analysis for Email Threat Detection

AURA: Adaptive Uncertainty-Routed Analysis for Email Threat Detection is a multimodal system that evaluates both email content and embedded URLs to detect spam and phishing. It uses a two-layer approach: first, a URL classifier estimates prediction uncertainty, and only messages with high uncertainty are passed to a fine-tuned transformer encoder for deeper semantic analysis. Evaluated on eight diverse training corpora and two real-world datasets covering a decade of attacks, AURA achieves a macro F1-score of 0.9858 in-distribution and maintains scores above 0.94 on the NazPhish-Eval and GuenterTrap-Eval datasets, demonstrating strong generalization to new attack scenarios.

By Omran Berjawi, Walid fahs, Rida Khatoun
arXiv Machine Learning
Sep 18

Graph-Based Stochastic Power-UCT: Monte-Carlo Graph Search with Power Mean Estimation

Graph-Based Stochastic-Power-UCT (GS-Power-UCT) is a Monte‑Carlo graph search algorithm that shares states reached at the same planning depth while keeping separate values for different depths, thereby reducing duplicate simulations in stochastic MDPs. The method guarantees that, for a fixed horizon, the root estimate converges to the finite‑horizon value at an $O(n^{-1/2})$ rate, matching tree‑based Stochastic‑Power‑UCT but with improved sample reuse. Two full‑state variants—GS‑Power‑UCT‑F and GS‑Power‑UCT‑F$^+$—extend the approach to single‑node per physical state and adaptive horizons, respectively, with the latter converging to the optimal infinite‑horizon discounted value when cross‑depth bias vanishes. Experiments on stochastic planning benchmarks demonstrate that GS‑Power‑UCT outperforms both tree‑based and other graph‑based baselines in sample efficiency.

By Tung Tran, Viet Bao Mai, Hoang Ta, Tuan Dam
arXiv Machine Learning
Sep 18

QoS-Aware Federated Learning for Multimodal In-Cabin Interaction in Smart Vehicles

The paper introduces FedQoS, an asynchronous federated learning framework designed for multimodal in‑cabin interaction in smart vehicles. It uses a two‑phase gating mechanism: a resource‑aware training gate that starts local learning only when sensing buffers and energy reserves meet safety thresholds, and a QoS‑aware transmission policy that gates uplink updates based on an efficiency score balancing model novelty, latency, and energy costs. Experiments on vehicular datasets show FedQoS achieves competitive personalized accuracy with only marginal loss compared to FedAvg, while reducing communication overhead by 76.7% and latency cost by 26.0%.

By Baran Can G\"ul, Mert Nak{\i}p, Nasser Jazdi, Michael Weyrich
arXiv Machine Learning
Sep 18

Robust Federated Q-Learning with Almost No Communication

The paper introduces Robust Fed-Q, a federated Q‑learning algorithm designed for settings where multiple agents interact with a shared Markov Decision Process and communicate through a central server. It combines model‑based and model‑free reinforcement learning techniques with a median‑of‑means strategy from robust statistics to handle a small fraction of adversarial agents. The authors prove that Robust Fed-Q achieves exact convergence to the optimal value function with high probability, attains near‑optimal finite‑time rates that benefit from collaboration, and requires only “~O(1)” communication rounds per guarantee.

By Sreejeet Maity, Aritra Mitra
arXiv Machine Learning
Sep 18

How Far Can Sub-3B Open Language Models Go in Zero-Shot Essay Scoring on an 8 GB Consumer GPU?

The study evaluates zero‑shot essay scoring using sub‑3B open language models that run locally on a single 8 GB consumer GPU. Four instruction‑tuned models (Qwen2.5‑0.5B, 1.5B, 3B and SmolLM2‑1.7B) were tested on all eight ASAP‑AES prompts, comparing rubric‑decomposed versus holistic prompting, different aggregation methods, and trait‑mapping strategies. Results show rubric‑decomposed prompting consistently outperforms holistic prompting, trait‑mapping is sensitive to calibration, and longer essays reduce error, yet the best local configuration (macro QWK 0.388) still falls short of human agreement and a length‑only baseline, suggesting these models are best suited for formative, human‑supervised feedback.

By Nguyen Dung Son, Dang Quang Minh, Nguyen Huu Loi, Truong Viet Vu, Nguyen Thai Anh
arXiv Machine Learning
Sep 18

Score Centering Stabilizes Off-policy Reinforcement Learning

The paper introduces a method called score centering to address the training‑inference mismatch (TIM) that destabilizes reinforcement learning for large language models. By adding an additive correction term that cancels drift between training and inference engines, score centering stabilizes RL and can match or surpass importance‑sampling techniques, especially as model size and mismatch severity increase. The approach also composes with importance sampling, yielding further performance gains in staleness experiments.

By Martin Marek, Max Ryabinin
arXiv Machine Learning
Sep 18

ALIBI: Adversarial Legitimacy Injection in Binary Input against LLM Malware Analyzers

The paper introduces ALIBI, a semantic cover story attack that injects a small, non-executed read‑only section into compiled binaries to mislead large language model (LLM) malware analyzers. By embedding a coherent but false security narrative, ALIBI can cause LLMs such as Gemini 2.5 Pro, GPT‑5.5 Pro, and Claude Opus 4.7 to downgrade or flip the verdicts of malicious samples. The attack also transfers to ELF binaries, and even a verification‑guided defense prompt only partially mitigates the effect, leaving a significant portion of malicious samples classified as benign.

By Hyeongjun Choi, Wonyoung Jung, Haehoon Seo, Sungyup Nam
arXiv Machine Learning
Sep 18

Delphi Scanner: efficient and interpretable static malware detection via API sequence modeling

Delphi Scanner is a static malware detection system for Windows PE files that balances efficiency and interpretability. It employs a convolutional neural network to model Windows API sequences and a rule‑based interpretation layer to map APIs to high‑level malicious capabilities. Tested on over 190,000 PE files, it achieves 95.35% accuracy with a 1.53 MB model, and demonstrates robustness against out‑of‑distribution samples and adversarial manipulations.

By Bijied Brahimi, Vincent Cohadon, Gabriel Glazman, Rayan Al Mohaize, Omran Berjawi, Rida Khatoun
arXiv Machine Learning
Sep 18

Correlation-Free Transition Path Sampling through Shooting Point Generation Guided by Committor Learning

The paper introduces GenAIMMD, an iterative algorithm that learns the committor function and trains a conditioned Boltzmann Generator to generate uncorrelated transition paths without prior knowledge of the reaction coordinate. This method combines transition path sampling with committor learning, enabling fully parallelizable sampling. Benchmarks on a toy model and a polymer system show a substantial performance improvement over standard TPS.

By Maximilian Negedly, Sebastian Falkner, Alessandro Coretti, Christoph Dellago
arXiv Machine Learning
Sep 18

MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Driving

MILER is an end‑to‑end reinforcement learning framework that achieves zero‑shot sim‑to‑real transfer for autonomous driving in unstructured environments. It uses a custom semantic mid‑level representation (MLR) simulator for offline training, and during deployment it processes real camera and LiDAR data with BEVFusion to produce a compatible bird’s‑eye‑view representation. The policy’s actions are applied via a trajectory‑alignment strategy, allowing the system to drive 17.3 km on a 3.0 km test track without human intervention, all running on a Jetson AGX Orin.

By Thomas Steinecker, Denis Trescher, Alexander Bienemann, Thorsten Luettel, Mirko Maehlisch
arXiv Machine Learning
Sep 18

OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher

OPTED is a method for on‑policy fine‑tuning of end‑to‑end driving models that separates reinforcement learning from the policy update. A privileged teacher trained with RL on vectorized inputs (HD‑maps and bounding boxes) supervises the pre‑trained student during closed‑loop post‑training. Applied to the camera‑based models TransFuser and VaVAM in AlpaSim, OPTED boosts driving scores by 1.6× and 9.5×, respectively, while requiring roughly three orders of magnitude fewer simulator interactions than direct RL post‑training.

By Damiano Da Col, Maximilian Igl, Peter Karkus, Kashyap Chitta, Boris Ivanovic, Marco Pavone, Konrad Schindler, Christos Sakaridis
arXiv Machine Learning
Sep 18

A Generative-AI Modeling Framework for Explainable Decision Support in Complex Geosteering Scenarios

The paper presents a real‑time, AI‑driven geosteering workflow that combines Generative Adversarial Networks for geological parameterization, ensemble methods for model updating, and dynamic programming optimization for decision support during directional drilling. The framework uses offline GAN training to generate realistic geology realizations and a Forward Neural Network to predict Logging‑While‑Drilling tool responses, enabling progressive reduction of subsurface uncertainty around the drilling bit. Tested on a low‑net‑to‑gross drilling scenario, the prototype delivers steering recommendations and automatically maps formation boundaries along the well path.

By Sergey Alyaev, Kristian Fossum, Hibat Errahmen Djecta, Jan Tveranger, Ahmed H. Elsheikh
arXiv Computation and Language
Sep 18

What Users Think of Generative AI: A Cross-Platform NLP Analysis of Trust and Friction in App Store Reviews

The study analyzes 17,012 app‑store reviews for six major generative‑AI apps, using BERTopic and RoBERTa to uncover topics and sentiment. Negative sentiment is most common around advertising, authentication, server reliability, and subscription pricing, with significant differences across apps—Claude shows the highest negative sentiment yet a highly enthusiastic user base. The authors also note geopolitical and privacy concerns for DeepSeek and propose a Trust Friction Score to quantify trust and usability barriers.

By Md Jafrin Hossain, Umme Nusrat Jahan, Shouvaggo Sharif Shammo
arXiv Computation and Language
Sep 18

The Role of Fine-grained Harm Signals in LLM Safety

The study investigates how category‑specific harmfulness signals, isolated by removing the shared general harmfulness component, influence large language model (LLM) safety. Using activation steering across 11 risk categories in three instruction‑tuned LLMs, the authors find that the presence of harmfulness in these category residuals varies by category and that the pattern of inducing refusal is even more model‑dependent. Additionally, category residuals were shown to enhance the models’ downstream alignment with the shared general harmfulness representation, indicating that fine‑grained signals play a role beyond the general component.

By Soyeon Park (KAIST), Seogyeong Jeong (KAIST), Sunwoo Kim (KAIST), Alice Oh (KAIST)