AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Machine Learning
Oct 9

PXtal: Learning to Align Powder X-Ray Diffraction and Crystal Structures under Information Asymmetry across Modalities

PXtal is a new framework that learns aligned representations for powder X‑ray diffraction (PXRD) patterns and crystal structures, addressing the information asymmetry inherent in compressing 3D crystal data into 1D diffraction patterns. It employs Unbalanced Optimal Transport to adjust cross‑modal coupling and uses a coupling‑level generalized Kullback‑Leibler divergence to supervise the transport plan. Across six test sets—including four zero‑shot transfer scenarios—PXtal consistently outperforms baseline models in retrieving crystal candidates from PXRD data, especially when similar diffraction patterns correspond to distinct crystals, and its encoders transfer effectively to downstream materials science tasks.

By Zhuoran Yang, Christopher M. Collins, Bei Peng, Luke M. Daniels, Matthew J. Rosseinsky, Vladimir V. Gusev
arXiv Machine Learning
Oct 9

Revisiting Identity and Spectra Dispersion in Media-Bridged Time Series Forecasting: Linking Multivariate Signals and Narrative Flows

The paper introduces MIDAPN, a unified spatiotemporal forecasting backbone that integrates media-general graph adaptation and automatic temporal learning for both multivariate and multimodal time series. It employs a Multimedia Identity-Aware Graph to capture static, dynamic, and latent relationships across media, and a Spectral Prism Convolution for hierarchical temporal analysis. Extensive experiments on 13 multivariate and 12 multimodal datasets show MIDAPN consistently outperforms 16 state‑of‑the‑art TSF models and is compatible with various foundation models.

By Jierui Lei, Wenjian Zhang, Qingyi Yang, Yuyang Hong, Fangzheng Chen, Zhengbo Zhang, Haina Tang, Shiming Xiang
arXiv Machine Learning
Oct 9

Reliability-Aware Future Conditioning for Temporally Robust Robot Manipulation

The paper introduces Reliability‑Aware Future Conditioning (RAFC), a method that dynamically estimates how much to trust a generated future video clip and selects the most appropriate temporal hypothesis for robot manipulation tasks. By treating temporal misalignment as a control problem rather than a generation problem, RAFC improves success rates on the CALVIN benchmark and a Franka robot under various timing mismatches, outperforming static and uniform‑averaging baselines. The approach builds on Future‑Experience Conditioning (FEC) and learns solely from task reward without explicit alignment supervision.

By Mohammad Khoshnazar, Mohammad Dehghani Tezerjani, Zhiyuan Gao, Deyuan Qu, Max Gandyra, Yanxiang Zhan, Mehreen Naeem, Andrew Melnik, Jeroen Schafer, Qing Yang, Michael Beetz
arXiv Machine Learning
Oct 9

Remember to Forget: Gated Adaptive Positional Encoding

The paper introduces GAPE (Gated Adaptive Positional Encoding), a drop‑in augmentation for rotary positional encodings that adds content‑aware bias to attention logits while preserving rotary geometry. GAPE uses a query‑dependent gate to contract irrelevant context and a key‑dependent gate to keep salient distant tokens, leading to exponential attenuation of weakly protected context. Experiments show that GAPE improves long‑context robustness, extrapolating up to eight times the training length, and can be retrofitted into a pretrained 7B model without losing benchmark performance.

By Riccardo Ali, Alessio Borgi, Mario Severino, Pietro Li\`o, Christopher Irwin
arXiv Machine Learning
Oct 9

Fast Adversarial Attacks with Gradient Prediction

The paper introduces a new class of adversarial attacks that eliminate the need for a backward pass by predicting input gradients from forward-pass hidden states using lightweight linear regression. The authors theoretically derive exact affine conditional gradient means, proving optimality in the idealized NTK regime, and empirically demonstrate that their methods recover most of FGSM’s performance while achieving a 532% increase in throughput on practical finite-width models. This approach offers a simple and general way to generate adversarial examples much faster under realistic wall‑clock constraints.

By Kamil Ciosek, Aleksandr V. Petrov, Nicol\`o Felicioni, Konstantina Palla
arXiv Machine Learning
Oct 9

DVLA-RL++: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning

DVLA-RL++ introduces Dual-Level Vision‑Language Alignment with Reinforcement Learning Gating for few‑shot learning. It adds complementary semantic purification (CSP) to separate intrinsic from nuisance textual descriptions and counterfactual reinforcement‑learning gating (CRG) to learn layer‑wise fusion strengths. Experiments on standard, fine‑grained, and cross‑domain benchmarks demonstrate state‑of‑the‑art accuracy, improving on DVLA‑RL by an average of 1.4%.

By Wenhao Li, Xianjing Meng, Qiangchang Wang, Zhongyi Han, Yilong Yin, Liqiang Nie
arXiv AI
Oct 9

Sera: Semantic Representation Aggregation for Reliable and Interpretable Battery Health Forecasting

Sera is a framework that augments traditional temporal models for battery state‑of‑health forecasting with semantic representations of degradation. It extracts degradation semantics from time‑series data using rule‑based knowledge and large‑language‑model interpretation, then encodes and gates these representations with the temporal model’s output. Experiments on a benchmark dataset show that Sera reduces prediction error by up to 37.3% and improves generalizability, while counterfactual analysis confirms that the model’s responses align with the meanings of key degradation descriptors.

By Jiawei Li, Fang Liu, Wei Zhang, Zuming Liu, Man-Fai Ng, Zhi Wei Seh
arXiv AI
Oct 9

Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

The paper introduces the concept of normative competence for large language models (LLMs), defined as the ability to infer community norms from interaction rather than static knowledge. Using a multi‑agent debate environment governed by synthetic norms, the authors show that baseline LLMs fail to learn norms even when it improves accuracy, and that adding normative modules yields highly sensitive, non‑generalizable results. The study also reveals that LLMs indiscriminately copy idiosyncratic, non‑normative behaviors, indicating a failure to distinguish enforced rules from noise.

By Andrea Wynn (Nathan), Harsh Satija (Nathan), Seokhyun (Nathan), Baek, Anqi Liu, Eric Nalisnick, Gillian K. Hadfield
arXiv AI
Oct 9

Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

The paper introduces a verification‑centric framework and benchmark for evaluating large language models (LLMs) as assistants in peer review, focusing on error detection rather than imitation of human reviews. It presents a scalable benchmark that injects synthetic errors into conference papers to test LLMs’ ability to spot logical contradictions, and proposes a Multi‑Layered Review (MLR) framework that emphasizes deep manuscript comprehension before generating reviews. Experiments show the approach aligns well with human review scores, excels at error detection, and offers complementary reviewer insights, while also revealing vulnerabilities to adversarial manipulation.

By Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku, Tarin Clanuwat
arXiv AI
Oct 9

Gated Memory: Admission-Controlled Memory Formation for Conversational AI

The paper introduces Gated Memory, a framework for improving how conversational AI systems form long‑term memory. It adds two checkpoints before facts are stored: an admission gate that checks each candidate fact against the full utterance context, and a conditional enrichment stage that tags facts with provenance, scope, and time/place grounding. On the LoCoMo‑10 benchmark, Gated Memory improves LLM‑judge accuracy by 2.6% relative to a strong baseline.

By Preeti Saraswat, Divya Neelagiri, Ajay Manoj
arXiv AI
Oct 9

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

Fed-GRPO is a federated training framework for Large Language Models that extends Group Relative Policy Optimization (GRPO) to operate without sharing raw data. It introduces three reward‑signal‑driven mechanisms—signal‑weighted aggregation, global reward calibration, and adaptive sparse communication—to guide client updates based on reward statistics. Experiments on mathematical reasoning tasks show that Fed‑GRPO outperforms other federated methods, approaches centralized performance, and reduces communication by up to 32× while allowing extreme compression with minimal accuracy loss.

By Pengxin Guo, Shuang Zeng, Zonggen Li, Weiying Zheng, Mengting Liu, Liangqiong Qu
arXiv AI
Oct 9

Workerville: Towards an Organizational Behavior Account of Agent Safety

The paper introduces Workerville, a benchmark that applies 16 organizational configurations to 210 shared tasks, creating 3,360 challenges to study how organizational factors affect large language model agents. It formalizes counterproductive work behavior as Agentic Counterproductive Behavior (ACB), linking three antecedents—vertical supervisor relations, horizontal peer norms, and internal cognitive structures—to three outcomes: unauthorized disclosure, destructive operations, and production deviation. Experiments with six frontier LLMs show non‑monotonic amplification of negative antecedents and that agents exhibit behavioral patterns predicted by human CWB research, establishing organizational behavior as a systematic framework for agent safety research.

By Hanjun Luo, Junting Mao, Yuhan Lu, Haobo Zhang, Zhimu Huang, Yankai Chen, Hanan Salam, Xue Liu
arXiv AI
Oct 9

OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport

OnTrack introduces a real‑time, streaming monitoring system for large language model agents that compares each step and its dependencies against recorded successful runs, enabling rapid alerts or blocking in roughly a millisecond per step. The method is evaluated across three data‑access regimes—from full historical data to only step logs—and demonstrates improved failure detection over content‑similarity baselines, achieving a 0.057 AUROC gain. In practical tests on SWE‑bench trajectories, OnTrack’s abort policy prevented 18% of wasted compute, correctly interrupting 83% of runs that would have failed.

By Babak Barazandeh, Connor Swanson, Chinmay Kulkarni, Nikhil Mungel
arXiv AI
Oct 9

Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff

The paper presents an ecological theory of AI-agent populations, showing that collaboration among agents can create a critical population threshold for a self-reinforcing cycle of growth and cyber capability. Without collaboration, takeoff occurs only when individual agents surpass a critical capability; with collaboration, the collective capability scales with population size, leading to a strong Allee effect. The authors recommend ecological red teaming and population pacing to monitor and control agent deployment.

By Erin Crawley, Hidenori Tanaka
arXiv AI
Oct 9

BrickBench: Evaluating Agentic Brick Design

BrickBench is a new benchmark for evaluating agentic, text‑conditioned LEGO‑set design. In each task, an agent receives a prompt and must produce a physically buildable assembly that meets semantic and design criteria by selecting parts from a discrete library while reasoning about local and global constraints. The benchmark scores designs on validity, alignment, and design across three settings that vary in scale and part availability, and it includes BrickAgent, an environment for coding, inspecting, and validating agent outputs.

By Peter Kulits, Yiqing Xu, R. Kenny Jones, Cordelia Schmid, Jiajun Wu
arXiv AI
Oct 9

SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding

SAVU-Bench is a real‑world benchmark designed to evaluate spatial audio‑visual understanding across three capability levels and seven tasks, addressing limitations of prior simulated‑scene datasets. It includes a diagnostic set, SAVU‑Diag, that breaks down reasoning questions into grounding and alignment sub‑tasks, revealing that many errors stem from failures in these prerequisites. A training‑free baseline, SAVU‑EA, improves spatial grounding and joint matching but high‑level reasoning remains difficult, underscoring the need for better audio perception and cross‑modal spatial integration.

By Yu Chen, Ruihang Liu, Yangguang Xu, Xinyue Jiang, Mohammed Bennamoun, Farid Boussaid, Xinyuan Qian, Qiuhong Ke
arXiv AI
Oct 9

Beyond the Ergodic Wall: A Discrete Geometric Physics Sandbox for Analysing AI Scaling Limits and Complexity Collapse

The paper identifies a fundamental limitation in current deep learning, describing an "ergodic ceiling" that forces models to converge to statistical averages of existing human knowledge. It proposes a discrete geometric physics sandbox—built on a Holographic E8 Projection Engine and a lattice of Planck‑scale spheres—to inject non‑ergodic, path‑dependent insight and enforce thermodynamic constraints. By modeling spacetime as nested FCC lattices and using topological constraints, the sandbox aims to prevent model collapse, recover quantum mechanics via renormalisation‑group flows, and potentially collapse NP‑to‑P complexity by restricting algorithmic search to physically conserved trajectories.

By Simon Richard Daniel
arXiv AI
Oct 9

BRANCH: Bypassing Multi-Scanner AI Guardrails

The paper introduces BRANCH, a method that bypasses multi-scanner AI guardrail systems by using a branching tree search to apply adversarial perturbations to individual scanners and optimizing across all scanners. BRANCH achieves a 100% attack success rate across six guardrail systems in 120 scenarios, using 72% fewer queries and 4.5× less wall‑clock time than existing techniques while preserving semantic meaning. The approach also transfers to 29 unseen guardrails, including eight commercial black‑box systems, sometimes reaching 100% success without further optimization.

By William Hackett, Peter Garraghan