AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv AI
Oct 21

AutoRedClaw: Automated Red-Teaming for Claw-Style Agents

AutoRedClaw is an automated red‑teaming framework designed for claw‑style agents such as OpenClaw, which maintain persistent configuration, memory, and reusable skills. It generates end‑to‑end attacks and the necessary environments, using a strict quality‑checking protocol and execution‑based validation to ensure realistic, effective samples. The framework produces a 330‑case benchmark, RedClaw‑Bench, achieving over 30% effective attack rate at less than $0.10 per successful sample and demonstrating higher attack success rates than existing benchmarks across multiple agent systems.

By Hao Li, Tongyan Hu, Weidi Luo, Shanghao Shi, Xiaogeng Liu, Aiden Suh, Yulong Cao, Ning Zhang, William Yeoh, Yevgeniy Vorobeychik, Chaowei Xiao
arXiv Machine Learning
Oct 21

Glass-Box Deep Reinforcement Learning for High-Speed Flow Control

The article discusses how glass-box deep reinforcement learning (DRL) can discover actuation strategies for high‑speed, nonlinear, unsteady flows without explicit models, potentially revealing mechanisms beyond classical control. It highlights the challenges of applying DRL to real high‑speed flows, including simulation cost, data efficiency, sensing in extreme environments, reward design, explainability, and generalization across Mach and Reynolds numbers. The authors argue that overcoming these constraints will require physics‑aware learning, robust real‑time operation, and extensive validation to translate DRL capabilities into reliable, certified technology.

By Ameya D. Jagtap
arXiv Machine Learning
Oct 21

Baseline and Incremental: User, Item, and Request Debiasing in Instagram Feed Ranking

The paper proposes a multi‑signal value model for Instagram Feed ranking that separates baseline predictions from incremental gains, arguing that retaining raw predictions and adding incremental terms improves ranking quality. It describes how three independent debiasing families—viewer habit, item popularity, and request context—are combined into a single parameterized operator, with a design procedure selecting optimal points. Controlled ablation studies show that the two‑term model outperforms single‑term variants in most metrics, and the approach has been adopted across Meta’s products, boosting key engagement metrics.

By Hsin-Chiang Chen, Hao Wang, Gedi Zhou, Mingze Gao, Abhishek Kumar, Yiben Yang, Haoyu Wang, Hongjing Xia, Bo Chen, Zhengyu Su, Xian Sun, Chang Xu, Kevin Han, Weiwei Li, Zhongchang Sun, Yao Zhang, Qifei Wang, Peggy Yang, Yihan Wu, Haoran Liu, Siyang Sun, Xingyu Pan, Shannon Phu, Haoming Chen, Kevin Zhao, Chen Liang
arXiv Computation and Language
Oct 21

Decomposed Relation Extraction from Aviation Accident Narratives: Evaluating Transformer-Based Safety Knowledge Discovery under Relational Sparsity

The paper investigates relation extraction from aviation accident narratives, noting that true safety relations are extremely sparse among concept pairs. It evaluates a decomposed framework that separates relation classification from concept identification, using a benchmark of 300 NTSB narratives expanded to 21,038 candidate pairs, of which only 248 represent key safety relations. Experiments with a TF‑IDF SVM and a weighted RoBERTa classifier show high false‑positive rates and a significant drop in performance from development to exhaustive evaluation, highlighting challenges in controlling false positives and managing candidate‑space sparsity.

By Aziida Nanyonga, Mustafa Tahsin Yilmaz, Saad Ezzini, Ugur Turhan, Graham Wild
arXiv Computation and Language
Oct 21

Do LLMs Really Lack Difference Awareness? Uncovering Heuristic Substitution in Group-Differentiating Fairness

The paper examines whether large language models (LLMs) truly lack awareness of meaningful differences between social groups in fairness decisions. It proposes that instead of lacking difference awareness, LLMs often substitute a simpler equal‑treatment heuristic, a phenomenon called Fairness Heuristic Substitution. The authors introduce a Two‑stage Distinguish‑Decide (TDD) framework that separates the identification of relevant group differences from the fairness decision, and show that TDD outperforms one‑stage baselines across multiple benchmarks and models, while also revealing mechanistic evidence for the heuristic substitution hypothesis.

By Jingshen Zhang, Bo Wang, Yanlin Fu, Yuewen Dang, Dongming Zhao, Zifei Yu, Yuexian Hou
arXiv Computer Vision
Oct 21

Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

The paper introduces Pangaea-Latents, an extension of the Pangaea benchmark that provides frozen embeddings from five geospatial foundation models (GFMs) across eleven datasets, showing that these embeddings are neither redundant nor disjoint. It then proposes GeoMixture, a framework that aligns a Vision Transformer (ViT) to multiple GFMs per dataset using a multi-representation alignment (mREPA) objective, resulting in an encoder that achieves the best average rank across nine Pangaea datasets while maintaining cross-task consistency.

By Arthur Ouaknine, David Rolnick