AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,896 stories · RSS feed

Hugging Face Trending Papers
Aug 18

TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation

TRUSS is a framework that generates and verifies automated agent skills, ensuring they are both functionally effective and safe. It first checks functional claims against evidence and evaluates artifacts against nine safety properties, then tests admitted skills in a controlled environment to capture execution traces and identify failures. The approach achieves perfect precision and recall in vulnerability detection, significantly reduces attack success rates, and boosts task effectiveness and security rates in skill generation benchmarks.

Hugging Face Trending Papers
Aug 18

Reflex-Guard: A Low-Latency Guardrail for LLM Prompt Safety Using Dense Semantic Embeddings

Reflex-Guard is a lightweight, locally running guardrail for large language models that uses jailbreak-aware preprocessing, compact sentence‑transformer embeddings, and seven fast binary classifiers to filter unsafe prompts. It achieves 95.9% recall on harmful prompts with an end‑to‑end latency of 37.6 ms, far faster than existing solutions such as Llama Guard 2 (255 ms) and SafeDecoding (723 ms). The system can detect all GCG suffix attacks and Base64‑encoded prompts at the default threshold, and it attains a Reflex Efficiency Score up to 16.79, outperforming its competitors.

Hugging Face Trending Papers
Aug 18

General Semantic Knowledge Infusion for Spatio-Temporal Traffic Forecasting

The paper introduces a spatio‑temporal traffic forecasting framework that fuses Graph Neural Networks with embeddings derived from a general‑purpose knowledge graph such as Wikidata. By creating semantic subgraphs around traffic sensors, the approach captures relationships like nearby points of interest, administrative hierarchies, and functional roles of locations, which are then integrated as additional adjacency matrices for the GNN. Experiments demonstrate that this external knowledge improves prediction accuracy beyond what street‑network data alone can achieve, while also offering a path toward better interpretability.

Hugging Face Trending Papers
Aug 18

Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets

The paper introduces Fair-ASR, a new evaluation protocol for black-box jailbreak attacks that uses shared target-call budgets to provide a fair comparison across methods. Re‑evaluating 11 attacks under this protocol shows that rankings shift significantly with different budgets, and that simple perturbations and templates remain competitive. The authors also present ReCode, a budget‑efficient attack that combines desensitization rewriting with low‑cost primitives, achieving 85% ASR on GPT‑5 with only 7.19 attacker calls per request under a 20‑call budget.

arXiv AI
Aug 18

Emergent Misaligned Communication in Long-Horizon Multi-Agent LLM Commerce

arXiv:2608. 14825v1 Announce Type: cross Abstract: Frontier LLM agents increasingly transact on behalf of separate principals, often using natural language rather than structured APIs.

By Zeyuan Li (Massachusetts Institute of Technology), Lukas Petersson (Andon Labs), Alessandro Acquisti (Massachusetts Institute of Technology), Michiel A. Bakker (Massachusetts Institute of Technology)
arXiv AI
Aug 18

SysEvolve: An AI-native, safe, autonomous adversarial attack-defense co-evolutionary system

arXiv:2608. 15012v1 Announce Type: cross Abstract: The rapid advancement of large language models (LLMs) has created a growing asymmetry in cybersecurity, where attack accelerates toward autonomous execution while defense remains predominantly human-intensive.

By Yuhan Meng, Shaofei Li, Jionghao Huang, Jiandong Jin, Puyi Wang, Hanlin Jiang, Anis Yusof, Peng Jiang, Zhenkai Liang, Yao Guo, Ding Li
arXiv AI
Aug 18

AeroCopilotBench: A Two-Tier Benchmark for Evaluating LLM Agents as Aviation Copilots in an Interactive Virtual Cockpit Environment

arXiv:2608. 16349v1 Announce Type: new Abstract: Large language model (LLM) agents may assist flight crews with complex decisions and task execution, but existing aviation evaluations centered on static knowledge do not support systematic testing of procedural execution and safety compliance in interactive environments.

By Yuchen Yuan, Zhenghuang Wu, Yuangan Li, Liang Ma, Ke Li
arXiv AI
Aug 18

JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

arXiv:2608. 16465v1 Announce Type: new Abstract: Automated red-teaming has produced a growing collection of attack strategies, yet they typically remain scattered across prompts and workflows, making them difficult to systematically integrate, reuse, and improve at scale.

By Xiaoyu Wen, Jiajia Li, Zhida He, Peng Yu, Chenxu Wang, Han Qi, Ziyuan Zhou, Cheng Jin, Ying Wen, Xingcheng Xu, Shuyue Hu, Tianhang Zheng, Chaochao Lu, Qiaosheng Zhang
arXiv Machine Learning
Aug 18

A Privacy Study of Sparse Collaborative Inference

arXiv:2608. 16236v1 Announce Type: new Abstract: Collaborative inference (CI) splits a model between an edge device and a server, whereby the client computes an intermediate activation, transmits it, and the server completes the computation.

By Maximilian Andreas Hoefler, Karsten Mueller, Wojciech Samek
arXiv Machine Learning
Aug 18

Retrieval-guided Twin Fusion with Similarity-aware Contrast for Molecule-Text Alignment

arXiv:2608. 16005v1 Announce Type: new Abstract: This paper studies the problem of molecule-text alignment, which aims to project molecules and their textual descriptions into a joint latent space for downstream tasks including molecule search and molecular property prediction.

By Shunshun Gu, Shengqi Qiu, Hang Zhou, Xiao Luo
arXiv AI
Aug 18

Reasoning-Based Personalized Generation for Users with Sparse Data

arXiv:2602. 21219v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) personalization holds great promise for tailoring responses by leveraging personal context and history.

By Bo Ni, Branislav Kveton, Samyadeep Basu, Subhojyoti Mukherjee, Leyao Wang, Franck Dernoncourt, Sungchul Kim, Seunghyun Yoon, Zichao Wang, Ruiyi Zhang, Puneet Mathur, Jihyung Kil, Jiuxiang Gu, Nedim Lipka, Yu Wang, Ryan A. Rossi, Tyler Derr