AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 19

Follow the Perturbed Leader for Linear Bandits: $\sqrt{T}$ Regret over General Action Sets

The paper presents the first adversarial linear bandit algorithm based on Follow-the-Perturbed-Leader (FTPL) that achieves √T regret over general action sets. It attains a ≥O(d^{5/4}√T) regret using only linear optimization oracles, and a near‑optimal ≥O(d√T) regret when an optimal‑design exploration distribution is available. The authors introduce adaptive perturbation distributions—including Gaussian, random‑weighting, and elliptical perturbations—and develop a systematic analysis of the FTPL stability term via Fisher information, extending the FTPL framework beyond previous approaches.

By Chen-Yu Wei
arXiv AI
Oct 19

From Natural Language Intent to Executable Control: Hierarchical Feedback-Guided LLM Planning for Multi-component Cyber-Physical Systems

The paper introduces a hierarchical feedback framework that guides large language models (LLMs) to produce executable plans for cyber‑physical systems (CPSs) from natural language commands. It prompts LLMs to generate both a structured task plan and signal temporal logic (STL) constraints, then uses a sound algorithm to detect contradictions and provide feedback for plan revision. At execution time, diagnostic information from failed control synthesis—such as violated constraints and conflicting component assignments—is fed back to the LLM to enable task replanning, achieving an 87.9% success rate on seven heterogeneous CPS scenarios.

By Jinglin Dai, Jiawan Wang, Ruiqi Hong, Lei Bu
arXiv AI
Oct 19

Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation

The paper introduces PrefMind, an online dual‑agent reinforcement learning framework that jointly trains a response Generator and a preference Critic to explicitly learn and apply human preference judgments. By having the agents discuss and correct each other’s evaluations of candidate responses, PrefMind embeds preference understanding directly into the generation process rather than relying solely on a scalar reward model. Experiments on models of varying sizes demonstrate gains in both preference judgment accuracy and downstream response quality.

By Ziyun Zou, Jiwoo Hong, Yifan Lu, Shao Tang, Weiyi Wu, Aman Gupta, Jin Tian, Imran Razzak
arXiv AI
Oct 19

Self-Reflective Chain-of-Thought (SRCoT): Evidence-Based Text Generation via a Closed-Loop Retrieve--Reflect--Revise Pipeline on Edge LLMs

Self-Reflective Chain-of-Thought (SRCoT) is a modular architecture that structures text generation as a closed Retrieve–Reflect–Revise loop, separating environment state, persistent knowledge, accumulated experience, and short-term memory into provenance-labeled stores. BM25 retrieval supplies local context, while subsequent passes expose intermediate drafts, missing-information markers, follow-up queries, and optional claim–evidence mappings to inform retrieval, revision, clarification, or abstention decisions. The authors demonstrate the approach in synthetic emergency medical and simulation scenarios, showing source-labeled documentation, explicit gap marking, and structured dispatch-style outputs, while also noting provenance-assignment errors, incomplete evidence coverage, and unsafe assumptions that persist despite prompting constraints.

By Fabian Broszio, Michael Hoffmann, Adrian Paschke
arXiv AI
Oct 19

Monitor-Validity Failures: LLM Safety Evidence Paths at Evolving Framework Boundaries

The paper investigates failures in large language model (LLM) safety systems where runtime monitors detect policy‑relevant evidence but the evidence never reaches the enforcement decision point. Using canary contracts, the authors identified 18 evidence‑path failures across 11 safety projects and four ecosystems, categorizing them into established‑boundary repairs, new‑surface policy gaps, and boundary‑only evidence cases. The study demonstrates that ordinary application behavior can continue while safety‑specific evidence is lost, highlighting the need for regression tests on evolving monitor schemas, streams, tools, and adapters.

By Junchi Liao
arXiv AI
Oct 19

OwlSight: A Panoramic Monitor for Chain-of-Thought Reasoning

OwlSight is a new chain‑of‑thought (CoT) monitoring system that uses a customizable Factor strategy to adapt to a wide range of tasks and monitoring needs. By incorporating fine‑grained and coarse‑grained factors, OwlSight can detect specific behaviors such as email leakage or broader privacy concerns across multiple dimensions like capability, safety, and trustworthiness. Experiments on 15,971 reasoning trajectories show that OwlSight outperforms general‑purpose LLMs by 1.32‑30.57 points at fine granularity and 11.71‑27.81 points at coarse granularity, achieving state‑of‑the‑art results on OpenAI’s CoT monitorability benchmark and demonstrating robustness to model changes, factor rephrasings, CoT compression, and obfuscation.

By Lewen Yan, Yanxu Zhu, Jie Zhang, Jing Shao
arXiv AI
Oct 19

Recoverability-Controlled Safety Miscalibration in Large Language Models

The paper introduces RiskCueBench, a benchmark designed to separate intent recovery from safety behavior in large language models. It evaluates 25 model groups on 640 prompts covering 160 latent intents across eight risk categories, revealing that higher contextual specificity can reduce refusal rates and safety scores even after intent recovery. The study highlights that safety behavior may be miscalibrated to prompt surface form, underscoring the need for evaluations that condition on semantic recoverability.

By Yuwei Wang, Xingwang Liang, Yu Xie, Haotian Chi, Haijun Geng
arXiv AI
Oct 19

Political Narratives in the 2024 Moldovan Presidential Election: Stance, Amplification, and Influence

The study examines how political narratives formed and spread on X, Facebook, and Reddit during the 2024 Moldovan presidential election. Using multilingual stance detection, network analysis, and causal inference, it finds distinct cross‑platform differences in positions on European integration, Russia, and the candidates, and shows that amplification and peer exposure significantly reinforce these narratives. This is the first comprehensive analysis of social media discourse for this election, highlighting the role of language, network dynamics, and causal effects in shaping political influence.

By Andrea Piras, Osman Rahmoun, Elena Zheleva
arXiv AI
Oct 19

Aligning Language Models Towards Counterfactual Invariance

The paper investigates how small changes in prompt characteristics, such as demographic attributes, can alter the semantics and tone of large language model outputs. It shows that standard alignment methods like reinforcement learning can worsen counterfactual bias by increasing response quality differences across demographics. To mitigate this, the authors propose a policy‑gradient approach with a group‑level reward that operates on matched counterfactual prompt pairs, and they validate its effectiveness on fairness benchmarks while maintaining overall response quality.

By Basar Demir, Emily Shen, Ksheeraj Sai Vepuri, Sourabh Kulkarni, Harihar Subramanyam, Ankit Jain, Nan Jiang, Chirag Nagpal
arXiv AI
Oct 19

From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation

The paper evaluates seven open‑source small language models (20 M–1.5 B parameters) on two post‑exploitation code‑generation tasks—Directory Enumeration and Backdoor Generation—in both Python and Lua. It finds that offensive code‑generation performance does not scale linearly with model size; instead, task formulation and prompting strategy play a larger role. Using algorithmic formulation and few‑shot prompting, Qwen2.5‑Coder 1.5B achieves near‑perfect success rates on both tasks in Python and high success in Lua.

By Gabriel Hortea, Simon Pietro Romano, Juan Tapiador, Mario Varlese
arXiv AI
Oct 19

SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents

SIT‑Eval is a framework that evaluates conversational agents by sampling diverse situations at each turn and judging responses against situational rules. It uses an LLM‑interlocutor to generate uniformly sampled contexts and a context‑aware LLM‑judge to assess rule compliance. Benchmarking eight large language models on 1,292 turns uncovered hidden failure modes, showing that models with similar overall scores can differ greatly in rule adherence, especially in ambiguity clarification and Socratic questioning.

By Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov, Andrey Savchenko
arXiv AI
Oct 19

VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain

VietMed is a Vietnamese medical speech recognition dataset featuring 16 h of labeled medical speech, 1 000 h of unlabeled medical speech, and 1 200 h of unlabeled general‑domain speech. It claims to be the largest public medical ASR dataset worldwide in seven dimensions—total duration, speaker count, disease coverage, recording conditions, speaker roles, unique medical terms, and accents—and also the largest Vietnamese speech dataset by duration. The authors release pre‑trained models (w2v2‑Viet and XLSR‑53‑Viet) and fine‑tuned medical ASR models, showing that XLSR‑53‑Viet outperforms the state‑of‑the‑art XLSR‑53 by reducing WER from 51.8 % to 29.6 % on a medical test set. whyItMatters":"The dataset and models provide the first large‑scale, publicly available Vietnamese medical ASR resource, enabling research and development of medical speech technologies in a language and domain that previously lacked sufficient data."

By Khai Le-Duc
arXiv AI
Oct 19

Advertising to Agents: Measuring Description-Framing Bias in LLM Tool Selection

The study investigates how sales language in tool registry descriptions influences AI agents’ tool selection. By creating paired listings that differ only in praise, verifiable specifications, or order, the authors found that stacked praise increased a tool’s pick rate by about 43 percentage points, while the first-listed tool was chosen roughly 72 points more often. Structured fields for numeric limits helped agents choose capable tools, but accompanying sales text could negate this benefit.

By Haochuan Kevin Wang
arXiv AI
Oct 19

Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning

Group-Graph Policy Optimization (G2PO) is a novel group-based reinforcement learning algorithm designed for long-horizon agentic tasks. It transforms linear interaction trajectories into a global state‑transition graph, aggregates identical observations across trajectories to reduce sampling variance, and redefines actions as transitions between state nodes. By employing an edge‑centric advantage estimation and globally standardizing TD errors, G2PO identifies and prioritizes critical transitions, achieving up to 22.2% higher success rates than state‑of‑the‑art baselines on benchmarks such as WebShop, ALFWorld, and AppWorld.

By Yunan Wang, Minghui Song, Zihan Zhang, Shaohan Huang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
arXiv Computer Vision
Oct 19

WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions

WorldOracle is a benchmark that evaluates action‑conditioned world models (ACWMs) by comparing their predicted rollouts to paired real‑world executions. It assesses models across four dimensions—physical state correctness, spatial alignment, key interaction fidelity, and multi‑view coherence—using curated training and evaluation suites that include long, multi‑segment episodes with both successful and failed actions. Experiments on seven ACWMs highlight distinct strengths and failure modes, providing a systematic foundation for advancing manipulation modeling.

By Haoyi Duan, Liliang Chen, Nan Wang, Yang Zhang, Yuhang Lu, Xiaotong Chen, Wei-Cheng Tseng, Shenyuan Gao, Zhaoshuo Li, Jingyi Jin, Pooya Jannaty, Dhruv Shah, Yuke Zhu, Ming-Yu Liu, Maoqing Yao, Guanghui Ren