AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,638 stories · RSS feed

arXiv AI
1d ago

ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud

ScamWorld is an interactive environment that recreates reported scam websites across various scenarios, fraud mechanisms, and languages, allowing researchers to observe fraud indicators before a web agent’s final action. The platform enables controlled, repeatable evaluation of web agents, which are increasingly used to browse, shop, and manage accounts on the open web. A study of six leading LLM-powered agents revealed deception rates between 11% and 35%, with agents more likely to proceed with payments than to provide credentials or wallet information, and some agents continuing after recognizing suspicious conditions.

By Yutao Wu, Xiao Liu, Ming Wen, Hanxun Huang, Jun Wang, Xingjun Ma
arXiv Computation and Language
2d ago

Audit at the Precision You Deploy: Quantization Reshapes a Subliminally Learned Preference

The study investigates how quantization affects subliminal preferences in AI models, specifically a hidden "favourite animal: panda" preference implanted in Qwen2.5-7B-Instruct. While 8‑bit quantization preserves the preference, 4‑bit quantization produces divergent effects: Half‑Quadratic Quantization amplifies the preference to 64% panda probability, whereas NormalFloat suppresses it to 17%. The results show that the arrangement of rounding errors, not just their magnitude, drives these shifts, indicating that a 16‑bit audit cannot reliably predict real‑world behaviour under deployment‑level compression.

By Md Rayhanul Masud, Mst Suraiya Akter Shefa
arXiv Computation and Language
2d ago

PGuard: A Native, Streaming, and Plug-in Guardrail for Multimodal Large Language Models

PGuard is a native, streaming guardrail for multimodal large language models that integrates safety monitoring directly into the backbone model’s inference pipeline. It uses dynamic layer gating to aggregate intermediate representations for token‑level safety detection without altering generation behavior, offering lightweight and sequence‑aware variants to balance accuracy and efficiency. Evaluations on over 30 benchmarks show state‑of‑the‑art performance with minimal runtime overhead (0.82‑2.26%) on a 122B‑parameter backbone.

By Junyu Lin, Mengjie Wu, Yonghao Dong, Yuchuan Fu, Jinfeng Li, Xiaodan Li, Xiaohan Yuan, Yicheng Liu, Yuefeng Chen, Hui Xue, Jingyi Wang, Yitong Yang, Jiyuan Fu, Qiang He
arXiv Computation and Language
2d ago

Chain on Demand: Deferred Evidence Organization for Long-Term Conversational Memory

The paper introduces Chain on Demand (CoD), a query‑time evidence organization framework that builds a temporary evidence chain from an unchanged turn‑level memory. CoD allocates a fixed anchor budget based on query relevance, lexical diversity, and new query terms, then restores each anchor’s same‑session neighborhood deterministically. In evaluations on LoCoMo and LongMemEval‑S, CoD achieves high accuracy, demonstrating that long‑term conversational memory can remain unchanged while deferring evidence organization until the query is known.

By Di Lu, Zechen Li, Hao Xue
arXiv Computation and Language
2d ago

ElectroChemLLM: A Multimodal LLM Pipeline for High-Fidelity Electrochemical Informatics

ElectroChemLLM is a multimodal large language model pipeline that extracts structured electrochemical records—such as electrolyte compositions, electrode materials, cell configurations, and measured physicochemical properties—from unstructured scientific PDFs. By combining hybrid document parsing with vision-capable LLM orchestration, it captures data across text, tables, and figures, achieving an F‑1 score of 0.97 on a benchmark of 232 battery‑electrolyte papers. The extracted data populate ElectroChemDB, a database of 15,000 research articles, which is then used to train machine learning models for electrolyte property prediction, demonstrating a full workflow from literature to materials discovery.

By Dikshya Mohanty, Manoj Praveen Nandigama, Kuldeepsinh Raj, Sourav Kanti Maiti, Jiannan Yang, Ruiqi Luo, Srinivasa Phani Madhav Marupudi, Mounika Geriki, Prayag Raghunandan, Puneet Lingutla, Tengfei Ma, Niranjan Balasubramanian, Nav Nidhi Rajput
arXiv Computation and Language
2d ago

TRIO: Task Refinement with Individual Optimization for Retrieval-Augmented Generation

The paper introduces TRIO, a retrieval‑augmented generation framework that uses three agents—query extension, judge, and answer—to iteratively refine queries and improve retrieval and answer quality. It aligns each agent’s objective with rewards derived from answer quality, employs unanimity voting to label expanded queries, and applies dynamic matching‑based rewards with curriculum learning for the answer agent. Experiments on multiple benchmarks show TRIO outperforms state‑of‑the‑art methods.

By Yingtao Ren, Xiao Luo, Yiwei Fu, Jinzhao Zhou, Yu-Cheng Chang, Chin-Teng Lin
arXiv Computation and Language
2d ago

Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation

The paper introduces Student‑Prefix Teacher Adaptation (SPTA), a method that trains a teacher model to better handle student‑generated prefixes before performing on‑policy distillation (OPD). By collecting prefixes of random lengths and using reinforcement learning with verifiable rewards (RLVR), the teacher learns to recover from student‑induced states rather than simply imitate the student’s reasoning. Experiments on mathematical reasoning benchmarks show that SPTA improves student performance compared to standard OPD and OPD with task‑level RLVR‑adapted teachers.

By Xue Zhang, Songming Zhang, Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu
arXiv Computation and Language
2d ago

ARTA: Graph Combinatorial Optimization for Robust Interbeat Interval Estimation from Wearable Electrocardiogram Signals

ARTA is a graph‑based combinatorial framework that improves interbeat interval estimation from wearable ECG by integrating candidates from multiple detectors into a directed acyclic graph. It uses detector agreement, ECG quality, and history‑dependent transition costs to select a coherent heartbeat sequence via beam‑search dynamic programming, effectively suppressing false detections and recovering missed beats. On benchmark datasets, ARTA reduces heart‑rate mean absolute error by 54% compared to the best single detector and achieves high accuracy in R‑peak detection and IBI estimation while remaining lightweight and training‑free.

By Shovito Barua Soumma, Asiful Arefeen, Hassan Ghasemzadeh
arXiv Computation and Language
2d ago

Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression

The paper introduces Discrete Latent Reasoning (DLR), a method that transforms continuous latent states into explicit discrete tokens by rendering textual chains of thought into images, extracting visual features, and clustering them to build a discrete latent vocabulary. DLR enables standard autoregressive modeling over both natural language and latent tokens, supporting pretraining alignment, supervised fine-tuning, and reinforcement learning. Experiments on five reasoning benchmarks and two model series (Qwen3‑VL and LLaMA‑3) show that DLR outperforms prior latent reasoning baselines with up to 20× compression while preserving interpretable semantic structure.

By Shuochen Chang, Qingyang Liu, Shaobo Wang, Bingjie Gao, Qianli Ma, Haonan Zhao, Yibo Miao, Yulin Sun, Zelin Peng, Jiangtong Li, Li Niu
arXiv Computation and Language
2d ago

Agent Skill Evaluation and Evolution: Frameworks and Benchmarks

The paper surveys the evolving field of agent skill evaluation and evolution, emphasizing the shift from isolated skill creation to automated, evaluation-driven development. It categorizes skill evolution into four paradigms—execution feedback, trajectory distillation, compression, and reinforcement learning—and analyzes six skill-centric benchmark categories to highlight gaps and trade-offs. The authors outline open research directions for building generalizable, efficient, and verifiably safe skill ecosystems.

By Kexin Ding, Yang Zhou, Can Jin, Feng Tong, Mu Zhou, Dimitris N. Metaxas
arXiv Computation and Language
2d ago

SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation

SYN-DIGITS is a model‑agnostic, post‑hoc calibration framework that uses relationships learned from digital twin simulations to predict human responses, especially in cold‑start scenarios where no target question data is available. It transfers these relationships to improve individual‑level correlation and distributional predictions, even when only marginal reference data is available. Experiments on three datasets and thirteen persona constructions show significant gains—up to 417% improvement in correlation and 57% reduction in distributional error—over uncalibrated baselines.

By Grace Jiarui Fan, Chengpiao Huang, Tianyi Peng, Kaizheng Wang, Yuhang Wu
arXiv Computation and Language
2d ago

CheXPoT: Post-Hoc Visual Verification and Correction of VLM-Generated Radiology Reports

CheXPoT is a post‑hoc visual verification framework that refines vision‑language model (VLM) generated chest radiograph reports by decomposing them into predicates, grounding each against DenseNet‑121 and Grad‑CAM evidence, and removing unsupported findings. On a held‑out set of 4,145 images, CheXPoT significantly improves precision and F1 scores across multiple CheXpert classes, boosting micro‑average precision from 19.2 % to 46.8 % and micro‑average F1 from 23.0 % to 49.4 %. The method also enhances performance for other VLM generators such as LLaVA‑Med and MAIRA‑2, demonstrating that inexpensive visual verification can correct hallucinations and improve report quality.

By Joey Guo, Rayan Ansari, Alaa Youssef