AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Computation and Language
2d ago

PGuard: A Native, Streaming, and Plug-in Guardrail for Multimodal Large Language Models

PGuard is a native, streaming guardrail for multimodal large language models that integrates safety monitoring directly into the backbone model’s inference pipeline. It uses dynamic layer gating to aggregate intermediate representations for token‑level safety detection without altering generation behavior, offering lightweight and sequence‑aware variants to balance accuracy and efficiency. Evaluations on over 30 benchmarks show state‑of‑the‑art performance with minimal runtime overhead (0.82‑2.26%) on a 122B‑parameter backbone.

By Junyu Lin, Mengjie Wu, Yonghao Dong, Yuchuan Fu, Jinfeng Li, Xiaodan Li, Xiaohan Yuan, Yicheng Liu, Yuefeng Chen, Hui Xue, Jingyi Wang, Yitong Yang, Jiyuan Fu, Qiang He
arXiv Computation and Language
2d ago

Chain on Demand: Deferred Evidence Organization for Long-Term Conversational Memory

The paper introduces Chain on Demand (CoD), a query‑time evidence organization framework that builds a temporary evidence chain from an unchanged turn‑level memory. CoD allocates a fixed anchor budget based on query relevance, lexical diversity, and new query terms, then restores each anchor’s same‑session neighborhood deterministically. In evaluations on LoCoMo and LongMemEval‑S, CoD achieves high accuracy, demonstrating that long‑term conversational memory can remain unchanged while deferring evidence organization until the query is known.

By Di Lu, Zechen Li, Hao Xue
arXiv Computation and Language
2d ago

ElectroChemLLM: A Multimodal LLM Pipeline for High-Fidelity Electrochemical Informatics

ElectroChemLLM is a multimodal large language model pipeline that extracts structured electrochemical records—such as electrolyte compositions, electrode materials, cell configurations, and measured physicochemical properties—from unstructured scientific PDFs. By combining hybrid document parsing with vision-capable LLM orchestration, it captures data across text, tables, and figures, achieving an F‑1 score of 0.97 on a benchmark of 232 battery‑electrolyte papers. The extracted data populate ElectroChemDB, a database of 15,000 research articles, which is then used to train machine learning models for electrolyte property prediction, demonstrating a full workflow from literature to materials discovery.

By Dikshya Mohanty, Manoj Praveen Nandigama, Kuldeepsinh Raj, Sourav Kanti Maiti, Jiannan Yang, Ruiqi Luo, Srinivasa Phani Madhav Marupudi, Mounika Geriki, Prayag Raghunandan, Puneet Lingutla, Tengfei Ma, Niranjan Balasubramanian, Nav Nidhi Rajput
arXiv Computation and Language
2d ago

TRIO: Task Refinement with Individual Optimization for Retrieval-Augmented Generation

The paper introduces TRIO, a retrieval‑augmented generation framework that uses three agents—query extension, judge, and answer—to iteratively refine queries and improve retrieval and answer quality. It aligns each agent’s objective with rewards derived from answer quality, employs unanimity voting to label expanded queries, and applies dynamic matching‑based rewards with curriculum learning for the answer agent. Experiments on multiple benchmarks show TRIO outperforms state‑of‑the‑art methods.

By Yingtao Ren, Xiao Luo, Yiwei Fu, Jinzhao Zhou, Yu-Cheng Chang, Chin-Teng Lin
arXiv Computation and Language
2d ago

Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation

The paper introduces Student‑Prefix Teacher Adaptation (SPTA), a method that trains a teacher model to better handle student‑generated prefixes before performing on‑policy distillation (OPD). By collecting prefixes of random lengths and using reinforcement learning with verifiable rewards (RLVR), the teacher learns to recover from student‑induced states rather than simply imitate the student’s reasoning. Experiments on mathematical reasoning benchmarks show that SPTA improves student performance compared to standard OPD and OPD with task‑level RLVR‑adapted teachers.

By Xue Zhang, Songming Zhang, Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu
arXiv Computation and Language
2d ago

ARTA: Graph Combinatorial Optimization for Robust Interbeat Interval Estimation from Wearable Electrocardiogram Signals

ARTA is a graph‑based combinatorial framework that improves interbeat interval estimation from wearable ECG by integrating candidates from multiple detectors into a directed acyclic graph. It uses detector agreement, ECG quality, and history‑dependent transition costs to select a coherent heartbeat sequence via beam‑search dynamic programming, effectively suppressing false detections and recovering missed beats. On benchmark datasets, ARTA reduces heart‑rate mean absolute error by 54% compared to the best single detector and achieves high accuracy in R‑peak detection and IBI estimation while remaining lightweight and training‑free.

By Shovito Barua Soumma, Asiful Arefeen, Hassan Ghasemzadeh
arXiv Computation and Language
2d ago

Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression

The paper introduces Discrete Latent Reasoning (DLR), a method that transforms continuous latent states into explicit discrete tokens by rendering textual chains of thought into images, extracting visual features, and clustering them to build a discrete latent vocabulary. DLR enables standard autoregressive modeling over both natural language and latent tokens, supporting pretraining alignment, supervised fine-tuning, and reinforcement learning. Experiments on five reasoning benchmarks and two model series (Qwen3‑VL and LLaMA‑3) show that DLR outperforms prior latent reasoning baselines with up to 20× compression while preserving interpretable semantic structure.

By Shuochen Chang, Qingyang Liu, Shaobo Wang, Bingjie Gao, Qianli Ma, Haonan Zhao, Yibo Miao, Yulin Sun, Zelin Peng, Jiangtong Li, Li Niu
arXiv Computation and Language
2d ago

Agent Skill Evaluation and Evolution: Frameworks and Benchmarks

The paper surveys the evolving field of agent skill evaluation and evolution, emphasizing the shift from isolated skill creation to automated, evaluation-driven development. It categorizes skill evolution into four paradigms—execution feedback, trajectory distillation, compression, and reinforcement learning—and analyzes six skill-centric benchmark categories to highlight gaps and trade-offs. The authors outline open research directions for building generalizable, efficient, and verifiably safe skill ecosystems.

By Kexin Ding, Yang Zhou, Can Jin, Feng Tong, Mu Zhou, Dimitris N. Metaxas
arXiv Computation and Language
2d ago

SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation

SYN-DIGITS is a model‑agnostic, post‑hoc calibration framework that uses relationships learned from digital twin simulations to predict human responses, especially in cold‑start scenarios where no target question data is available. It transfers these relationships to improve individual‑level correlation and distributional predictions, even when only marginal reference data is available. Experiments on three datasets and thirteen persona constructions show significant gains—up to 417% improvement in correlation and 57% reduction in distributional error—over uncalibrated baselines.

By Grace Jiarui Fan, Chengpiao Huang, Tianyi Peng, Kaizheng Wang, Yuhang Wu
arXiv Computation and Language
2d ago

CheXPoT: Post-Hoc Visual Verification and Correction of VLM-Generated Radiology Reports

CheXPoT is a post‑hoc visual verification framework that refines vision‑language model (VLM) generated chest radiograph reports by decomposing them into predicates, grounding each against DenseNet‑121 and Grad‑CAM evidence, and removing unsupported findings. On a held‑out set of 4,145 images, CheXPoT significantly improves precision and F1 scores across multiple CheXpert classes, boosting micro‑average precision from 19.2 % to 46.8 % and micro‑average F1 from 23.0 % to 49.4 %. The method also enhances performance for other VLM generators such as LLaVA‑Med and MAIRA‑2, demonstrating that inexpensive visual verification can correct hallucinations and improve report quality.

By Joey Guo, Rayan Ansari, Alaa Youssef
arXiv Computation and Language
2d ago

AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Web Penetration Agents

AgentSnare is a trajectory‑adaptive deception system that expands a decoy environment while autonomous web‑penetration agents explore. It uses an LLM‑driven decoy constructor and a virtual kernel to generate and materialize decoy artifacts that persist across interactions, enabling long‑horizon exploration. In experiments on 15 CVE‑Bench web applications and four attacker models, AgentSnare delays 49.9% of tool calls, diverts 57.2% of post‑entry actions into decoys, and defuses 89.5% of completion reports, preventing any real target exploitation at pass@3 across 60 attacker‑CVE pairs.

By Ruoyu Wang, Heng Zhao, Renjie Wu, Tianhang Zheng, Mengnan Zhao, Wanyu Lin, Kui Ren
arXiv Computation and Language
2d ago

CuriousTogether: Implications of Curiosity and Cultural Grounding in Multi-Agent LLM Interactions

CuriousTogether is a framework that evaluates curiosity-driven interactions among multiple large language models (LLMs) across novelty-seeking, epistemic, and social curiosity dimensions. In eight tasks covering reasoning, creative generation, and social bias mitigation, curiosity-guided multi-agent settings outperform non-curious ones on seven of the eight tasks, achieving up to a 20.6% increase in story semantic diversity and a 14.9 percentage point drop in stereotype rates. The study also shows that country-conditioned cultural grounding can improve story generation and social tasks, though it may reduce precision in reasoning, and that inference-time curiosity-based multi-agent approaches offer better quality-token trade-offs compared to single-agent baselines.

By Angana Borah, Zoha Hayat Bhatti, Zhijing Jin, Rada Mihalcea
arXiv Computation and Language
2d ago

ShiftRAG: Bypassing the Textual Bottleneck via Decoupled Learning and Continuous Soft Tokens

ShiftRAG is a new retrieval‑augmented generation framework that decouples discriminative retrieval learning from autoregressive generation while still using a single LLM backbone. It addresses the textual bottleneck of existing RAG pipelines by projecting retrieval alignments into continuous soft tokens, creating a high‑bandwidth interface that feeds retrieval signals into the generator. The approach also introduces a soft orthogonality objective to reduce rank collapse and layer‑wise relevance dynamics to preserve fine‑grained latent space, resulting in significant improvements in retrieval and question‑answering performance over state‑of‑the‑art baselines.

By Youmin Ko, Jihong Jeong, Hyunjoon Kim
arXiv Computation and Language
2d ago

Specificity Bias and the Wikipedia Paradox: An xAI Analysis of AI Text Detectors under Domain Shift

The paper examines how fine‑tuned Transformer detectors (DistilBERT, RoBERTa) perform on an out‑of‑distribution dataset of hyper‑local news generated by LLMs. Using SHAP‑based explainable AI, the authors uncover three phenomena: a Call‑to‑Action bias, a Specificity Bias where hallucinated named entities evade detection, and a Wikipedia Paradox where formal human prose is misclassified as AI. These results highlight systematic shortcomings of purely statistical detectors in structured human domains and underscore the need for semantically‑aware, interpretable detection methods.

By Nico Morloher, Johannes Reschke, Mohamed Mady
arXiv Computation and Language
2d ago

Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models

The paper introduces a comprehensive audit protocol for System One decision models, which provide typed, probability‑scored answers without human oversight. Using three adapted benchmarks—BBQ, CrowS‑Pairs, and SilverSpoon—it evaluates twelve models plus a no‑thinking baseline across 1,638,000 decisions, revealing widespread stereotype alignment, order‑dependent responses, and confidence instability. The study demonstrates that swapping answer options and reversing question order significantly alters model outputs, underscoring the need for order‑invariant scoring in procurement.

By Alejandro Mozo Quesada (Universidad de M\'alaga, Spain), Sergio G\'alvez Rojas (Universidad de M\'alaga, Spain), Francisco L\'opez Valverde (Universidad de M\'alaga, Spain)
arXiv Computation and Language
2d ago

Wrong Together: Shared Bias in User Simulation

The paper investigates whether disagreement among user simulators can flag unreliable estimates of group response proportions and group differences. Using six language‑model simulators on 60 questions across 12 countries, disagreement more accurately predicts invalidity for group differences (AUC 0.87) than for single group proportions (AUC 0.71). The authors find that shared bias among simulators can obscure errors, but correcting for this bias reduces prediction error without altering disagreement signals.

By Saber Zerhoudi
arXiv Computation and Language
2d ago

VIAS: Value-Isolating Activation Steering for Multi-Value Alignment and Compliance for Organisations

The paper introduces VIAS, a method for Value-Isolating Activation Steering that aims to improve multi-value compliance in large language model agents. VIAS constructs steering vectors by contrasting responses that include all relevant values with those that omit a single target value, ensuring the target value changes from compliant to non-compliant while maintaining compliance on other values. Evaluations on the PROTECT workplace compliance dataset show that VIAS reduces vector overlap and, when combined with prompting, achieves higher compliance rates across three models compared to prompting alone.

By Avni Mittal, Avinash Anand, Dikshant Kukreja, Erik Cambria, Rajiv Ratn Shah, Sandipan Dandapat
arXiv Computation and Language
2d ago

Do Agents Still Follow the Rules? Benchmarking Rule Compliance under Agent Context Compaction

The paper introduces ARCBench, a configurable benchmark for measuring rule compliance in language‑model agents that compact long histories to fit context windows. Using ARCBench, the authors find that rule violations after compaction rarely stem from rule deletion; instead, agents often ignore the rule while it remains present in the compacted history. The study shows that the source of violations depends on how the rule is delivered and on the compaction strategy, with targeted compactor attacks significantly increasing violations, and that simply retaining a rule in the compacted summary does not guarantee compliance.

By Yiwei Zha, Qingyu Meng, Herbert Bos, Min Chen
arXiv Computation and Language
2d ago

Style over Substance? Linguistic Bias in LLM-based Peer Review

arXiv:2610.35254v2 Announce Type: replace Abstract: The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review worklo...

By Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz