arXiv AI

SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing

arXiv:2606. 29887v1 Announce Type: new Abstract: In real-world applications, guardrails are often expected to identify unsafe user-model interactions according to application-specific safety policies, rather than relying on predefined risk taxonomies.

Hugging Face Trending Papers
Jun 22

SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning

Vision-language models (VLMs) are increasingly deployed in consumer, medical, financial, and enterprise applications. This broad deployment expands the safety surface: risks can arise from multimodal question answering, assistant responses, and cross-modal composition, while moderation policies may vary across products, regions, and deployment stages.

arXiv AI
Jul 8

DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrail

arXiv:2607. 06326v1 Announce Type: new Abstract: Large language models deployed in open-world applications require safety guardrails that are both robust to complex risks and efficient enough for low-latency runtime moderation.

By He Liu, Changtao Miao, Xinjie Yang, Tianle Song, Yin Wu, Junchi Chen, Bintao He, Xinyuan Zhang, Bo Zhang, Shi Yan, Wei Lu, Wei Wang, Danyang Xu, Jiansheng Cai, Zhe Li
arXiv AI
Aug 7

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

arXiv:2608. 05695v1 Announce Type: new Abstract: As large language model (LLM) agents increasingly invoke external tools and interact with real-world systems, unsafe actions may cause irreversible consequences on external states, user data, and downstream services.

By Wenhao Lin, Chenyu Yu, Xingwei Lin, Sicong Cao, Xiang Chen, Lei Xue, Le Yu, Letian Sha, Chunming Wu
arXiv AI
Aug 26

TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models

TRACE is a new benchmark that evaluates the safety of Large Reasoning Models (LRMs) across the entire inference pipeline, including prompts, reasoning traces, and final responses. It provides prompts in two languages covering nine risk categories and ten attack strategies, and for each prompt four LRMs generate traces and responses that are annotated for safety with supporting evidence extracted from the source text. Evaluation of 18 guardrail models on TRACE shows that detecting unsafe content in reasoning traces is much harder than in prompts or final responses, and that current models struggle to extract the necessary evidence.

By Zhenyu Wu, Siyuan Chen, Changchun Yang, Jiaqi Dong, Min Zhou, Ali Almadan, Talal Hammad, Faisal Wahbo, Aminullah Tora, Mona Alshahrani, Xin Gao
arXiv AI
Jun 6

From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents

arXiv:2606. 05805v1 Announce Type: new Abstract: LLM-based guardrails typically safeguard agents by evaluating proposed actions or inputs before execution, producing safety signals such as binary allow/deny decisions, risk categories, and/or explanatory rationales about potential policy violations.

By Yuhao Sun, Jiacheng Zhang, Shaanan Cohney, Zhexin Zhang, Feng Liu, Xingliang Yuan