arXiv Computation and Language October 22, 2026 By Hongyu Li, Xinlong Huang, Zhan Yu, Jing Zhao, Zhe Zhao, Juntao Zhong, Xiong Kuang, Rui Yuan, ZhenXiang Yan, Huilin Xu, Haitao Yu, Lilin Wang, Rui Chen, Blaze Chen, Fengzong Lian, Renfen Hu

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

Read the original on arXiv Computation and Language →

The paper introduces DiagSafe, a diagnostic evaluation framework that replaces binary safe/unsafe labels with prompt-level rubrics grounded in AI safety policies. DiagSafe uses a bilingual benchmark with an 84‑label safety taxonomy, comprising DiagBase and DiagAttack subsets, to assess safety and usefulness independently. Experiments show DiagSafe achieves high agreement with human annotations (95.34%) and strong inter‑judge consistency (Cohen’s κ = 0.864), revealing significant differences in unsafe rates and vulnerabilities across leading LLMs, especially in Chinese and multi‑turn contexts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Oct 15

Specification-Based Testing of Large Language Models

arXiv:2610.14443v1 Announce Type: cross Abstract: Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined benc...

By Kausar Moshood, Jace Bolante, Roshan Aditya Murali, Manish Motwani

llms benchmarks safety

arXiv AI

Aug 19

Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models

The paper investigates whether existing AI safety benchmarks, designed for large language models, are suitable for evaluating small language models (SLMs). By testing five benchmark suites on 26 open‑source SLMs with a unified scoring rubric, the authors find that ambiguous judgments dominate, especially for complex prompts and certain architectures. This ambiguity, linked to factors like lexical density and output perplexity, undermines the reliability of aggregate leaderboards and reveals a confound between model capability and perceived safety.

By Nyamtulla Shaik, Fengjun Li, Bo Luo

llms benchmarks safety

arXiv Machine Learning

Jun 25

RAS: Measuring LLM Safety Through Refusal Alignment

arXiv:2606. 25750v1 Announce Type: cross Abstract: Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy.

By Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu, Wei-Bin Lee

llms safety

arXiv AI

Aug 12

Why Do Safety Guardrails Degrade Across Languages?

arXiv:2605. 17173v2 Announce Type: replace-cross Abstract: Large language models exhibit safety degradation in non-English languages.

By Max Zhang, Ameen Patel, Sang T. Truong, Sanmi Koyejo

llms safety

arXiv AI

Jul 31

Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control

arXiv:2607. 01153v3 Announce Type: replace-cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model followed an instruction, refused appropriately, complied with a policy, or misreported progress in an agentic task.

By Brett Reynolds

llms agents benchmarks safety

arXiv AI

Oct 15

XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

arXiv:2605.05662v2 Announce Type: replace-cross Abstract: Current LLM safety benchmarks are predominantly English-centric and often rely on translation, failing to capture country-specific harms. Mor...

By Dasol Choi, Eugenia Kim, Jaewon Noh, Sang Seo, Eunmi Kim, Myunggyo Oh, Yunjin Park, Brigitta Jesica Kartono, Josef Pichlmeier, Helena Berndt, Sai Krishna Mendu, Glenn Johannes Tungka, \"Ozlem G\"ok\c{c}e, Suresh Gehlot, Katherine Pratt, Amanda Minnich, Haon Park

llms multimodal benchmarks safety

One email a morning, machine-written

Related stories

Specification-Based Testing of Large Language Models

Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models

RAS: Measuring LLM Safety Through Refusal Alignment

Why Do Safety Guardrails Degrade Across Languages?

Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control

XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity