arXiv AI August 19, 2026 By Nyamtulla Shaik, Fengjun Li, Bo Luo

Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models

The paper investigates whether existing AI safety benchmarks, designed for large language models, are suitable for evaluating small language models (SLMs). By testing five benchmark suites on 26 open‑source SLMs with a unified scoring rubric, the authors find that ambiguous judgments dominate, especially for complex prompts and certain architectures. This ambiguity, linked to factors like lexical density and output perplexity, undermines the reliability of aggregate leaderboards and reveals a confound between model capability and perceived safety.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 2

Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity

arXiv:2607. 01153v1 Announce Type: cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model has followed an instruction, refused appropriately, complied with a policy, resisted an embedded command, or misreported progress in an agentic task.

By Brett Reynolds

llms agents benchmarks safety

arXiv AI

Jun 30

CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models

arXiv:2501. 14940v4 Announce Type: replace-cross Abstract: Aligning large language models (LLMs) with human values is essential for their safe deployment and widespread adoption.

By Guangzhi Sun, Xiao Zhan, Shutong Feng, Philip C. Woodland, Jose Such

llms benchmarks safety

arXiv AI

Jul 31

Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control

arXiv:2607. 01153v3 Announce Type: replace-cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model followed an instruction, refused appropriately, complied with a policy, or misreported progress in an agentic task.

By Brett Reynolds

llms agents benchmarks safety

arXiv AI

Jun 11

Models That Know How Evaluations Are Designed Score Safer

arXiv:2605. 28591v2 Announce Type: replace-cross Abstract: The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings.

By Katharina Deckenbach, Haritz Puerto, Jonas Geiping, Sahar Abdelnabi

fine-tuning benchmarks safety

arXiv AI

Aug 7

What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)

arXiv:2608. 06202v1 Announce Type: cross Abstract: Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness.

By Ro Encarnaci\'on, Tina Behzad, Emma Lurie, Dana\'e Metaxa

llms benchmarks safety

arXiv Machine Learning

Sep 23

SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation

SSP-Bench is a dynamic benchmarking framework designed to evaluate large language models on safety, security, and privacy (SSP) by generating evaluation instances on demand while maintaining domain consistency. It ensures label validity through external sources, enforces scope with service-specific validation, and calibrates difficulty using a multi-model steering panel, framing benchmark construction as a multi-objective optimization over difficulty, separability, novelty, and diversity. Across 24 models and four SSP services, SSP-Bench exposes systematic failures of static benchmarks, such as near-zero correlation in safety rankings, strong safety–over-refusal coupling, and hidden within-family regressions.

By Fatih Deniz, Yazan Boshmaf, Issa Khalil

llms benchmarks safety

One email a morning, machine-written

Related stories

Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity

CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models

Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control

Models That Know How Evaluations Are Designed Score Safer

What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)

SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation