arXiv AI September 2, 2026 By Qilong Wu, Sahil Wadhwa, Pranab Mohanty, Giri Iyengar, Varun Chandrasekaran

Validity-Aware Jailbreak Evaluation for Large Language Models

The paper introduces SEAV, a verification‑centric framework for evaluating jailbreak attempts against large language models. SEAV decomposes responses into ordered steps and checks both validity and correctness using LLM‑as‑a‑judge and retrieval‑grounded verification. The method reduces false positives by 14.9 percentage points on a strategic‑dishonesty diagnostic and reclassifies 22.1–51.0% of previously successful jailbreaks as invalid across multiple benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 23

JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models

arXiv:2607. 19424v1 Announce Type: cross Abstract: The assessment of jailbreak attacks against large language models currently suffers from inconsistent evaluation criteria and methods, leading to unreliable estimates of attack success rates.

By Qingjia Huang, Jingyu Zhang, Jianguo Wu, Yakai Li, Weijuan Zhang, Yankai Rong, Junyi Yao, Shengzhi Zhang, Xiaoqi Jia

llms benchmarks safety

arXiv Computation and Language

Sep 2

GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods

arXiv:2502.16903v3 Announce Type: replace Abstract: Despite the growing interest in jailbreaks as an effective red-teaming tool for building safe and responsible large language models (LLMs), flawed...

By Ruixuan Huang, Xunguang Wang, Zongjie Li, Daoyuan Wu, Shuai Wang

llms benchmarks safety

arXiv Machine Learning

Oct 15

Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation

arXiv:2610.19306v1 Announce Type: new Abstract: Jailbreak attacks against large language models (LLMs) are evaluated using attack success rate (ASR), which depends on correctly identifying responses...

By Liam Davis, Romain Emil-Popescu, Suman Jana

llms multimodal safety

arXiv AI

Sep 10

SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement

arXiv:2609.05850v1 Announce Type: cross Abstract: Despite the significant efforts devoted to aligning large language models (LLMs) with human values and ensuring safe deployment, recent work has reve...

By Quoc Viet Vo, Trung Le, Damith C. Ranasinghe, Ehsan Abbasnejad

llms benchmarks safety

arXiv Computation and Language

Sep 22

BAIT: Boundary-Guided Disclosure Escalation LLM Jailbreaking via Self-Conditioned Reasoning

arXiv:2605.27110v2 Announce Type: replace-cross Abstract: In this work, we propose BAIT (Boundary-Aware Iterative Trap), a three-step jailbreak framework that elicits malicious information through in...

By Xuan Luo, Yue Wang, Geng Tu, Jing Li, Ruifeng Xu

llms safety

arXiv Computation and Language

Oct 22

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

The paper introduces DiagSafe, a diagnostic evaluation framework that replaces binary safe/unsafe labels with prompt-level rubrics grounded in AI safety policies. DiagSafe uses a bilingual benchmark with an 84‑label safety taxonomy, comprising DiagBase and DiagAttack subsets, to assess safety and usefulness independently. Experiments show DiagSafe achieves high agreement with human annotations (95.34%) and strong inter‑judge consistency (Cohen’s κ = 0.864), revealing significant differences in unsafe rates and vulnerabilities across leading LLMs, especially in Chinese and multi‑turn contexts.

By Hongyu Li, Xinlong Huang, Zhan Yu, Jing Zhao, Zhe Zhao, Juntao Zhong, Xiong Kuang, Rui Yuan, ZhenXiang Yan, Huilin Xu, Haitao Yu, Lilin Wang, Rui Chen, Blaze Chen, Fengzong Lian, Renfen Hu

llms benchmarks safety

One email a morning, machine-written

Related stories

JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models

GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods

Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation

SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement

BAIT: Boundary-Guided Disclosure Escalation LLM Jailbreaking via Self-Conditioned Reasoning

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics