arXiv Machine Learning June 16, 2026 By Luca Zhou

Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives

Read the original on arXiv Machine Learning →

arXiv:2606. 16511v1 Announce Type: new Abstract: Recent work motivates moving large language model (LLM) evaluation from mean-based to tail-aware metrics, including conditional value-at-risk and tail-index estimates of reward-model error.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 5

ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models

arXiv:2606. 05170v1 Announce Type: new Abstract: At matched accuracy, open-weight LLMs differ substantially in the shape of their error severity distribution -- a difference invisible to the scalar error rate.

By Jason Z Wang

llms benchmarks safety

arXiv AI

2d ago

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

arXiv:2608. 14509v1 Announce Type: new Abstract: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt.

By Zhelun Wu

llms

arXiv Machine Learning

Jun 24

Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

arXiv:2606. 23915v1 Announce Type: cross Abstract: Practice often treats automatic metrics for attribution in LLM retrieval-augmented generation as interchangeable.

By Tianyu Ding, Aditya Nannapaneni, Juan Pablo De la Cruz Weinstein

llms rag

arXiv AI

1d ago

HarmProfile: Characterizing Harmful Distributions in Frontier LLMs

arXiv:2608. 14577v1 Announce Type: cross Abstract: Frontier large language models (LLMs) safety evaluation has largely treated harmful generation as an attack outcome rather than as an object of analysis.

By Zhouyuan Ma, Yutao Wu, Hanxun Huang, Xiang Zheng, Xiao Liu, Yixin Cao, Zuxuan Wu, Xingjun Ma, Yu-Gang Jiang

llms benchmarks safety

arXiv AI

Jun 2

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models

arXiv:2605. 05427v2 Announce Type: replace Abstract: Refusal rates are a poor proxy for LLM safety, i.

By Alif Al Hasan, Sumon Biswas

llms benchmarks safety

arXiv AI

Jun 11

Estimating Tail Risks in Language Model Output Distributions

arXiv:2604. 22167v2 Announce Type: replace-cross Abstract: Language models are increasingly capable and are being rapidly deployed on a population-level scale.

By Rico Angell, Raghav Singhal, Zachary Horvitz, Zhou Yu, Rajesh Ranganath, Kathleen McKeown, He He

llms benchmarks safety

Related stories

ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

HarmProfile: Characterizing Harmful Distributions in Frontier LLMs

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models

Estimating Tail Risks in Language Model Output Distributions