arXiv AI
Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 15
Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration
arXiv:2610.14542v1 Announce Type: new Abstract: Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has r...
Hugging Face Trending Papers
Oct 20
DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or...
arXiv AI
Oct 21
DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
arXiv:2610.28522v1 Announce Type: new Abstract: Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make...
arXiv Computation and Language
Sep 1
Auditing MCQA Benchmarks through Probability Landscapes
arXiv:2608.30372v1 Announce Type: new Abstract: As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While th...
arXiv Machine Learning
Jun 2
FLaG: Fine-Grained Latent Grouping for Hallucination Detection
arXiv:2606. 00301v1 Announce Type: new Abstract: Hallucinations in large language models (LLMs) arise from heterogeneous failure mechanisms, making reliable detection difficult for any single global uncertainty score.
arXiv AI
Sep 1
Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation
arXiv:2608.30044v1 Announce Type: new Abstract: Language models are commonly compared by averaging scores across a benchmark list with equal weight. Such lists grow through publication outside an exp...