Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,494 stories · RSS feed

arXiv Machine Learning

Jun 2

EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation

arXiv:2605. 29977v2 Announce Type: replace-cross Abstract: High-fidelity ECG interpretation is increasingly reliant on massive foundation models, yet their deployment in clinical edge-care remains hindered by extreme computational demands.

By Dang Nguyen Hong, Nhi Ngoc-Yen Nguyen, Huy-Hieu Pham

efficiency benchmarks safety

arXiv Machine Learning

Jun 2

From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction

arXiv:2511. 12081v2 Announce Type: replace-cross Abstract: Despite massive investments in scale, deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns -- a stark contrast to the {predictable scaling laws} seen in large language models (LLMs).

By Bencheng Yan, Yuejie Lei, Zhiyuan Zeng, Zheye Deng, Di Wang, Kaiyi Lin, Pengjie Wang, Chuan Yu, Jian Xu, Bo Zheng

llms benchmarks safety

arXiv Machine Learning

Jun 2

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

arXiv:2602. 03203v2 Announce Type: replace-cross Abstract: Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces.

By Zican Dong, Peiyu Liu, Junyi Li, Zhipeng Chen, Han Peng, Shuo Wang, Wayne Xin Zhao

llms reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 2

SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning

arXiv:2603. 08000v2 Announce Type: replace-cross Abstract: Large reasoning models (LRMs) like OpenAI o1 and DeepSeek-R1 achieve high accuracy on complex tasks by adopting long chain-of-thought (CoT) reasoning paths.

By Chenzhi Hu, Qinzhe Hu, Yuhang Xu, Junyi Chen, Ruijie Wang, Shengzhong Liu, Jianxin Li, Fan Wu, Guihai Chen

llms benchmarks

arXiv AI

Jun 2

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

arXiv:2512. 20638v2 Announce Type: replace-cross Abstract: The evaluation of large language models relies heavily on standardized benchmarks.

By Maty Bohacek, Nino Scherrer, Nicholas Dufour, Thomas Leung, Christoph Bregler, Stephanie C. Y. Chan

llms benchmarks

arXiv AI

Jun 2

Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills

arXiv:2503. 05641v4 Announce Type: replace-cross Abstract: Combining existing pre-trained LLMs is a promising approach for diverse reasoning tasks.

By Justin Chih-Yao Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, Mohit Bansal

llms agents benchmarks

arXiv AI

Jun 2

v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound

arXiv:2509. 25773v3 Announce Type: replace-cross Abstract: AI models capable of comprehending humor hold real-world promise -- for example, enhancing engagement in human-machine interactions.

By Zhengpeng Shi, Yanpeng Zhao, Jianqun Zhou, Yuxuan Wang, Qinrong Cui, Wei Bi, Songchun Zhu, Bo Zhao, Zilong Zheng

llms multimodal benchmarks

arXiv AI

Jun 2

A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

arXiv:2606. 01992v1 Announce Type: cross Abstract: Industrial anomaly detection has historically been a unimodal task.

By Stefano Samele, Eugenio Lomurno, Teodora Jovanovic, Sanjay Shivakumar Manohar, Alberto Crivellaro, Matteo Matteucci

llms rag multimodal benchmarks

arXiv AI

Jun 2

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization

arXiv:2606. 01252v1 Announce Type: cross Abstract: Multi-target cross-lingual text summarization (MTXLS), which summarizes a source document into multiple target languages, is increasingly important as users consume content in diverse languages, but remains underexplored.

By Sangwon Ryu, Yihong Liu, Mingyang Wang, Yunsu Kim, Jungseul Ok, Gary Geunbae Lee, Hinrich Schuetze

llms nlp benchmarks

arXiv AI

Jun 2

Multi-Contrast MRI Motion Correction via Parameter-Informed Disentanglement and Adaptive Experts

arXiv:2606. 00146v1 Announce Type: cross Abstract: Motion artifacts in magnetic resonance imaging (MRI) degrade diagnostic reliability.

By Honglin Xiong, Yuxian Tang, Feng Li, Yulin Wang, Lei Xiang, Dinggang Shen, Qian Wang

llms rag computer-vision benchmarks

arXiv Machine Learning

Jun 2

Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

arXiv:2606. 00206v1 Announce Type: new Abstract: Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood.

By Sanae Lotfi, Polina Kirichenko, Steven Li, Zechun Liu

llms efficiency benchmarks

arXiv Machine Learning

Jun 2

Physics-Informed Residuals for Adaptive Mesh Refinement in Finite-Difference PDE Solvers

arXiv:2606. 02475v1 Announce Type: cross Abstract: Classical finite-difference solvers remain reliable tools for partial differential equations, but their efficiency depends on where mesh resolution is placed.

By Henry Kasumba, Ronald Katende

benchmarks

arXiv AI

Jun 2

DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

arXiv:2606. 00798v1 Announce Type: cross Abstract: Parameter compression of class-conditional diffusion models reveals an underexplored limitation in output-level distillation: the unconditional score branch remains unsupervised, leaving the classifier-free guidance gap underdetermined in the student.

By Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain, Engelbert Mephu Nguifo

diffusion efficiency benchmarks

arXiv AI

Jun 2

TCAR-Gen: Temporal Graph Retrieval with Evidence Fusion for Knowledge-Grounded Generation

arXiv:2606. 00029v1 Announce Type: cross Abstract: Retrieval-augmented generation systems struggle with temporal reasoning and evidence fusion when answering complex questions over historical criminal case narratives.

By Sidra Nasir, Muhammad Noman Zahid, Rizwan Ahmed Khan

llms rag nlp benchmarks

arXiv Machine Learning

Jun 2

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

arXiv:2606. 02384v1 Announce Type: new Abstract: Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures.

By Andrej Tschalzev, Nick Erickson, Yuyang Wang, Huzefa Rangwala, Stefan L\"udtke, Heiner Stuckenschmidt, Christian Bartelt

benchmarks

arXiv AI

Jun 2

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

arXiv:2605. 02122v2 Announce Type: replace-cross Abstract: Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation.

By Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, Houman Homayoun

agents benchmarks safety

arXiv Machine Learning

Jun 2

PE-means: Improved Differentially Private $k$-means Clustering through Private Evolution

arXiv:2606. 00342v1 Announce Type: new Abstract: We study the problem of differentially private (DP) $k$-means clustering in Euclidean space.

By Thomas Humphries, Zinan Lin, Sergey Yekhanin

benchmarks

arXiv Machine Learning

Jun 2

Molecular Embedding-Based Algorithm Selection in Protein-Ligand Docking

arXiv:2512. 02328v2 Announce Type: replace-cross Abstract: Selecting an effective docking algorithm is highly context-dependent, and no single method performs reliably across structural, chemical, and protocol regimes.

By Jiabao Brad Wang, Siyuan Cao, Hongxuan Wu, Yiliang Yuan, Mustafa Misir

rag benchmarks

arXiv AI

Jun 2

Beyond Visual Fidelity: Benchmarking Super-Resolution Models for Large-Scale Remote Sensing Imagery via Downstream Task Integration

arXiv:2605. 00310v2 Announce Type: replace-cross Abstract: Super-resolution (SR) techniques have made major advances in reconstructing high-resolution images from low-resolution inputs.

By Zhili Li, Kangyang Chai, Zhihao Wang, Xiaowei Jia, Yanhua Li, Gengchen Mai, Sergii Skakun, Dinesh Manocha, Yiqun Xie

llms diffusion computer-vision benchmarks

arXiv Machine Learning

Jun 2

Towards Stable, Globally Expressive Graph Representations with Laplacian Eigenvectors

arXiv:2410. 09737v2 Announce Type: replace Abstract: A popular way to improve the expressive power of graph neural networks (GNNs) is to use Laplacian eigenvectors as additional node features, since they can serve both as structural identifiers and global coordinates of nodes.

By Junru Zhou, Cai Zhou, Xiyuan Wang, Pan Li, Muhan Zhang

benchmarks

Get the daily briefing