← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 27

Waypoint: Analytically Modeling GEMM for CPUs

The paper introduces Waypoint, an analytical model that automatically determines the optimal tile size for General Matrix Multiplication (GEMM) on CPUs. Waypoint predicts performance by accounting for memory hierarchy, instruction-level parallelism, and multi-threading, and uses these estimates to search for the tile size that maximizes throughput. The authors also present OPAM, a GEMM library that employs Waypoint to match state‑of‑the‑art performance and accelerate large language model inference on CPUs with 1.36–1.98× speedups.

By Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, Prakash Raghavendra
llmsbenchmarks
More like this →
arXiv AI
Oct 27

Single-Cycle Battery Remaining Useful Life Prediction with Compressed Continuous Wavelet Features and Deep Neural Networks

The paper presents a method for predicting the remaining useful life of lithium‑ion batteries using only the first few charge‑discharge cycles. It transforms voltage, current, and capacity data into time‑frequency representations via continuous wavelet transform, compresses the resulting scalograms to reduce redundancy, and feeds them into deep neural networks. Experiments on eight benchmark datasets show the approach outperforms previous methods across diverse chemistries and cycling protocols, and visual explanations confirm the model focuses on meaningful time‑frequency regions.

By Sungeun Kim, Mingyu Jung, Yuri Kim, Jin-Sung Park, Jongbin Ryu
benchmarkssafety
More like this →
arXiv AI
Oct 27

SRA: Selective Residual Adaptation for LLM Reasoning Enhancement

Selective Residual Adaptation (SRA) is a plug‑in framework that improves multi‑step reasoning in distilled large language models by correcting sparse distribution divergences between teacher and student along reasoning paths. It uses a Discrepancy Gate to spot divergence points and a Candidate Residual Adapter to calibrate top‑K logits, thereby avoiding changes to the student backbone or runtime teacher computation. Experiments on Qwen3 models from 0.6B to 4B show that SRA raises average Pass@16 by up to 4.37 percentage points while maintaining general capabilities.

By Zhenxing Dou, Kun He, Mingyang Yi, Yong Liu
llmsefficiencybenchmarks
More like this →
arXiv AI
Oct 27

Entity-Aware Evaluation of OCR Systems for Historical Document Transcription

The paper introduces an entity‑aware evaluation framework for OCR systems applied to historical documents, focusing on how well names, dates, and identifiers are preserved. Using a benchmark of 175 Berrutti Archive documents and 5,209 manually annotated entity mentions, the authors compare sequence‑recognition OCR, OCR‑specialized vision‑language models, and general‑purpose vision‑language models. Their results show that OCR rankings vary with the evaluation criterion: Qwen 3.6 achieves the highest entity coverage, while some systems recover many entities when small character‑level errors are tolerated, indicating that aggregate transcription metrics alone do not capture meaningful information preservation.

By Camilo Mari\~no (UDELAR, CB), Diego Belzarena (UDELAR, CB), Marina Gardella (CB), Ignacio Ram\'irez (UDELAR), Gregory Randall (UDELAR), Jean-Michel Morel (LU - Hong Kong)
llmscomputer-visionmultimodalbenchmarks
More like this →
arXiv AI
Oct 27

UMAS: System-Level Uncertainty Quantification for Multi-Agent LLM Systems

UMAS is a parameter‑free, cost‑neutral framework for system‑level uncertainty quantification in multi‑agent large language model systems. It estimates the credibility of inter‑agent influence using each generation’s intrinsic confidence, modulates uncertainty updates at each node, and aggregates these into a system‑level estimate. Evaluations on Debate and DyLAN show UMAS outperforms state‑of‑the‑art baselines by an average of 10.54 AUROC points and improves hallucination detection and answer selection, boosting MAS accuracy by up to 12.6 points on specific tasks.

By Hanwen Li, Jinhao Duan, Xiaoshuang Shi, Yue Zhang, Tianlong Chen, Kaidi Xu, Chenxi Yuan
llmsagentsbenchmarkssafety
More like this →
arXiv AI
Oct 27

Unlocking the Specialization Capability of PPG Foundation Models for Wearable Signals through Structured Pruning and Adaptive Fine-Tuning

arXiv:2610.33788v1 Announce Type: new Abstract: Photoplethysmography (PPG) plays a central role in wearable health monitoring and clinical decision support. However, existing approaches to universal...

By Huaming Du, Zhixian Xie, Gaoyang Zhen, Gang Kou, Guisong Liu, Yanchao Tan, Wendan Tao, Yuhao Xu, Carl Yang
fine-tuningefficiencybenchmarks
More like this →