Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,220 stories · RSS feed

arXiv Machine Learning

Aug 4

Why Large Language Models Fail at Tabular Prediction

arXiv:2608. 02412v1 Announce Type: new Abstract: Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data.

By Marta Garnelo, Wojciech M. Czarnecki

llms agents fine-tuning benchmarks

arXiv Machine Learning

Aug 4

GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning

arXiv:2608. 02585v1 Announce Type: new Abstract: Optimization-based latent reasoning improves large language model outputs by optimizing instance-specific continuous states at test time while keeping model parameters frozen.

By Zhaoxin Yu, Qi Shen, Hengli Li, Zhaowei Zhang, Song-Chun Zhu, Chi Zhang, Zilong Zheng

llms benchmarks safety

arXiv Machine Learning

Aug 4

Cost-Effective Automated Judging of Natural-Language Mathematical Proofs

arXiv:2608. 00004v1 Announce Type: cross Abstract: Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier LLM judges are expensive.

By Benjamin Grayzel

llms benchmarks

arXiv Machine Learning

Aug 4

SCALP: Semi-Supervised Statistical Shape Modeling from Imperfect 3D Photogrammetry via Landmark-Anchored Spectral Warp

arXiv:2608. 00187v1 Announce Type: cross Abstract: Correspondence-based statistical shape modeling (SSM) is vital for population-level morphometric analysis, but conventional pipelines assume clean, fully registered surfaces.

By Nawazish Khan, Sanjay Bhandari, Sarang Joshi, Alzbeta Novotna, Tiffany Jeong, Loretta Bowman, Michael Hernandez, Tobi Somorin, Viraj Govani, Jesse Glodstein, Shireen Elhabian

llms benchmarks

arXiv Machine Learning

Aug 4

TRACE-TS: Attribution-Grounded and Traceable Sensor-Language Reasoning for Human Activity Understanding

arXiv:2608. 00200v1 Announce Type: cross Abstract: Wearable sensors capture fine-grained motion patterns that support rich behavioral understanding, yet most existing methods reduce these signals to activity labels.

By Sparsh Rastogi, Tanmay Kumar, Baiyu Chen, Jatin Bedi, Zechen Li, Flora D. Salim

llms benchmarks

arXiv Machine Learning

Aug 4

CurveShift: Is Agent Progress Scalar? Separating Level from Shape

arXiv:2608. 00355v1 Announce Type: cross Abstract: Progress in large language models is often summarized using a single scalar measure, such as a time horizon, a latent ability estimate, or an aggregate benchmark score.

By Hanwen Xing, Pengyun Wang, BingXu Meng, Kumail Alhamoud, Xiang Li, Jicheng Wang, Xin Yu, Xinyang Han, Xiaomin Li, Philip Torr, Yuexing Hao

llms agents benchmarks

arXiv Machine Learning

Aug 4

Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors

arXiv:2608. 00675v1 Announce Type: cross Abstract: Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.

By Alexander Scheinker

diffusion benchmarks

arXiv Machine Learning

Aug 4

Staged Multi-Agent Training (SMAT) for Hip Exoskeletons: Metabolic and Biomechanical Validation of a Simulation-Trained Co-Adaptive Controller

arXiv:2608. 00715v1 Announce Type: cross Abstract: Learning-based controllers can deliver exoskeleton assistance after training entirely in physics-based simulation, yet few controllers that address human-device co-adaptation have been validated on real users by whole-body metabolic measurement, the standard benchmark for assistive walking.

By Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou

agents benchmarks

arXiv Machine Learning

Aug 4

PhenoStitch: Training-Free Panoptic Crop Mapping from Satellite Image Time Series

arXiv:2608. 00870v1 Announce Type: cross Abstract: Panoptic crop mapping requires both delineating individual agricultural parcels and assigning a crop type to each parcel from satellite image time series.

By Xuechen Li

computer-vision benchmarks

arXiv Machine Learning

Aug 4

Rethinking PPG-based Sleep Staging: Datasets, Metrics, and Benchmarks

arXiv:2608. 00943v1 Announce Type: cross Abstract: Automated sleep staging assigns discrete stage labels to successive time epochs throughout an overnight recording; conventionally each window spans at least 30 seconds, reflecting the minimum temporal resolution of the clinical scoring standard.

By Shuntian Zheng, Jiawei Wang, Cong Fu, Huan Yu, Chen Chen, Yu Guan, Sai Gu

benchmarks

arXiv Machine Learning

Aug 4

When Collaboration Becomes a Trigger: Collective Evidence-Threshold Backdoors in Multi-Agent Systems

arXiv:2608. 01085v1 Announce Type: cross Abstract: LLM-based multi-agent systems (MAS) extend LLM capabilities through iterative communication and shared contexts.

By Jia-Hao Xiao, Lei Feng, Min-Ling Zhang

llms agents benchmarks safety

arXiv Machine Learning

Aug 4

Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

arXiv:2608. 01193v1 Announce Type: cross Abstract: An AI development race creates a multi-agent safety dilemma.

By Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh, Phu Quy Nguyen Lam, Chi Nguyen Tran, Minh Trung Le, Phong Hao Le, Dinh Nam Nguyen, Thien Ky Nguyen Dong, Elias Fernandez Domingos, Le Hong Trang, The Anh Han

llms agents benchmarks safety

arXiv Machine Learning

Aug 4

Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning

arXiv:2608. 01418v1 Announce Type: cross Abstract: Autoregressive rollout generation is a major computational cost in reinforcement learning for large language models.

By Wenhao Zhang, Yibo Xie, Rui Wang, Jiahua Yang, Lei Jiang, Zibo Yang, Yawei Wang, Jiali Xu, jasperawang, Haoyang Long, Huan Xiong, alantzhao

llms reinforcement-learning benchmarks

arXiv Machine Learning

Aug 4

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

arXiv:2608. 01454v1 Announce Type: cross Abstract: Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols.

By Lorenzo Guerra, Thomas Chapuis, Guillaume Duc, Pavlo Mozharovskyi, Van-Tam Nguyen

benchmarks

arXiv Machine Learning

Aug 4

The Label Defines the Timescale: Trait-State Limits of Temporal-Aggregate Learning

arXiv:2608. 01587v1 Announce Type: cross Abstract: Machine-learning benchmarks often pair a label that aggregates a long temporal horizon with input observed through one or a few short windows.

By Xizhe Zhang

benchmarks

arXiv Machine Learning

Aug 4

Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models

arXiv:2608. 01624v1 Announce Type: cross Abstract: Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars.

By Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim, Unggi Lee

llms benchmarks

arXiv Machine Learning

Aug 4

LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

arXiv:2608. 01662v1 Announce Type: cross Abstract: DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer.

By Wen Zan, Jiaqi Zhang, Jianchao Tan, Hong Liu, Cunguang Wang, Xiang Li, Duyue Ma, Guanyu Wu, Yifan Lu, Fengcun Li, Yerui Sun, Peng Pei, Yuchen Xie, Xunliang Cai

efficiency benchmarks

arXiv Machine Learning

Aug 4

Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

arXiv:2608. 01676v1 Announce Type: cross Abstract: Sparse attention is widely deployed in long-context serving stacks, yet no framework audits how discarding blocks changes the influence of specific content on model output.

By Xingyu Ren, Youran Sun, Chugang Yi, Haizhao Yang

efficiency benchmarks

arXiv Machine Learning

Aug 4

Automatic Annotation of Ancient Greek Vowel Length

arXiv:2608. 01935v1 Announce Type: cross Abstract: Prior work in Ancient Greek NLP relies on corpora that do not disambiguate the phonemic vowel length of alpha, iota, and ypsilon, together known as the dichrona.

By Albin Th\"orn Cleland, Eric Cullhed

llms nlp benchmarks

arXiv Machine Learning

Aug 4

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study

arXiv:2608. 02235v1 Announce Type: cross Abstract: Recent advances in neural text-to-speech (TTS) systems have substantially improved speech naturalness and intelligibility across many languages.

By Ali Jafar, Amal Sarmad, Shifa Yousaf, Maryam Bashir

llms multimodal benchmarks

Get the daily briefing