Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,907 stories · RSS feed

arXiv Machine Learning

Aug 4

Online Algorithms via Minimax and Posterior Matching

arXiv:2608. 01616v1 Announce Type: new Abstract: Competitive analysis is central to the study of online algorithms, but upper bounds are often highly problem-specific.

By Thomas Kesselheim, Marco Molinaro, Kalen Patton, Sahil Singla

benchmarks

arXiv Machine Learning

Aug 4

Heterogeneous Multi-Agent Reinforcement Learning for Radio Resource Management under Coupled Finite-Horizon Constraints

arXiv:2608. 01745v1 Announce Type: new Abstract: Maximizing throughput under proportional fairness in dense wireless networks requires jointly managing user association, scheduling, base station (BS) activation, and handover control under hard finite-horizon energy and handover budgets, which induces a fundamental tension between BS-side energy management and user-side handover regulation.

By Yeonseo Jeong, Wonhyeok Ko, Sungweon Hong, Songnam Hong

agents reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Aug 4

ReFP-AD: Rectified Flow Preconditioning for Energy-Based Anomaly Detection

arXiv:2608. 01793v1 Announce Type: new Abstract: Unified anomaly detection requires modeling highly heterogeneous normal data without access to anomalous samples.

By Camile Lendering, Erkut Akdag, Joaqu\'in Figueira, Egor Bondarev

rag diffusion benchmarks

arXiv Machine Learning

Aug 4

ChaosProbe: A Neurochaotic Lens on Frozen Transformer Input-Embedding Spaces

arXiv:2608. 01968v1 Announce Type: new Abstract: Transformer models are most often understood through what they do: their benchmark performance, generation quality, or behavior on downstream tasks.

By Kunal Kumar Pant, Nithin Nagaraj

llms rag benchmarks

arXiv Machine Learning

Aug 4

Constrained Co-Design for Photonic Bayesian Neural Networks

arXiv:2608. 02229v1 Announce Type: new Abstract: Classical neural networks frequently produce overconfident predictions on ambiguous or out-of-distribution (OOD) data, a liability that grows with each AI system deployed in safety-critical real-world scenarios.

By Hendrik Borras, Xiao Wang, Bernhard Klein, Robin Janssen, Frank Br\"uckerhoff-Pl\"uckelmann, Wolfram Pernice, Holger Fr\"oning

efficiency benchmarks safety

arXiv Machine Learning

Aug 4

Qwen-CUA: Native Computer Use for (almost) Everything

arXiv:2608. 02352v1 Announce Type: new Abstract: Native computer use offers a general interface for agents to operate almost any software available to people, but requires long-horizon state tracking, large-scale interactive experience, and learning from sparse yet verifiable outcomes.

By Dunjie Lu, Shuai Bai, Tianyi Bai, Sicheng Fan, Chang Gao, Jian Guan, Feng Hu, Mianqiu Huang, Xingyang Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Ning Li, Dayiheng Liu, Shixuan Liu, Zheng Liu, Que Shen, Bowen Wang, Junli Wang, Chencan Wu, Rui Xie, Tianbao Xie, Zhihui Xie, Haiyang Xu, An Yang, Tao Yu, Wenzhen Yuan, Xi Zhang, Zhenru Zhang, Mingkang Zhu, Zhaoqing Zhu, Yizhong Cao, Kai Dang, Binyuan Hui, Kaixin Li, Junyang Lin, Haiquan Wang, Zekun Wang, Yiheng Xu, Fan Yan, Mengqi Yuan, Danyang Zhang, Jiajun Zhang, Zhipeng Zhang, Fan Zhou, Fan Zhou

agents benchmarks

arXiv Machine Learning

Aug 4

BRiG-AFA: Bellman Risk-to-Go Learning for Non-Myopic Active Feature Acquisition

arXiv:2608. 02305v1 Announce Type: new Abstract: Active feature acquisition (AFA) asks which unobserved feature to measure next for each test instance under a budget.

By Jiaorong Feng, Qian Li, Ying Li

benchmarks

arXiv Machine Learning

Aug 4

Why Large Language Models Fail at Tabular Prediction

arXiv:2608. 02412v1 Announce Type: new Abstract: Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data.

By Marta Garnelo, Wojciech M. Czarnecki

llms agents fine-tuning benchmarks

arXiv Machine Learning

Aug 4

GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning

arXiv:2608. 02585v1 Announce Type: new Abstract: Optimization-based latent reasoning improves large language model outputs by optimizing instance-specific continuous states at test time while keeping model parameters frozen.

By Zhaoxin Yu, Qi Shen, Hengli Li, Zhaowei Zhang, Song-Chun Zhu, Chi Zhang, Zilong Zheng

llms benchmarks safety

arXiv Machine Learning

Aug 4

Cost-Effective Automated Judging of Natural-Language Mathematical Proofs

arXiv:2608. 00004v1 Announce Type: cross Abstract: Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier LLM judges are expensive.

By Benjamin Grayzel

llms benchmarks

arXiv Machine Learning

Aug 4

SCALP: Semi-Supervised Statistical Shape Modeling from Imperfect 3D Photogrammetry via Landmark-Anchored Spectral Warp

arXiv:2608. 00187v1 Announce Type: cross Abstract: Correspondence-based statistical shape modeling (SSM) is vital for population-level morphometric analysis, but conventional pipelines assume clean, fully registered surfaces.

By Nawazish Khan, Sanjay Bhandari, Sarang Joshi, Alzbeta Novotna, Tiffany Jeong, Loretta Bowman, Michael Hernandez, Tobi Somorin, Viraj Govani, Jesse Glodstein, Shireen Elhabian

llms benchmarks

arXiv Machine Learning

Aug 4

TRACE-TS: Attribution-Grounded and Traceable Sensor-Language Reasoning for Human Activity Understanding

arXiv:2608. 00200v1 Announce Type: cross Abstract: Wearable sensors capture fine-grained motion patterns that support rich behavioral understanding, yet most existing methods reduce these signals to activity labels.

By Sparsh Rastogi, Tanmay Kumar, Baiyu Chen, Jatin Bedi, Zechen Li, Flora D. Salim

llms benchmarks

arXiv Machine Learning

Aug 4

CurveShift: Is Agent Progress Scalar? Separating Level from Shape

arXiv:2608. 00355v1 Announce Type: cross Abstract: Progress in large language models is often summarized using a single scalar measure, such as a time horizon, a latent ability estimate, or an aggregate benchmark score.

By Hanwen Xing, Pengyun Wang, BingXu Meng, Kumail Alhamoud, Xiang Li, Jicheng Wang, Xin Yu, Xinyang Han, Xiaomin Li, Philip Torr, Yuexing Hao

llms agents benchmarks

arXiv Machine Learning

Aug 4

Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors

arXiv:2608. 00675v1 Announce Type: cross Abstract: Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.

By Alexander Scheinker

diffusion benchmarks

arXiv Machine Learning

Aug 4

Staged Multi-Agent Training (SMAT) for Hip Exoskeletons: Metabolic and Biomechanical Validation of a Simulation-Trained Co-Adaptive Controller

arXiv:2608. 00715v1 Announce Type: cross Abstract: Learning-based controllers can deliver exoskeleton assistance after training entirely in physics-based simulation, yet few controllers that address human-device co-adaptation have been validated on real users by whole-body metabolic measurement, the standard benchmark for assistive walking.

By Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou

agents benchmarks

arXiv Machine Learning

Aug 4

PhenoStitch: Training-Free Panoptic Crop Mapping from Satellite Image Time Series

arXiv:2608. 00870v1 Announce Type: cross Abstract: Panoptic crop mapping requires both delineating individual agricultural parcels and assigning a crop type to each parcel from satellite image time series.

By Xuechen Li

computer-vision benchmarks

arXiv Machine Learning

Aug 4

Rethinking PPG-based Sleep Staging: Datasets, Metrics, and Benchmarks

arXiv:2608. 00943v1 Announce Type: cross Abstract: Automated sleep staging assigns discrete stage labels to successive time epochs throughout an overnight recording; conventionally each window spans at least 30 seconds, reflecting the minimum temporal resolution of the clinical scoring standard.

By Shuntian Zheng, Jiawei Wang, Cong Fu, Huan Yu, Chen Chen, Yu Guan, Sai Gu

benchmarks

arXiv Machine Learning

Aug 4

When Collaboration Becomes a Trigger: Collective Evidence-Threshold Backdoors in Multi-Agent Systems

arXiv:2608. 01085v1 Announce Type: cross Abstract: LLM-based multi-agent systems (MAS) extend LLM capabilities through iterative communication and shared contexts.

By Jia-Hao Xiao, Lei Feng, Min-Ling Zhang

llms agents benchmarks safety

arXiv Machine Learning

Aug 4

Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

arXiv:2608. 01193v1 Announce Type: cross Abstract: An AI development race creates a multi-agent safety dilemma.

By Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh, Phu Quy Nguyen Lam, Chi Nguyen Tran, Minh Trung Le, Phong Hao Le, Dinh Nam Nguyen, Thien Ky Nguyen Dong, Elias Fernandez Domingos, Le Hong Trang, The Anh Han

llms agents benchmarks safety

arXiv Machine Learning

Aug 4

Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning

arXiv:2608. 01418v1 Announce Type: cross Abstract: Autoregressive rollout generation is a major computational cost in reinforcement learning for large language models.

By Wenhao Zhang, Yibo Xie, Rui Wang, Jiahua Yang, Lei Jiang, Zibo Yang, Yawei Wang, Jiali Xu, jasperawang, Haoyang Long, Huan Xiong, alantzhao

llms reinforcement-learning benchmarks

Get the daily briefing