Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,220 stories · RSS feed

arXiv Machine Learning
Aug 4

Capability Provenance in Language Models: A Case Study in Social Reasoning

arXiv:2606. 19625v2 Announce Type: replace-cross Abstract: We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in OLMo3-7B.

By Glenn Matlin, Chandreyi Chakraborty, Saehee Eom, Mika Okamoto, Rayan Castilla, Louis Jaburi, Alvin Deng, Taywon Min, Lucia Quirke, Stella Biderman, Mark Riedl
arXiv Machine Learning
Aug 4

Self-Improving Large Language Models via Progressive Experience Evolution

arXiv:2608. 02139v1 Announce Type: cross Abstract: Large language models (LLMs) capable of self-improvement require not only effective policy optimization, but also a principled mechanism for transforming transient interaction experience into persistent model capabilities.

By Shijie Ren, Xiting Wang, Meng Li, Yujie Guo, Yunhang Yao, Ziheng Peng, Xunlong Wang, Yuetan Chen, Haoyang Zhou, Yunlong Liang, Fandong Meng
arXiv Machine Learning
Aug 4

Learned Digital Over-the-Air Computing for Federated Edge Learning

arXiv:2509. 16577v2 Announce Type: replace Abstract: Over-the-air (OTA) aggregation enables federated edge learning (FEEL) by exploiting the superposition property of the wireless channel to merge communication with computation, eliminating the need to schedule and decode devices individually.

By Antonio Tarizzo, Mohammad Kazemi, Deniz G\"und\"uz
arXiv Machine Learning
Aug 4

T-TAMER: Provably Taming Trade-offs in ML Serving

arXiv:2509. 22992v2 Announce Type: replace Abstract: As machine learning models continue to grow in size and complexity, efficient serving faces increasingly broad trade-offs spanning accuracy, latency, resource usage, and other objectives.

By Yuanyuan Yang, Ruimin Zhang, Jamie Morgenstern, Haifeng Xu
arXiv Machine Learning
Aug 4

EEG-FM-Compass: Progress, Benchmarking, and Future Directions for EEG Foundation Models

arXiv:2601. 17883v3 Announce Type: replace Abstract: Electroencephalography (EEG) foundation models (FMs) have recently emerged as a promising paradigm for brain-computer interfaces, aiming to learn transferable neural representations from large-scale heterogeneous recordings.

By Dingkun Liu, Yuheng Chen, Zhu Chen, Zhenyao Cui, Yaozhi Wen, Jiayu An, Jingwei Luo, Dongrui Wu
arXiv Machine Learning
Aug 4

Retrieval of Coastal Biogeochemical Parameters From Near-Surface Hyperspectral Remote Sensing Reflectance Using Physics-Aware Meta-Learning

arXiv:2605. 05623v3 Announce Type: replace Abstract: Hyperspectral in situ sensing has shown promise in retrieving aquatic biogeochemical (BGC) parameters, such as total suspended solids, dissolved organic carbon, and total chlorophyll-a, for cost-effective monitoring of coastal water quality.

By Yiqing Guo, Nagur R. C. Cherukuru, Eric A. Lehmann, S. L. Kesav Unnithan, Tim J. Malthus, Gemma Kerrisk, Xiubin Qi, Faisal Islam, Tisham Dhar, Mark J. Doubell