Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,220 stories · RSS feed

arXiv AI

Aug 3

MOSAIC: Masked Outsourcing of Secure AI Computations

arXiv:2607. 29221v1 Announce Type: cross Abstract: We address the challenge of securely and efficiently outsourcing AI computations from a trusted but computationally weak client to an untrusted but powerful server, in the setting where the client holds both the input and the model, and the server must learn neither.

By James Hsin-yu Chiang, Sheila Zingg, Kari Kostiainen, Srdjan Capkun

llms efficiency benchmarks

arXiv AI

Aug 3

SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

arXiv:2607. 29209v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it.

By Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen

reinforcement-learning efficiency benchmarks

arXiv AI

Aug 3

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

arXiv:2607. 29252v1 Announce Type: cross Abstract: Reliable evaluation of open-ended LLM outputs requires fine-grained rubrics, yet expert curation is costly and difficult to scale.

By Mengting Chen, Yanshu Sun, Wanting Liang, Beidi Luan, Rui Sun, Dezhi Chen, Jing Li, Zuo Bai

llms benchmarks

arXiv AI

Aug 3

SeekBrain: An Autonomous Multi-Agent System for Accelerating Neuroscience Discovery

arXiv:2607. 29347v1 Announce Type: cross Abstract: Modern neuroscience relies on integrating multi-scale, multimodal datasets to uncover the neural principles underlying intelligence.

By Jiamin Wu, Peishan Xiang, Jingyang Chen, Yuqing Zhu, Yuxi Li, Ling Luo, Qihao Zheng, Jialiang Zu, Yongchao Wu, Mindong Liu, Haitao Wu, Chaofan Hu, Yijie Sun, Yuqi Hang, Yu Zhu, Shuo Li, Yue Fan, Shiyang Feng, Wanghan Xu, Tianlei Zhang, Jie Zhang, Wenlong Zhang, Bo Zhang, Kai Wang, Lei Bai, Mianxin Liu, Wanli Ouyang, Jiulin Du, Chunfeng Song

agents multimodal benchmarks

arXiv AI

Aug 3

Versatile On-device Adaptation at the Edge by Unifying Few-shot, Zero-shot, Continual, and In-context Learning

arXiv:2607. 29353v1 Announce Type: cross Abstract: With the ever-increasing pervasiveness of smart edge devices, the demand is growing for applications that can be tailored to users (e.

By Douwe den Blanken, Martin Lefebvre, Charlotte Frenkel

llms benchmarks safety

arXiv AI

Aug 3

Dense Temporal Contrast Synthesis via Conditioned Latent Transport

arXiv:2607. 29394v1 Announce Type: cross Abstract: Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is essential for breast cancer management, but reliance on gadolinium-based contrast agents (GBCAs) restricts use in contraindicated populations, prolongs scan protocols, and presents environmental toxicity concerns.

By Smriti Joshi, Apostolia Tsirikoglou, Daniel M. Lang, Richard Osuala, Noah M\'arquez Varaa, Alejandro Guzman, Grzegorz Skorupko, Sebastian Ibarra Arregui, Lidia Garrucho, Akane Ohashi, Dimitra Ntoula, Eugen Divjak, O\u{g}uz Lafc{\i}, Jan C. Peeken, Julia A. Schnabel, Fredrik Strand, Oliver Diaz, Karim Lekadir

agents computer-vision benchmarks safety

arXiv AI

Aug 3

Combining Large Language Models and Symbolic Reasoning for Multi-Robot Temporal Planning through Explainable Knowledge Bases

arXiv:2502. 19135v2 Announce Type: replace Abstract: We present PLANTOR, a framework for generating and executing multi-robot task plans from natural-language task descriptions through LLM-assisted knowledge-base construction.

By Enrico Saccon, Matteo Saveriano, Edoardo Lamon, Luigi Palopoli, Marco Roveri

llms robotics benchmarks

arXiv AI

Aug 3

M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities

arXiv:2601. 02854v2 Announce Type: replace Abstract: As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning.

By Ao Li, Jinghui Zhang, Luyu Li, Yuxiang Duan, Lang Gao, Mingcai Chen, Weijun Qin, Shaopeng Li, Fengxian Ji, Ning Liu, Lizhen Cui, Xiuying Chen, Yuntao Du

agents multimodal benchmarks safety

arXiv AI

Aug 3

Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

arXiv:2607. 22186v2 Announce Type: replace Abstract: Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse.

By Guanqun Zhao, Zijun Xie, Binbin Zheng, Enlei Gong, Jiafeng Lu, Yehan Yang, Aoqi Hu, Zeyu Chen

llms agents reinforcement-learning benchmarks

arXiv AI

Aug 3

Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery

arXiv:2603. 03322v2 Announce Type: replace-cross Abstract: Recent advancements in Large Language Model (LLM) agents have demonstrated remarkable potential in automatic knowledge discovery.

By Chaoqun Yang, Xinyu Lin, Shulin Li, Wenjie Wang, Ruihan Guo, Fuli Feng, Tat-Seng Chua

llms agents benchmarks

arXiv AI

Aug 3

The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

arXiv:2606. 21008v2 Announce Type: replace-cross Abstract: The metanym game is a competitive word game for LLMs that measures structural intelligence against established cognitive-science constructs.

By David Nordfors

llms benchmarks

arXiv Machine Learning

Aug 3

Fast Rates for Swap-Agnostic Learning of Proper Losses

arXiv:2607. 28856v1 Announce Type: new Abstract: Swap-agnostic learning strengthens classical agnostic learning by allowing the comparator to select a different hypothesis on each level set of the learner's predictions.

By Princewill Okoroafor

benchmarks

arXiv Machine Learning

Aug 3

Learning Optimal Dynamic Matching via Graph Neural Networks

arXiv:2607. 28925v1 Announce Type: new Abstract: Dynamic matching markets require decisions about whom to match and when: matching now yields value but removes participants who may create better future opportunities.

By Genta Okada, Shunya Noda, Junpei Komiyama, Akira Matsushita

benchmarks

arXiv Machine Learning

Aug 3

Mining Verdict Boundaries for Neural Network Verification

arXiv:2607. 28954v1 Announce Type: new Abstract: Branch and Bound (BaB) aims to achieve complete verification of neural networks by adaptively partitioning the problem and applying off-the-shelf verifiers to subproblems.

By Jiawei Ren, Guanqin Zhang, Zhenya Zhang, Yulei Sui

benchmarks

arXiv Machine Learning

Aug 3

Dynamics-aware identification of governing equations from sparse and noisy data

arXiv:2607. 29036v1 Announce Type: new Abstract: Sparse identification of nonlinear dynamics (SINDy) and PDE functional identification (PDE-FIND) recover parsimonious ordinary and partial differential equations (ODEs and PDEs) from data.

By Pongpisit Thanasutives, Yoshinobu Kawahara

diffusion benchmarks

arXiv Machine Learning

Aug 3

What Is Missing in Surgical Risk Stratification and Outcome Prediction: A Scoping Review of End-to-End Machine Learning Approaches

arXiv:2607. 29090v1 Announce Type: new Abstract: Postoperative adverse events, including mortality and morbidity, remain a major global burden, many of which are preventable through early identification of high-risk patients and targeted perioperative care.

By Yizhi Dong, Yuhe Ke, Hairil Rizal Abdullah, Yucheng Xing, Kevan Kai Bing Teo, Ling Huang, Mengling Feng

multimodal benchmarks

arXiv Machine Learning

Aug 3

Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery

arXiv:2607. 29225v1 Announce Type: new Abstract: Bayesian Optimization (BO) is widely adopted for data-efficient optimization in scientific and engineering applications, yet its computational cost is rarely evaluated alongside optimization performance.

By Panagiotis Krokidas, Christoforos Rekatsinas, Vassilis Sioros, Grigorios M. Chatziathanasiou, Efi-Maria Papia, George Giannakopoulos

robotics benchmarks

arXiv Machine Learning

Aug 3

Assessing the Generalization of Graph Neural Networks for Fault Location Across Increasing Distributed Energy Resource Penetration Levels

arXiv:2607. 29293v1 Announce Type: new Abstract: Accurate fault location is critical for distribution network reliability.

By Burak Karabulut, Olayiwola Arowolo, Carlo Manna, Chris Develder, Jochen L. Cremer

benchmarks

arXiv Machine Learning

Aug 3

Conditioning Tree-Based Diffusions and Flows for Probabilistic Tabular Regression

arXiv:2607. 28864v1 Announce Type: cross Abstract: Tree-based diffusion models fit flexible conditional predictive distributions for tabular regression without a neural density estimator, but they inherit their design defaults---noising path, parameterization, training distribution, features, sampler---from the neural setting.

By Silas Koemen

diffusion benchmarks

arXiv Machine Learning

Aug 3

Open-Source LLM-Driven Formal Verification: A Multi-Agent Pipeline for RTL Repair

arXiv:2607. 28877v1 Announce Type: cross Abstract: Verification consumes the majority of modern chip design effort, yet the formal verification tools that provide mathematical guarantees of correctness remain expensive and restrictively licensed.

By Ha Trung Tran

llms agents benchmarks

Get the daily briefing