Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,794 stories · RSS feed

arXiv AI

Aug 3

Combining Large Language Models and Symbolic Reasoning for Multi-Robot Temporal Planning through Explainable Knowledge Bases

arXiv:2502. 19135v2 Announce Type: replace Abstract: We present PLANTOR, a framework for generating and executing multi-robot task plans from natural-language task descriptions through LLM-assisted knowledge-base construction.

By Enrico Saccon, Matteo Saveriano, Edoardo Lamon, Luigi Palopoli, Marco Roveri

llms robotics benchmarks

arXiv AI

Aug 3

M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities

arXiv:2601. 02854v2 Announce Type: replace Abstract: As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning.

By Ao Li, Jinghui Zhang, Luyu Li, Yuxiang Duan, Lang Gao, Mingcai Chen, Weijun Qin, Shaopeng Li, Fengxian Ji, Ning Liu, Lizhen Cui, Xiuying Chen, Yuntao Du

agents multimodal benchmarks safety

arXiv AI

Aug 3

Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

arXiv:2607. 22186v2 Announce Type: replace Abstract: Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse.

By Guanqun Zhao, Zijun Xie, Binbin Zheng, Enlei Gong, Jiafeng Lu, Yehan Yang, Aoqi Hu, Zeyu Chen

llms agents reinforcement-learning benchmarks

arXiv AI

Aug 3

Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery

arXiv:2603. 03322v2 Announce Type: replace-cross Abstract: Recent advancements in Large Language Model (LLM) agents have demonstrated remarkable potential in automatic knowledge discovery.

By Chaoqun Yang, Xinyu Lin, Shulin Li, Wenjie Wang, Ruihan Guo, Fuli Feng, Tat-Seng Chua

llms agents benchmarks

arXiv AI

Aug 3

The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

arXiv:2606. 21008v2 Announce Type: replace-cross Abstract: The metanym game is a competitive word game for LLMs that measures structural intelligence against established cognitive-science constructs.

By David Nordfors

llms benchmarks

arXiv Machine Learning

Aug 3

Fast Rates for Swap-Agnostic Learning of Proper Losses

arXiv:2607. 28856v1 Announce Type: new Abstract: Swap-agnostic learning strengthens classical agnostic learning by allowing the comparator to select a different hypothesis on each level set of the learner's predictions.

By Princewill Okoroafor

benchmarks

arXiv Machine Learning

Aug 3

Learning Optimal Dynamic Matching via Graph Neural Networks

arXiv:2607. 28925v1 Announce Type: new Abstract: Dynamic matching markets require decisions about whom to match and when: matching now yields value but removes participants who may create better future opportunities.

By Genta Okada, Shunya Noda, Junpei Komiyama, Akira Matsushita

benchmarks

arXiv Machine Learning

Aug 3

Mining Verdict Boundaries for Neural Network Verification

arXiv:2607. 28954v1 Announce Type: new Abstract: Branch and Bound (BaB) aims to achieve complete verification of neural networks by adaptively partitioning the problem and applying off-the-shelf verifiers to subproblems.

By Jiawei Ren, Guanqin Zhang, Zhenya Zhang, Yulei Sui

benchmarks

arXiv Machine Learning

Aug 3

Dynamics-aware identification of governing equations from sparse and noisy data

arXiv:2607. 29036v1 Announce Type: new Abstract: Sparse identification of nonlinear dynamics (SINDy) and PDE functional identification (PDE-FIND) recover parsimonious ordinary and partial differential equations (ODEs and PDEs) from data.

By Pongpisit Thanasutives, Yoshinobu Kawahara

diffusion benchmarks

arXiv Machine Learning

Aug 3

What Is Missing in Surgical Risk Stratification and Outcome Prediction: A Scoping Review of End-to-End Machine Learning Approaches

arXiv:2607. 29090v1 Announce Type: new Abstract: Postoperative adverse events, including mortality and morbidity, remain a major global burden, many of which are preventable through early identification of high-risk patients and targeted perioperative care.

By Yizhi Dong, Yuhe Ke, Hairil Rizal Abdullah, Yucheng Xing, Kevan Kai Bing Teo, Ling Huang, Mengling Feng

multimodal benchmarks

arXiv Machine Learning

Aug 3

Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery

arXiv:2607. 29225v1 Announce Type: new Abstract: Bayesian Optimization (BO) is widely adopted for data-efficient optimization in scientific and engineering applications, yet its computational cost is rarely evaluated alongside optimization performance.

By Panagiotis Krokidas, Christoforos Rekatsinas, Vassilis Sioros, Grigorios M. Chatziathanasiou, Efi-Maria Papia, George Giannakopoulos

robotics benchmarks

arXiv Machine Learning

Aug 3

Assessing the Generalization of Graph Neural Networks for Fault Location Across Increasing Distributed Energy Resource Penetration Levels

arXiv:2607. 29293v1 Announce Type: new Abstract: Accurate fault location is critical for distribution network reliability.

By Burak Karabulut, Olayiwola Arowolo, Carlo Manna, Chris Develder, Jochen L. Cremer

benchmarks

arXiv Machine Learning

Aug 3

Conditioning Tree-Based Diffusions and Flows for Probabilistic Tabular Regression

arXiv:2607. 28864v1 Announce Type: cross Abstract: Tree-based diffusion models fit flexible conditional predictive distributions for tabular regression without a neural density estimator, but they inherit their design defaults---noising path, parameterization, training distribution, features, sampler---from the neural setting.

By Silas Koemen

diffusion benchmarks

arXiv Machine Learning

Aug 3

Open-Source LLM-Driven Formal Verification: A Multi-Agent Pipeline for RTL Repair

arXiv:2607. 28877v1 Announce Type: cross Abstract: Verification consumes the majority of modern chip design effort, yet the formal verification tools that provide mathematical guarantees of correctness remain expensive and restrictively licensed.

By Ha Trung Tran

llms agents benchmarks

arXiv Machine Learning

Aug 3

SciFigPlag-Bench: A Benchmark for Provenance-Aware Scientific Figure Plagiarism Detection

arXiv:2607. 29124v1 Announce Type: cross Abstract: Scientific figures often encode the visual evidence behind scientific findings, yet figure plagiarism remains underexplored as a benchmarked multimodal evaluation problem.

By Zhiying Cui, Minghao Yang, Linlin Gao, Jie Liu, Pengyuan Li

llms multimodal benchmarks

arXiv Machine Learning

Aug 3

GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System

arXiv:2607. 29213v1 Announce Type: cross Abstract: Modern recommender systems in food delivery increasingly leverage multimodal signals, including images, text, and user interaction histories, to enhance user experience, yet effective fusion of these heterogeneous modalities remains challenging, hindering both the joint modeling of multimodal signals and adaptation to evolving user intent.

By Jiping Liu, Zhongmin Zhang, Zisen Sang, Zhijia Fang, Tao Ouyang, Ma Jiang, Shaopeng Liang, Zeyang Hou, Guodong Cao, Jia Jia

rag reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Aug 3

Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection

arXiv:2512. 22179v3 Announce Type: replace Abstract: Detecting previously unseen attacks remains a major challenge for machine learning-based intrusion detection systems.

By Rajeeb Thapa Chhetri, Saurab Thapa, Avinash Kumar, Zhixiong Chen

llms diffusion benchmarks

arXiv Machine Learning

Aug 3

GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems

arXiv:2602. 08913v3 Announce Type: replace Abstract: In underdetermined regression and classification problems, multiple feature subsets often yield equivalent predictive performance.

By Kate\v{r}ina Henclov\'a, V\'aclav \v{S}m\'idl

multimodal benchmarks

arXiv Machine Learning

Aug 3

LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning

arXiv:2603. 13319v2 Announce Type: replace Abstract: Diffusion Large Language Models (dLLMs) have emerged as a promising paradigm for parallel token generation, with block-wise variants garnering significant research interest.

By Yanzhe Hu, Yijie Jin, Pengfei Liu, Kai Yu, Zhijie Deng

llms diffusion reinforcement-learning benchmarks

arXiv Machine Learning

Aug 3

POSSE-kNN: Pathwise Out-of-Bag Selected Subspace Ensembles for Binary Classification

arXiv:2211. 11278v3 Announce Type: replace-cross Abstract: Nearest neighbour classification is attractive for tabular data, but its performance can deteriorate when a fixed query centred neighbourhood does not follow the local class geometry.

By Zardad Khan, Amjad Ali, Najd Adeed, Saeed Aldahmani

benchmarks

Get the daily briefing