Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,095 stories · RSS feed

arXiv Machine Learning

Jul 17

Online Neural Space Time Memory for Dynamic Novel View Synthesis

arXiv:2607. 15271v1 Announce Type: cross Abstract: Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints.

By Baback Elmieh, Lynn Tsai, Zeman Li, Srinivas Kaza, Tiancheng Sun, Gabor Csapo, Ali Behrouz, Yuan Deng, Stephen Lombardi, Steven M. Seitz, Xuan Luo

benchmarks

arXiv Machine Learning

Jul 17

SafeOR-Gym: A Benchmark Suite for Safe Reinforcement Learning Algorithms on Practical Operations Research Problems

arXiv:2506. 02255v2 Announce Type: replace Abstract: Most existing safe reinforcement learning (RL) benchmarks focus on robotics and control tasks, offering limited relevance to high-stakes domains that involve structured constraints, mixed-integer decisions, and industrial complexity.

By Asha Ramanujam (Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN), Adam Elyoumi (Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN), Hao Chen (Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN), Sai Madhukiran Kompalli (Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN), Akshdeep Singh Ahluwalia (Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN), Shraman Pal (Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN), Dimitri J. Papageorgiou (Energy Sciences, ExxonMobil Technology and Engineering Company, Annandale, NJ), Can Li (Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN)

reinforcement-learning robotics benchmarks

arXiv AI

Jul 17

RAD: Retrieval High-quality Demonstrations to Enhance Decision-making

arXiv:2507. 15356v2 Announce Type: replace Abstract: Offline reinforcement learning (RL) learns policies from fixed datasets, thereby avoiding costly or unsafe environment interactions.

By Lu Guo, Yixiang Shan, Zhengbang Zhu, Qifan Liang, Lichang Song, Ting Long, Weinan Zhang, Yi Chang

agents reinforcement-learning benchmarks

arXiv AI

Jul 17

Why Git Is the Memory Solution for the Agentic Development Lifecycle

arXiv:2607. 14390v1 Announce Type: cross Abstract: Coding agents now produce a growing share of a team's code, while the reasoning behind each change -- the alternatives weighed, the constraints discovered, the approaches rejected -- is trapped in assistant transcripts that vanish with the session.

By Frank Guo

agents benchmarks

arXiv AI

Jul 17

Towards a Unified Multidimensional Explainability Metric: Evaluating Trustworthiness in AI Models

arXiv:2607. 14315v1 Announce Type: cross Abstract: In this paper, we present a comprehensive framework for assessing the explainability of various XAI methods, such as LIME and SHAP, across multiple datasets and machine learning models, with the ultimate goal of creating a unified multidimensional explainability score.

By Georgios Makridis, Georgios Fatouros, Athanasios Kiourtis, Dimitrios Kotios, Vasileios Koukos, Dimosthenis Kyriazis, Jonh Soldatos

rag benchmarks

arXiv Machine Learning

Jul 17

Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization

arXiv:2605. 21751v2 Announce Type: replace Abstract: Text-to-optimization requires two separable capabilities: modeling -- choosing the right optimization structure -- and binding -- grounding every coefficient, index, and parameter in the concrete problem data.

By Zhiqi Gao, Albert Ge, Alexander Berenbeim, Nathaniel D. Bastian, Frederic Sala

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 17

A Weak Penalty Neural ODE for Learning Chaotic Dynamics from Noisy Time Series

arXiv:2511. 06609v4 Announce Type: replace Abstract: The accurate forecasting of complex, high-dimensional dynamical systems from observational data is a fundamental task across numerous scientific and engineering disciplines.

By Xuyang Li, John Harlim, Dibyajyoti Chakraborty, Romit Maulik

benchmarks

arXiv AI

Jul 17

Heterogeneous Element-Aware Cross-Version Differencing of Scientific Documents via Layout-Aware Alignment and Structure-Aware Reasoning

arXiv:2607. 14117v1 Announce Type: cross Abstract: Cross-version differencing of scientific documents is essential in scholarly publishing and technical documentation, but remains challenging because scientific documents are page-structured artifacts containing heterogeneous elements such as text, tables, formulas, figures, and layout cues.

By Zhen Yina, Wenkang An, Hao Wang, Keran You

benchmarks safety

arXiv AI

Jul 17

VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence

arXiv:2607. 14510v1 Announce Type: new Abstract: Industrial time series serve as the foundation for Prognostics and Health Management (PHM) to ensure the reliability and safety of industrial equipment such as aero-engines.

By Haiteng Wang, Jingheng Yan, Xiaokang Wang, Lei Ren

llms multimodal benchmarks safety

arXiv AI

Jul 17

Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility

arXiv:2607. 14108v1 Announce Type: cross Abstract: This paper introduces tool efficiency, a new quantitative metric to evaluate the rate of useful tool calls in an LLM agent trajectory.

By Nyx Iskandar

llms agents benchmarks

arXiv AI

Jul 17

Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning

arXiv:2607. 14817v1 Announce Type: cross Abstract: Current evaluation of epistemic uncertainty relies on tasks such as out-ofdistribution detection and active learning.

By Jakub Paplh\'am, Willem Waegeman, Eyke H\"ullermeier, Vojt\v{e}ch Franc

benchmarks

arXiv AI

Jul 17

Copy-on-Write Scoring: Application-Specific Agent Evaluations

arXiv:2607. 14336v1 Announce Type: cross Abstract: Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows, with enough granularity to localize where they succeed and fail.

By Joanna Roy, Sven Hoelzel

llms agents benchmarks

arXiv AI

Jul 17

Automatically Evolving Prompt Guidelines for Task-Specific Optimization

arXiv:2607. 14105v1 Announce Type: cross Abstract: For Large Language Models to reliably answer user queries, users must clearly specify requirements, context, and constraints.

By Cedric Richter, Salah Ghamizi, Mike Papadakis

llms nlp benchmarks

arXiv AI

Jul 17

Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

arXiv:2607. 14552v1 Announce Type: cross Abstract: A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors.

By Jungseob Lee, Seungyoon Lee, Suhyune Son, Dongyub Jude Lee, Sungbin Han, Sugyeong Eo, Heuiseok Lim

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 17

TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoning

arXiv:2607. 14658v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) excel in general tasks, rigorous scientific reasoning remains challenging due to the limitations of monolithic, linear planning.

By Mingze Xu, Yinghui Li, Jiayi Kuang, Zhanhui Kang, Di Yin, Ying Shen, Xing Sun, Yuxing Han

llms agents multimodal benchmarks safety

arXiv AI

Jul 17

Beyond Visual Grasping: Benchmarking Complex Grasping from Detection to Execution

arXiv:2607. 14341v1 Announce Type: cross Abstract: Robust robotic grasping remains a fundamental challenge for complex real-world applications.

By Hanyi Zhang, Khang Nguyen, Charith Munasinghe, Basu Hela, Tianyu Li, Zihong Luo, Hoan Nguyen, Hans Wernher van de Venn, Yalin Zheng, Ravi Prakash, Tung D. Ta, Anh Nguyen, Baoru Huang

robotics benchmarks

arXiv AI

Jul 17

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

arXiv:2505. 17974v2 Announce Type: replace-cross Abstract: The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks.

By Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva, Andrey Kuznetsov, Evgeny Frolov

llms benchmarks

arXiv AI

Jul 17

Can Tokens Compete? Token Representations against Supervised CNN Backbones for BirdCLEF+ 2026

arXiv:2607. 14474v1 Announce Type: cross Abstract: This paper details the DS@GT ARC team's approach to BirdCLEF+ 2026, multi-label detection of animal vocalizations in soundscapes from the Pantanal wetlands.

By Anthony Miyaguchi, Murilo Gustineli, Adrian Cheung

rag benchmarks

arXiv AI

Jul 17

Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs

arXiv:2607. 14107v1 Announce Type: cross Abstract: The inference efficiency of diffusion large language models (dLLMs) is constrained by two challenges: bidirectional attention precludes efficient KV-cache reuse, while increasing decoding parallelism with static confidence thresholds can compromise generation quality.

By Mingyu Lee, Akshat Ramachandran, Souvik Kundu, Tushar Krishna

llms diffusion efficiency benchmarks

arXiv AI

Jul 17

Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

arXiv:2607. 14816v1 Announce Type: cross Abstract: Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias.

By Saima Afrin, Alessandro Midolo, Camilo Escobar-Vel\'asquez, Mario Linares-V\'asquez, Weiyuan Ding, Bowen Xu, Massimiliano Di Penta, Antonio Mastropaolo

llms benchmarks safety

Get the daily briefing