Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv AI

Jul 21

When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

arXiv:2510. 22228v2 Announce Type: replace-cross Abstract: Layer pruning has emerged as a widely adopted technique for improving the efficiency of large language models (LLMs).

By Keyu Wang, Tian Lyu, Guinan Su, Lu Yin, Marco Canini, Jonas Geiping, Shiwei Liu

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 21

Exploratory and Assimilating Reflection: Reflective Recall Cycle for Long-term Memory

arXiv:2607. 17879v1 Announce Type: new Abstract: LLM-based autonomous agents require external memory to overcome their statelessness and limited context window for long-term interaction and dynamic knowledge reasoning.

By Ganesh Senrayan, Moyuru Yamada, Ishan Jindal, Kiran Purohit

llms agents benchmarks

arXiv AI

Jul 21

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

arXiv:2607. 16243v1 Announce Type: cross Abstract: Multimodal industrial anomaly inspection assistants are a critical component of next-generation smart factories, enabling interactive vision-language-based querying.

By Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

llms multimodal benchmarks safety

arXiv Machine Learning

Jul 21

Interpretable Factor Decomposition for Decision Intelligence in Large-Scale Financial Markets: Evidence from China's A-Share Market

arXiv:2606. 12843v2 Announce Type: replace Abstract: We present an interpretable machine learning pipeline to decompose cross-sectional equity return predictability into auditable factor contributions.

By Xiao Han, Yao Xiao, Zhen Zhang, Moxuan Zheng

benchmarks

arXiv AI

Jul 21

Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.

By Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin

llms agents multimodal benchmarks

arXiv AI

Jul 21

Joint-Embedding Predictive Architecture for Sensor-based Activity Recognition

arXiv:2607. 16350v1 Announce Type: cross Abstract: Sensor-based human activity recognition (HAR) has achieved significant progressed in fully supervised learning settings.

By Mohd Halim Mohd Noor, Abdulrahman M. A. Baraka

rag benchmarks

arXiv AI

Jul 21

Privacy-Aware Synthetic Video Benchmarking and Relational Evaluation for Worker-Under-Suspended-Load Detection

arXiv:2607. 16351v1 Announce Type: cross Abstract: Publicly shareable construction-video benchmarks remain scarce, especially for safety-critical hazards that are rare, dangerous to stage, and difficult to release.

By Anshu Singh, Alejandro Seif

diffusion computer-vision benchmarks safety

arXiv AI

Jul 21

Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration

arXiv:2607. 16352v1 Announce Type: cross Abstract: A fundamental intent asymmetry plagues modern 3D asset creation: while state-of-the-art 3D toolchains demand precise, executable parameters, ordinary users typically provide vague, underspecified instructions.

By Xiaoye Zhu, Weixin Li, Junan Huo, Bozhong Wang, Jia Zeng, Yi Yang, Cen Chen, Qi Liu

agents benchmarks

arXiv AI

Jul 21

A Quantum-Classical Hybrid Framework for Multivariate Time-Series Forecasting Complexity-Fidelity Trade-offs and Limitations

arXiv:2607. 16358v1 Announce Type: cross Abstract: This paper presents a unified quantum-classical hybrid framework for multi-horizon time-series forecasting, introducing two model variants Quantum Reservoir Forecaster (QRC-F) and Variational Quantum Forecaster (VQF-F).

By Sanjay Chakraborty, Fredrik Heintz

efficiency benchmarks

arXiv AI

Jul 21

Fantastic Adaptive Taxonomies and How to Use Them

arXiv:2607. 16387v1 Announce Type: cross Abstract: An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback.

By Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica

llms agents benchmarks

arXiv AI

Jul 21

AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Language

arXiv:2607. 16372v1 Announce Type: cross Abstract: Interactive theorem proving (ITP) underpins program verification and formalized mathematics, but its manual effort limits scalability.

By Qiyuan Xu, Joshua Ong Jun Leang, Renxi Wang, Wenda Li, Haonan Li, Luke Ong, Conrad Watt

llms agents benchmarks

arXiv AI

Jul 21

Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

arXiv:2607. 16448v1 Announce Type: cross Abstract: Interpretability methods for neural network activations span a wide cost spectrum, from cheap, training-free techniques (such as linear probes, PCA, SVD) to more expensive training-based ones (such as SAEs and activation oracles).

By Sriram Balasubramanian, Soheil Feizi

llms rag agents diffusion benchmarks safety

arXiv AI

Jul 21

Geometry-Enhanced Portion Estimation for Multimodal LLMs

arXiv:2607. 16514v1 Announce Type: cross Abstract: Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker.

By Lin Liao, Peng Li

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

arXiv:2607. 16476v1 Announce Type: cross Abstract: Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade.

By Chao Jiang, Yulong Ye, Tao Chen, Miqing Li

benchmarks

arXiv Machine Learning

Jul 21

Kernel Regression with Tensor Trains and Hadamard Overparameterization

arXiv:2607. 17390v1 Announce Type: cross Abstract: Kernel regression with tensor trains and Hadamard overparameterization (KReTTaH) is introduced as a training-data-free, interpretable, and nonparametric framework for multi-way data imputation.

By Duc Thien Nguyen, Konstantinos Slavakis, Eleftherios Kofidis, Dimitris Pados

efficiency benchmarks

arXiv AI

Jul 21

Building2Building: A Large Scale Benchmark for Generalizable Real-World Reinforcement Learning

arXiv:2607. 16534v1 Announce Type: cross Abstract: Reinforcement learning (RL) has achieved strong results in control, yet learned policies remain brittle to changes in dynamics, action spaces, observation spaces, or goals, a critical limitation for real-world deployment.

By Vincent Taboga, Justin Veilleux, Doseok Jang, Anushree Rankawat, Pierre-Luc Bacon

reinforcement-learning benchmarks

arXiv AI

Jul 21

Pediatric Bone Age Prediction Using Deep Learning

arXiv:2607. 16936v1 Announce Type: cross Abstract: Pediatric bone age prediction is a crucial task in clinical practice that can help diagnose endocrine disorders and provide insight into a child's growth and development.

By Al Zadid Sultan Bin Habib, Md. Ekramul Islam, Md Asif Bin Syed, Md Younus Ahamed, Tanpia Tasnim

computer-vision benchmarks

arXiv AI

Jul 21

DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines

arXiv:2607. 16617v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not automatically materialized as persistent, editable platform artifacts.

By Runming He, Zhen Hao Wong, Hao Liang, Zimo Meng, Chengyu Shen, Xiaochen Ma, Wentao Zhang

llms agents benchmarks

arXiv AI

Jul 21

Privacy Cost as Equity Input: A Group Fairness Criterion for Differentially Private Machine Learning

arXiv:2607. 16620v1 Announce Type: cross Abstract: Differential privacy (DP) is increasingly deployed to limit membership inference risk in machine-learning systems.

By Rakshit Naidu

nlp benchmarks safety

arXiv AI

Jul 21

What Do They See? Interpreting Complex Road Scenarios Through the Eyes of Vision-Language-Action Models for Safe and Trustworthy Autonomous Vehicle Learning

arXiv:2607. 16938v1 Announce Type: cross Abstract: End-to-end autonomous driving models are now able to navigate complex road scenarios, mapping raw sensor observations directly to observed paths for open-loop evaluation and often effective driving in closed-loop evaluation.

By Kalpana Panda, Wesley Maia, Vinti Agarwal, Ross Greer

agents robotics multimodal benchmarks safety

Get the daily briefing