Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,305 stories · RSS feed

arXiv Machine Learning

Jun 3

WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents

arXiv:2605. 20306v2 Announce Type: replace-cross Abstract: We introduce WildRoadBench, a wild aerial road-damage grounding benchmark that couples direct visual grounding by vision-language models with autonomous research-and-engineering by LLM-driven agents on a single professionally annotated UAV corpus.

By Bingnan Liu, Chenhang Cui, Rui Huang, Jiani Luo, Zhirong Shen, Tinghao Wang, Xiande Huang, Lingbei Meng, Fei Shen, An Zhang

llms agents multimodal benchmarks

arXiv AI

Jun 3

AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following

arXiv:2606. 03116v1 Announce Type: cross Abstract: The rapid advancement of instruction-guided audio generation has highlighted the critical need for robust alignment evaluation.

By Haitao Li, Tian Tan, Yuguang Yang, Shan Yang, Xie Chen

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 3

LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory for Trajectory Tracking and Stabilization

arXiv:2602. 04132v4 Announce Type: replace-cross Abstract: Reinforcement Learning (RL) has achieved remarkable success in solving complex sequential decision-making problems.

By Dhruv S. Kushwaha, Zoleikha A. Biron

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 3

Towards Blind Lens Aberration Correction via Large LensLib Pre-training and Discrete Degradation Priors

arXiv:2511. 17126v4 Announce Type: replace-cross Abstract: Emerging deep-learning-based lens library pre-training (LensLib-PT) pipeline offers a new avenue for blind lens aberration correction by training a universal neural network, demonstrating strong capability in handling diverse unknown optical degradations.

By Xiaolong Qian, Qi Jiang, Yao Gao, Lei Sun, Kailun Yang, Xian Wang, Zhonghua Yi, Wenyong Li, Ming-Hsuan Yang, Luc Van Gool, Kaiwei Wang

efficiency benchmarks

arXiv AI

Jun 3

Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation

arXiv:2606. 03128v1 Announce Type: cross Abstract: Smart contracts face critical security challenges that require thorough auditing in decentralized web services.

By Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan, Guntur Dharma Putra

llms efficiency benchmarks safety

arXiv AI

Jun 3

Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Models

arXiv:2601. 12247v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) present a promising non-sequential paradigm for text generation, distinct from standard autoregressive (AR) approaches.

By Miao Li, Hanyang Jiang, Sikai Cheng, Hengyu Fu, Yuhang Cai, Baihe Huang, Tinghan Ye, Xuanzhou Chen, Pascal Van Hentenryck

llms diffusion benchmarks

arXiv AI

Jun 3

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

arXiv:2512. 18552v3 Announce Type: replace-cross Abstract: While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.

By Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, Sida Wang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 3

High-Dimensional Latents Should Be Diagnosed Through Phase Structure

arXiv:2606. 02600v1 Announce Type: cross Abstract: We study autoencoder and variational-autoencoder latent spaces through the lens of spin-glass theory.

By Alejandro Ascarate, Leo Lebrat, Rodrigo Santa Cruz, Clinton Fookes, Olivier Salvado

benchmarks

arXiv AI

Jun 3

Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach

arXiv:2510. 23216v4 Announce Type: replace Abstract: While several high profile video games have served as testbeds for Deep Reinforcement Learning (DRL), this technique has rarely been employed by the game industry for crafting authentic AI behaviors.

By Alessandro Sestini, Joakim Bergdahl, Jean-Philippe Barrette-LaPierre, Florian Fuchs, Brady Chen, Fabio Zinno, Michael Jones, Linus Gissl\'en

agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 3

Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

arXiv:2606. 03980v1 Announce Type: new Abstract: Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines.

By Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, Qinliang Su, Xiaoxi Jiang, Guanjun Jiang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 3

PSViT: A Methodology for Structurally Pruning Spiking Vision Transformers

arXiv:2606. 03257v1 Announce Type: cross Abstract: Spiking Vision Transformer (SViT) models are promising low-power ViT models for solving vision-based tasks with state-of-the-art performance.

By Rachmad Vidya Wicaksana Putra, Achyuta Muthuvelan, Alberto Marchisio, Muhammad Shafique

llms computer-vision fine-tuning efficiency benchmarks

arXiv AI

Jun 3

Adaptive Latent Agentic Reasoning

arXiv:2606. 02871v1 Announce Type: cross Abstract: Large reasoning models improve performance by generating extended chain-of-thought (CoT) reasoning, but this behavior becomes inefficient when applied to LLM agents.

By Dongwon Jung, Peng Shi, Yi Zhang, Junshan Zhang, Muhao Chen

llms agents benchmarks

arXiv AI

Jun 3

GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory

arXiv:2606. 03144v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as self-study assistants in technical disciplines, yet their reliability as mathematical reasoning assistants remains poorly understood.

By Noujoud Nader, Ibrahem Aljabea, Patrick Diehl, Deepti Gupta

llms benchmarks

arXiv Machine Learning

Jun 3

ReciNet: Reciprocal Space-Aware Long-Range Modeling for Crystalline Property Prediction

arXiv:2502. 02748v4 Announce Type: replace Abstract: Predicting properties of crystals from their structures is a fundamental yet challenging task in materials science.

By Jianan Nie, Peiyao Xiao, Kaiyi Ji, Peng Gao

benchmarks

arXiv AI

Jun 3

From Long News to Accurate Forecast: Importance-Aware Fusion and PRM-Guided Reflection for Time Series Forecasting

arXiv:2606. 03097v1 Announce Type: new Abstract: Incorporating news into time series forecasting is appealing because news can reveal abrupt exogenous events that historical values alone cannot recover.

By Mingyang Liu, Qingcan Kang, Yuke Wang, Shixiong Kai, Kaichao Liang, Hui-Ling Zhen, Tao Zhong, Mingxuan Yuan, Linqi Song

llms reinforcement-learning benchmarks

arXiv AI

Jun 3

PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios

arXiv:2602. 05302v3 Announce Type: replace Abstract: We present an in-depth evaluation of LLMs' ability to negotiate, a central business task requiring strategic reasoning, theory of mind, and economic value creation.

By Chris Zhu, Sasha Cui, Will Sanok Dufallo, Runzhi Jin, Zhen Xu, Linjun Zhang, Daylian Cain

llms agents benchmarks

arXiv AI

Jun 3

What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents

arXiv:2606. 02965v1 Announce Type: new Abstract: Benchmarks for autonomous agents measure whether agents complete tasks, yet this framing is systematically blind to whether an agent should have proceeded at all.

By Victor Ojewale, Suresh Venkatasubramanian

agents benchmarks safety

arXiv Machine Learning

Jun 3

Forecasting Conceptual Diffusion in Science: The Case of Quantum Computing

arXiv:2606. 03919v1 Announce Type: cross Abstract: Understanding and anticipating scientific change requires models that distinguish between endogenous consolidation and exogenous diffusion of scientific concepts.

By Thomas Maillart, Thibaut Chataing, David Dosu, Paul Bagourd, Julian Jang-Jaccard, Alain Mermoud

diffusion robotics benchmarks

arXiv AI

Jun 3

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

arXiv:2601. 08173v2 Announce Type: replace Abstract: The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets performance upper bounds in static environments, overlooking robustness for stochastic real-world deployment.

By Daocheng Fu, Jianbiao Mei, Rong Wu, Xuemeng Yang, Jia Xu, Ding Wang, Pinlong Cai, Yong Liu, Licheng Wen, Botian Shi

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 3

RESCAST-100K: A Comprehensive Dataset for Cross-Domain Residential Load and Indoor Temperature Forecasting

arXiv:2606. 02852v1 Announce Type: new Abstract: Accurate short-term forecasting of residential energy load and indoor temperature is essential for home energy management systems, grid-level demand response, and community energy efficiency efforts.

By Jainam Dhruva, Yousaf Raza, A. B. Siddique, Simone Silvestri

llms robotics fine-tuning benchmarks

Get the daily briefing