Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,813 stories · RSS feed

arXiv AI

Jul 29

ScaleResfusion: Residual Rectified Flow based on Residual Vector Field

arXiv:2607. 25275v1 Announce Type: cross Abstract: Real-world Image Restoration (Real-IR) aims to recover high-quality (HQ) images from complex and unknown degradations.

By Zhenning Shi, Chen Xu, Junhao Zhang, Kefei Zhang, Linjie Liu, Zhedong Zheng, Tao Li

diffusion fine-tuning efficiency benchmarks

arXiv AI

Jul 29

OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation

arXiv:2607. 25656v1 Announce Type: new Abstract: Complex tasks often decompose into parallelizable yet interdependent subtasks, making orchestration critical to the performance of multi-agent systems (MAS).

By Zhenzhen Ren, Jiyan He, Xinpeng Zhang, Zhenxing Qian, Ke Han, Shuxin Zheng, GuoBiao Li, Xiaoqing Zhang

llms agents benchmarks

arXiv AI

Jul 29

Distributed Constraint Optimization via Online Learning and Iterative Pricing with Application to Large-Scale Satellite Scheduling

arXiv:2607. 25835v1 Announce Type: new Abstract: Distributed constraint optimization problems (DCOPs) provide a popular framework for distributed decision making under limited communication, but many real-world instances are too large to solve monolithically.

By Itai Zilberstein, Pranav Rajbhandari, Steve Chien, Tuomas Sandholm

benchmarks

arXiv AI

Jul 29

CADENCE: A Cardiac Atom Dictionary for Interpretable Neural Concept Extraction from ECG Foundation Models

arXiv:2607. 25244v1 Announce Type: new Abstract: Foundation models for 12-lead electrocardiograms (ECGs) transfer well across clinical tasks, but the physiological knowledge encoded in their representations remains opaque.

By Yixuan Duan, Arjun Naik, Sadeer Al-Kindi, Wei Qiu

llms rag benchmarks

arXiv AI

Jul 29

RRS-10K: A Multitask Vision-Language Model Benchmark for Rare Remote Sensing Image Interpretation

arXiv:2607. 24810v1 Announce Type: new Abstract: Vision-language models (VLMs) have achieved strong performance on general remote sensing tasks.

By Yuqiao Lai, Jiancheng Qi, Fei Wang, Yuxin Liu, Kun Li, Ye Chen, Yan Gao, Yanyan Wei

llms computer-vision multimodal benchmarks

arXiv AI

Jul 29

HiSkill: Empowering LLM Agents with Hierarchical Skill Graphs

arXiv:2607. 25853v1 Announce Type: new Abstract: Skills have become an important abstraction for enabling large language model (LLM) agents to reuse past experience in long-horizon interactive tasks.

By Yu Hao, Jinxuan Cai, Qi Zhang, Yawen Li, Zhiqiang Zhang, Chuan Shi, Cheng Yang

llms agents benchmarks

arXiv AI

Jul 29

Reliability Scales Inversely: Hallucinations Snowball Faster in Bigger Language Models

arXiv:2607. 18292v3 Announce Type: replace-cross Abstract: Bigger language models are less reliable.

By Kushal Chakrabarti

llms benchmarks safety

arXiv AI

Jul 29

Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

arXiv:2607. 25257v1 Announce Type: cross Abstract: Item Response Theory (IRT) has recently been proposed as a framework for evaluating large language model (LLM) benchmarks by separating a model's latent ability from the properties of individual benchmark items.

By Juan Francisco, Mandujano Reyes

llms benchmarks

arXiv AI

Jul 29

Penelope: Localized Latent Recurrence for Efficient Structured Reasoning

arXiv:2607. 25915v1 Announce Type: new Abstract: Complex structured reasoning tasks often require additional computation, yet current language models obtain it mainly by increasing parameter scale or by serializing intermediate steps as chain-of-thought (CoT) tokens.

By Yutong Chen, Shouqian Shi, Xinran Liu, Haochen Wang, Jiaying Wang, Tianxing Xu, Yuanxi Wang, Zirui Ding

llms diffusion efficiency benchmarks

arXiv AI

Jul 29

dtControl2+$\varepsilon$: Trading Optimality for Explainability in MDPs via Decision Trees

arXiv:2607. 25925v1 Announce Type: new Abstract: Over the past decade, decision trees have been used to represent controllers (a.

By Tereza Kinsk\'a, Jan K\v{r}et\'insk\'y, Tobias Meggendorfer, Sabine Rieder, Maximilian Weininger

reinforcement-learning benchmarks

arXiv AI

Jul 29

Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

arXiv:2607. 24996v1 Announce Type: cross Abstract: Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning.

By Luc McCutcheon, Evangelos Chatzaroulas, Saber Fallah

reinforcement-learning benchmarks

arXiv AI

Jul 29

Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

arXiv:2607. 25527v1 Announce Type: cross Abstract: Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities.

By Weiming Zhuang, Jiabo Huang, Jingtao Li, Zhizhong Li, Chen Chen, Sina Sajadmanesh, Lingjuan Lyu

llms multimodal benchmarks

arXiv AI

Jul 29

Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA

arXiv:2607. 25921v1 Announce Type: cross Abstract: In this work, we study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance (QA) pipeline focusing on geometry clipping.

By Carlos Celemin, Benedict Wilkins, Adri\'an Barahona-R\'ios, Saman Zadtootaghaj, Nabajeet Barman

llms agents multimodal benchmarks

arXiv AI

Jul 29

VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents

arXiv:2607. 24748v1 Announce Type: cross Abstract: Visually-rich documents such as reports, slides, and manuals often distribute the evidence needed to answer a question across multiple pages, mixing text with layout cues, tables, charts, and figures.

By Seonok Kim

rag agents nlp multimodal benchmarks

arXiv AI

Jul 29

RIDGE: An Autonomous Framework for Validation and Method Discovery in LLM-Generated Option Pricing

arXiv:2607. 25199v1 Announce Type: cross Abstract: Automated code generation is becoming an important tool in quantitative finance, where large language models can generate option pricing implementations directly from mathematical model specifications.

By Liexin Cheng, Xue Cheng, Shuaiqiang Liu, Cornelis W. Oosterlee

llms agents benchmarks

arXiv AI

Jul 29

CondPSE: A Polynomial-Filtered Structural Encoder with Conditional Modulation for Graphs

arXiv:2607. 25169v1 Announce Type: cross Abstract: Message-passing graph neural networks are bounded by the 1-WL test and can miss topological structure that distinguishes non-isomorphic graphs.

By Woohyun Lee, Hogun Park

benchmarks

arXiv AI

Jul 29

From Deterministic to Generative Deep Learning for Urban Air Quality Reconstruction from Sparse Observations

arXiv:2607. 25687v1 Announce Type: cross Abstract: Full-field reconstruction of air pollution is essential for evaluating pollution exposure and supporting public health decision-making.

By Abhishek A. Sabnis, Mihai Mitrea, Lya Lugon, Karine Sartelet, Marc Bocquet, Xiaoyuan Cheng, Shupeng Zhu, Sibo Cheng

diffusion benchmarks

arXiv AI

Jul 29

CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition

arXiv:2607. 25294v1 Announce Type: cross Abstract: Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge.

By Lai Wei, Chengqi Li, Jiapeng Li, Ruina Hu, Yue Wang, Weiran Huang

nlp multimodal benchmarks

arXiv AI

Jul 29

Localized Anomaly Detection via Differentiable D-vine Copulas

arXiv:2607. 25020v1 Announce Type: new Abstract: Vine copulas provide a flexible framework for modeling complex multivariate distributions through a hierarchical decomposition into bivariate pair-copulas.

By Nicholas Andrea Pearson, Francesca Zanello, Davide Russo, Luca Bortolussi, Francesca Cairoli

benchmarks

arXiv AI

Jul 29

CORF-GS: Real-Time Wireless Radiance Field Reconstruction via Coupled Optical-RF Gaussian Splatting

arXiv:2607. 25569v1 Announce Type: cross Abstract: Recent advances in 3D Gaussian Splatting (3DGS)-based wireless radiance field (WRF) reconstruction provide an efficient solution for wireless channel modeling.

By Jinya Zhang, Jiajia Guo, Chao-Kai Wen, Shi Jin

benchmarks

Get the daily briefing