Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,494 stories · RSS feed

arXiv AI

Jun 2

Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

arXiv:2602. 08058v3 Announce Type: replace-cross Abstract: In the presence of occlusions and measurement noise, geometrically accurate scene reconstructions -- which fit the sensor data -- can still be physically incorrect.

By Xihang Yu, Rajat Talak, Lorenzo Shaikewitz, Luca Carlone

benchmarks

arXiv Machine Learning

Jun 2

KG-Guard: Graph-Based Hallucination Detection for Knowledge Base Question Answering

arXiv:2606. 00328v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for knowledge base question answering (KBQA), where answering requires selecting entities from a question-specific knowledge-graph subgraph.

By Albert Sawczyn, Piotr Bielak, Tomasz Kajdanowicz

llms rag nlp benchmarks safety

arXiv Machine Learning

Jun 2

Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacity

arXiv:2606. 00414v1 Announce Type: new Abstract: When many reinforcement-learning policies achieve near-optimal return, a post-hoc auditor may have to distinguish among many behaviorally distinct but return-equivalent policies.

By Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

GLENS: Global Search via Learning from Solver Iterates with Diffusion Models

arXiv:2606. 00366v1 Announce Type: new Abstract: We consider the problem of generating a large collection of initial guesses for local minima of multimodal non-convex continuous optimization problems.

By Anjian Li, Bartolomeo Stellato, Ryne Beeson

diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jun 2

CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representations

arXiv:2606. 02221v1 Announce Type: cross Abstract: Multi-task learning (MTL) aims to construct a joint model for multiple tasks by sharing a common representation across domains.

By Chengfeng Wu, Tao Zou, Yanru Wu, Jingge Wang

benchmarks

arXiv Machine Learning

Jun 2

Echo State Networks for Time Series Forecasting: Hyperparameter Sweep and Benchmarking

arXiv:2602. 03912v4 Announce Type: replace Abstract: This paper investigates the performance of Echo State Networks (ESNs) for univariate forecasting of monthly and quarterly time series from the M4 Forecasting Competition dataset.

By Alexander H\"au{\ss}er

benchmarks

arXiv AI

Jun 2

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605. 09883v2 Announce Type: replace-cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual understanding?

By Xia Hu, Zhenrui Yue, Brian Potetz, Howard Zhou, Leonidas Guibas, Chun-Ta Lu, Zhicheng Wang

llms multimodal benchmarks

arXiv AI

Jun 2

PBT-Bench: Benchmarking AI Agents on Property-Based Testing

arXiv:2605. 15229v3 Announce Type: replace-cross Abstract: Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue.

By Lucas Jing, Xinqi Wang, Liao Zhang, Simon S. Du

llms agents benchmarks

arXiv Machine Learning

Jun 2

ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?

arXiv:2606. 01849v1 Announce Type: new Abstract: Differentially private (DP) text synthesis promises to unlock sensitive corpora for model training, but it remains unclear whether DP synthetic data transmits genuinely new knowledge and capabilities present only in those corpora.

By Peihan Liu, Lucas Rosenblatt, Weiwei Kong, Natalia Ponomareva, Gautam Kamath, Rachel Cummings, Roxana Geambasu, Yu Gan, Lillian Tsai, Alex Bie

benchmarks

arXiv AI

Jun 2

Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning

arXiv:2605. 27000v2 Announce Type: replace-cross Abstract: Repeated sampling with a verifier is the standard way to allocate test-time compute for code generation, with pass@$K$ as the canonical metric.

By Yilong Li, Suman Banerjee, Tong Che

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 2

ELF: A Family of Encoder-Free ECG-Language Models

arXiv:2601. 18798v2 Announce Type: replace-cross Abstract: ECG-Language Models (ELMs) extend recent advances in Multimodal Large Language Models (MLLMs) to automated ECG interpretation.

By William Han, Tony Chen, Chaojing Duan, Xiaoyu Song, Yihang Yao, Yuzhe Yang, Michael A. Rosenberg, Emerson Liu, Ding Zhao

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

arXiv:2603. 11653v2 Announce Type: replace Abstract: Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments.

By Jiaheng Hu, Jay Shim, Chen Tang, Yoonchang Sung, Bo Liu, Peter Stone, Roberto Martin-Martin

agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

MASCOT: Towards Multi-Agent Socio-Collaborative Companion Systems

arXiv:2601. 14230v2 Announce Type: replace-cross Abstract: Multi-agent systems (MAS) are emerging as promising socio-collaborative companions for emotional and cognitive support.

By Yiyang Wang, Yiqiao Jin, Alex Cabral, Josiah Hester

llms agents benchmarks safety

arXiv Machine Learning

Jun 2

Scalable Inference-Time Annealing with Surrogate Likelihood Estimators

arXiv:2605. 31498v2 Announce Type: replace Abstract: A long standing challenge in computational chemistry and biophysics is efficiently sampling the Boltzmann distribution of molecules.

By Daniel Pe\~naherrera, Rishal Aggarwal, David Ryan Koes

diffusion fine-tuning benchmarks

arXiv Machine Learning

Jun 2

Learning-based Directed Graph Abstraction of Combinatorial Spaces for Order-Preserving Search in Mixed-Combinatorial Nonlinear Optimization

arXiv:2606. 01425v1 Announce Type: new Abstract: Mixed-combinatorial nonlinear programming (MCNLP) problems arise in many engineering design and planning applications, e.

By Gishnu Madhu, Feng Liu, Souma Chowdhury

robotics benchmarks

arXiv AI

Jun 2

GiPL: Generative augmented iterative Pseudo-Labeling for Cross-Domain Few-Shot Object Detection

arXiv:2605. 29539v2 Announce Type: replace-cross Abstract: Vision-language foundation models have shown promising zero-shot generalization for Cross-Domain Few-Shot Object Detection (CD-FSOD).

By Jiacong Liu, Shu Luo, Yikai Qin, Yaze Zhao, Yongwei Jiang, Yixiong Zou

llms computer-vision fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

IstGPT: LLM-based Anomaly Detection for Spatial-Temporal Graph in Industrial Systems

arXiv:2606. 01691v1 Announce Type: cross Abstract: Industrial Internet systems face increasing threats from sophisticated industrial control system (ICS) attacks, resulting in critical safety incidents.

By Yuchen Zhang, Ning Xi, Pengbin Feng, Shigang Liu, Jianfeng Ma, Yulong Shen, Yanan Sun, Xiaolin Zhou

llms robotics multimodal benchmarks safety

arXiv AI

Jun 2

Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

arXiv:2605. 12969v3 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks.

By Feng Zhang, Xinhong Ma, Ziqiang Dong, Xi Leng, Jianfei Zhao, Xin Sun, Yang Yang, Guanjun Jiang

llms reinforcement-learning benchmarks

arXiv AI

Jun 2

AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

arXiv:2507. 08038v3 Announce Type: replace-cross Abstract: Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge.

By Talor Abramovich, Gal Chechik

llms agents benchmarks

arXiv AI

Jun 2

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

arXiv:2601. 06199v3 Announce Type: replace-cross Abstract: Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens.

By Junseok Lee, Sangyong Lee, Chang-Jae Chun

llms multimodal benchmarks

Get the daily briefing