Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,805 stories · RSS feed

arXiv AI

Jun 2

Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight

arXiv:2606. 00424v1 Announce Type: new Abstract: As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight.

By Can Jin, Jiakang Li, Rui Wu, Eddy Zhang, Dimitris N. Metaxas

llms efficiency benchmarks safety

arXiv Machine Learning

Jun 2

Modeling Spectral Energy Shifts in Spatio-Temporal Graph Anomaly Detection

arXiv:2606. 00304v1 Announce Type: new Abstract: Graph anomaly detection methods aim to distinguish anomalous nodes.

By Yilin Liu, Hongchao Zhang, Taylor T. Johnson, Ahmad F. Taha, Meiyi Ma

benchmarks

arXiv AI

Jun 2

CARES: Context-Aware Resolution Selector for VLMs

arXiv:2510. 19496v3 Announce Type: replace-cross Abstract: Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks.

By Moshe Kimhi, Nimrod Shabtay, Raja Giryes, Chaim Baskin, Eli Schwartz

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Towards Sparse Video Understanding and Reasoning

arXiv:2602. 13602v2 Announce Type: replace-cross Abstract: We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA).

By Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

llms agents nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

arXiv:2605. 23500v2 Announce Type: replace-cross Abstract: Segmentation is a fundamental task in computer vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis.

By Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms agents computer-vision reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 2

From Moments to Models: Graphon-Mixture Learning for Mixup and Contrastive Learning

arXiv:2510. 03690v4 Announce Type: replace Abstract: Real-world graph datasets often arise from mixtures of populations, where graphs are generated by multiple distinct underlying distributions.

By Ali Azizpour, Reza Ramezanpour, Santiago Segarra

benchmarks

arXiv AI

Jun 2

OPD+: Rethinking the Advantage Design for On-Policy Distillation

arXiv:2606. 01039v1 Announce Type: cross Abstract: On-policy distillation (OPD) is a widely used technique to transfer capabilities from capable teacher language models to the base student models, and can be formulated in a reinforcement learning style objective using student generated rollouts.

By Hanyang Zhao, Haoxian Chen, Han Lin, Genta Indra Winata, David Yao, Wenpin Tang

llms reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 2

Flow-Based Density Ratio Estimation for Intractable Distributions with Applications in Genomics

arXiv:2602. 24201v2 Announce Type: replace Abstract: Estimating density ratios between pairs of intractable data distributions is a core problem in probabilistic modeling, enabling principled comparisons of sample likelihoods under different data-generating processes across conditions.

By Egor Antipov, Alessandro Palma, Lorenzo Consoli, Stephan G\"unnemann, Andrea Dittadi, Fabian J. Theis

diffusion benchmarks

arXiv Machine Learning

Jun 2

TLG: Temporal-Logic Grounding for Video Question Answering via Source-Annotation Reconstruction and Category-Targeted Reasoning

arXiv:2606. 01591v1 Announce Type: cross Abstract: The TimeLogic Challenge evaluates formal temporal-logic reasoning over video - 16 operators (before, after, until, since, always, co-occur, ordering, ...

By Ali Alavi

llms nlp multimodal benchmarks

arXiv Machine Learning

Jun 2

Chebyshev Policies and the Mountain Car Problem: Reinforcement Learning for Low-Dimensional Control Tasks

arXiv:2605. 22305v2 Announce Type: replace Abstract: We analytically solve the Mountain Car problem, a canonical benchmark in RL, and derive an optimal control solution, closing a gap after 36 years.

By Stefan Huber, Hannes Unger, Georg Sch\"afer, Jakob Rehrl

agents reinforcement-learning benchmarks

arXiv AI

Jun 2

On Effectiveness and Efficiency of Agentic Tool-calling and RL Training

arXiv:2606. 00135v1 Announce Type: cross Abstract: Tool-calling is a central component of modern large language model (LLM) agents, equipping them with skills beyond their parametric knowledge.

By Tong Liu, Cheng Qian, Matej Cief, Yuan He, Daniele Dan, Nikolaos Aletras, Gabriella Kazai

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

ChWDTA: Channel-wise Wavelet-Domain Transformer Attention and Entropy Modeling for Learned Image Compression

arXiv:2606. 00111v1 Announce Type: cross Abstract: State-of-the-art learned image compression (LIC) schemes are increasingly based on hybrid CNN-transformer architectures.

By Haisheng Fu, Runyu Yang, Feng Ding, Siyu Zhu, Jie Liang, Xiaoxiao Li, Zhenman Fang, Jingning Han

llms nlp benchmarks

arXiv AI

Jun 2

Flow-Based Generative Modeling for Optimizing Sampling Policies in Compressed Sensing Applications

arXiv:2606. 00078v1 Announce Type: cross Abstract: Numerous modern applications in signal processing and medical imaging necessitate acquiring high-dimensional signals under tight resource constraints.

By Roman Pavelkin, Luis A. Zavala-Mondragon, Christiaan G. A. Viviers, Fons van der Sommen

diffusion computer-vision benchmarks

arXiv AI

Jun 2

Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Training

arXiv:2601. 23220v2 Announce Type: replace-cross Abstract: Despite recent Multimodal Large Language Models (MLLMs)' linguistic prowess in medical diagnosis, we find even state-of-the-art MLLMs suffer from a critical perceptual deficit: geometric blindness.

By Anglin Liu, Ruichao Chen, Yi Lu, Hongxia Xu, Jintai Chen

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

MMSkills: Towards Multimodal Skills for General Visual Agents

arXiv:2605. 13527v3 Announce Type: replace Abstract: Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines.

By Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu

agents multimodal benchmarks

arXiv AI

Jun 2

Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures

arXiv:2505. 24069v4 Announce Type: replace-cross Abstract: Large language models (LLMs) are deployed on increasingly complex tasks that require multi-step decision-making.

By Yu He, Yingxi Li, Colin White, Ellen Vitercik

llms benchmarks

arXiv Machine Learning

Jun 2

Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning

arXiv:2606. 01128v1 Announce Type: new Abstract: Communication overhead is a crucial bottleneck in scalable distributed learning.

By Tehila Dahan, Bassel Hamoud, Roie Reshef, Martin Jaggi, Kfir Y. Levy

benchmarks

arXiv AI

Jun 2

Topological Ignorability for Structural Causal Effects Beyond Means

arXiv:2606. 01184v1 Announce Type: cross Abstract: Many interventions alter the structure of an outcome distribution rather than its mean: they can split a population into disconnected regimes, create loops or holes, generate branches, or reorganize an outcome cloud while leaving the average response nearly unchanged.

By Usef Faghihi

benchmarks

arXiv AI

Jun 2

AGENTCL: Toward Rigorous Evaluation of Continual Learning in Language Agents

arXiv:2606. 02461v1 Announce Type: new Abstract: Language agents spend substantial inference time solving individual tasks, yet the experience acquired in one episode is often underutilized in future episodes.

By Yiheng Shu, Bernal Jim\'enez Guti\'errez, Saisri Padmaja Jonnalagedda, Yuguang Yao, Huan Sun, Yu Su

agents benchmarks

arXiv AI

Jun 2

PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Vision-Language Models

arXiv:2602. 00415v2 Announce Type: replace Abstract: Memory is not merely a storage mechanism for intelligent systems, but a structure for organizing evidence and constraining belief.

By Zhisheng Chen, Tingyu Wu, Zijie Zhou, Zhengwei Xie, Jinhan Li, Ziyan Weng, Liang Lin, Jingwei Song, Zikai Xiao, Yingwei Zhang

llms multimodal benchmarks

Get the daily briefing