Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,246 stories · RSS feed

arXiv Machine Learning

Jun 30

Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervision

arXiv:2505. 12526v2 Announce Type: replace Abstract: Temporal graph networks suffer from irregular supervision in realworld dynamic graphs, as most minibatches contain few labeled events.

By Alexander Panyshev, Dmitry Vinichenko, Oleg Travkin, Roman Alferov, Alexey Zaytsev

efficiency benchmarks

arXiv Machine Learning

Jun 30

TextClusterLab: An Integrated Framework for Reliable Text Clustering Studies

arXiv:2606. 28328v1 Announce Type: cross Abstract: In recent years, text clustering has become a critical technique for applications including intent discovery, topic mining, and recommendation systems.

By Daoming Wan, Yizheng Huang, Jimmy X. Huang

llms rag benchmarks

arXiv Machine Learning

Jun 30

Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs

arXiv:2606. 28401v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have shown strong performance in visual understanding, yet they still suffer from hallucinations, generating content that is not grounded in the image.

By Yunhun Nam, Jongheon Jeong

llms computer-vision multimodal benchmarks safety

arXiv AI

Jun 30

CaveAgent: Transforming LLMs into Stateful Runtime Operators

arXiv:2601. 01569v4 Announce Type: replace Abstract: LLM-based agents are increasingly capable of complex task execution, yet current agentic systems remain constrained by text-centric paradigms that struggle with long-horizon tasks due to fragile multi-turn dependencies and context drift.

By Maohao Ran, Zhenglin Wan, Cooper Lin, Yanting Zhang, Hongyu Xin, Hongwei Fan, Yibo Xu, Beier Luo, Yaxin Zhou, Wangbo Zhao, Lijie Yang, Lang Feng, Fuchao Yang, Jingxuan Wu, Yiqiao Huang, Chendong Ma, Yusen Huang, Dailing Jiang, Jianbo Deng, Sirui Han, Yang You, Bo An, Yike Guo, Jun Song

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 30

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

arXiv:2606. 28433v1 Announce Type: new Abstract: One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings.

By Matthew Vandergrift, Esraa Elelimy, Martha White

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 30

Momentum Guidance: Plug-and-Play Guidance for Flow Models

arXiv:2602. 20360v2 Announce Type: replace Abstract: Flow-based generative methods offer a simple and effective framework for high-fidelity generation, yet pretrained flow models are rarely used in their vanilla conditional form: in image generation, samples without guidance often appear diffuse and lack fine-grained detail.

By Runlong Liao, Jian Yu, Baiyu Su, Chi Zhang, Lizhang Chen, Qiang Liu

diffusion benchmarks

arXiv Machine Learning

Jun 30

ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data

arXiv:2408. 16028v4 Announce Type: replace-cross Abstract: Supervised-learning-based vulnerability detectors often fall short due to limited labelled training data.

By Weizhou Wang, Eric Liu, Xiangyu Guo, Xiao Hu, Ilya Grishchenko, David Lie

llms benchmarks

arXiv AI

Jun 30

FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

arXiv:2603. 25144v2 Announce Type: replace-cross Abstract: Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks.

By Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang

efficiency benchmarks

arXiv AI

Jun 30

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

arXiv:2606. 29579v1 Announce Type: cross Abstract: Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters.

By Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Pu Zhao, Yanzhi Wang

llms fine-tuning benchmarks

arXiv AI

Jun 30

Managing the Human Fallback: Skill Investment Under Improving AI and Worker Mobility

arXiv:2606. 29111v1 Announce Type: new Abstract: When firms deploy autonomous AI, they must decide how much work to leave to the system and how much to keep workers engaged.

By Simrita Singh, Naireet Ghosh, Tinglong Dai

agents benchmarks

arXiv Machine Learning

Jun 30

Alternative Graph Neural Networks: Synergizing GEV Models and Deep Learning for Travel Mode Choice Modeling

arXiv:2509. 07123v2 Announce Type: replace-cross Abstract: Generalized extreme value models capture dependence among choice alternatives in discrete choice modeling, but require this dependence to be predefined, symmetric, and shared uniformly across individuals.

By Yuqi Zhou, Zhanhong Cheng, Dingyi Zhuang, Lingqian Hu, Yuheng Bu, Shenhao Wang

benchmarks

arXiv Machine Learning

Jun 30

Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries

arXiv:2602. 04940v2 Announce Type: replace Abstract: Deep learning has emerged as a transformative tool for the neural surrogate modeling of partial differential equations (PDEs), known as neural PDE solvers.

By Hang Zhou, Haixu Wu, Haonan Shangguan, Yuezhou Ma, Huikun Weng, Jianmin Wang, Mingsheng Long

llms benchmarks

arXiv AI

Jun 30

Improvement of Robot's Simultaneous Localization and Mapping Using an Effective Transformation to Achieve Linear Model

arXiv:2606. 28475v1 Announce Type: cross Abstract: Nowadays mobile robots have wide engineering applications.

By Seyed Farzad Bahreinian, Maziar Palhang, Mohammad Reza Taban, Hasan Enami Eraghi

robotics benchmarks

arXiv AI

Jun 30

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

arXiv:2606. 28332v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains poorly understood.

By Yige Li, Jun Sun, Wei Zhao, Zhe Li, Yutao Wu, Hanxun Huang, Xiang Zheng, Xingjun Ma

llms fine-tuning benchmarks safety

arXiv AI

Jun 30

HyBIRD: Hyperbolic Bridge Retrieval and Diagnosis for Methodology Inspiration Retrieval

arXiv:2606. 28336v1 Announce Type: cross Abstract: Methodology Inspiration Retrieval (MIR) asks a system to retrieve prior papers whose methods can inspire a new research proposal.

By Yang Yang, Boyun Xu, Hao Fu, Jindong Li, Zining Zhong, Bowen Tian, Jiemin Wu, Menglin Yang, Yutao Yue

llms benchmarks

arXiv AI

Jun 30

Two-Stage Prompt Optimization for Few-Shot Relation Extraction: From Reasoning-Guided Search to Gradient-Guided Refinement

arXiv:2606. 29639v1 Announce Type: cross Abstract: Automatic prompt optimization is still underexplored for episodic few-shot relation extraction with smaller language models.

By Aunabil Chakma, Mihai Surdeanu, Eduardo Blanco

llms benchmarks

arXiv AI

Jun 30

Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation

arXiv:2606. 28593v1 Announce Type: cross Abstract: While recent vision-language models (VLMs) have achieved significant improvements on static visual-to-code tasks such as generating code for webpages, charts, or SVGs, it remains unclear whether they can recover temporal dynamics when motion is present.

By Anya Ji, Abhijith Varma Mudunuri, David M. Chan, Alane Suhr

llms fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks

arXiv:2606. 29082v1 Announce Type: cross Abstract: Would experience designing faster GPU kernels also help close in on a long-standing open mathematical conjecture?

By Young-Jun Lee, Seungone Kim, Minki Kang, Alistair Cheong Liang Chuen, Zerui Chen, Seungho Han, Taehee Jung, Dongyeop Kang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 30

Efficient 3D Gaussian Splatting with Axis-Shared Rasterization and Order-independent Transmittance

arXiv:2506. 07069v2 Announce Type: replace-cross Abstract: 3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, combining high-quality reconstruction with efficient rendering.

By Zhican Wang, Guanghui He, Lingjun Gao, Dantong Liu, Shell Xu Hu, Chen Zhang, Zhuoran Song, Nicholas Lane, Hongxiang Fan

agents robotics benchmarks

arXiv AI

Jun 30

When Models Know When They Do Not Know: Calibration, Cascading, and Cleaning

arXiv:2601. 07965v2 Announce Type: replace Abstract: When a model knows when it does not know, many possibilities emerge.

By Chenjie Hao, Weyl Lu, Yuko Ishiwaka, Zengyi Li, Weier Wan, Yubei Chen

llms benchmarks

Get the daily briefing