Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,805 stories · RSS feed

arXiv Machine Learning

Jun 2

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

arXiv:2602. 07223v2 Announce Type: replace Abstract: Long-context large language model (LLM) inference has become the norm for today's AI applications.

By Yikang Yue, Yuqi Xue, Jian Huang

llms efficiency benchmarks

arXiv AI

Jun 2

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

arXiv:2606. 01682v1 Announce Type: cross Abstract: Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths.

By Atoosa Chegini, Soheil Feizi

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 2

Mean Flow Policy Optimization

arXiv:2604. 14698v2 Announce Type: replace Abstract: Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL).

By Xiaoyi Dong, Xi Sheryl Zhang, Jian Cheng

diffusion reinforcement-learning benchmarks

arXiv AI

Jun 2

Boosting Multimodal Federated Learning via Chained Modality Optimization

arXiv:2606. 01856v1 Announce Type: cross Abstract: Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability.

By Zixin Zhang, Fan Qi, Shuai Li, Xiaoshan Yang, Changsheng Xu

multimodal benchmarks safety

arXiv AI

Jun 2

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

arXiv:2606. 01993v1 Announce Type: cross Abstract: Abundant procedural knowledge on the Web holds great potential for helping agents solve long-horizon tasks.

By Xinyu Che, Junqi Xiong, Yunfei Ge, Xinping Lei, Shihao Li, Hang Yan, Han Li, Yuanxing Zhang, Zhiqi Bai, Jinhua Hao, Ming Sun, Han Li, Jiaheng Liu

llms agents multimodal benchmarks

arXiv AI

Jun 2

The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue

arXiv:2606. 01901v1 Announce Type: cross Abstract: We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image generator across multiple turns, making accumulated common ground directly observable as a rendered image.

By Sherzod Hakimov, Mattia D'Agostini, Ivan Samodelkin, David Schlangen

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation

arXiv:2605. 29977v2 Announce Type: replace-cross Abstract: High-fidelity ECG interpretation is increasingly reliant on massive foundation models, yet their deployment in clinical edge-care remains hindered by extreme computational demands.

By Dang Nguyen Hong, Nhi Ngoc-Yen Nguyen, Huy-Hieu Pham

efficiency benchmarks safety

arXiv AI

Jun 2

TCAR-Gen: Temporal Graph Retrieval with Evidence Fusion for Knowledge-Grounded Generation

arXiv:2606. 00029v1 Announce Type: cross Abstract: Retrieval-augmented generation systems struggle with temporal reasoning and evidence fusion when answering complex questions over historical criminal case narratives.

By Sidra Nasir, Muhammad Noman Zahid, Rizwan Ahmed Khan

llms rag nlp benchmarks

arXiv AI

Jun 2

Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing

arXiv:2606. 02218v1 Announce Type: cross Abstract: Synchronous reinforcement learning methods such as Group Relative Policy Optimization (GRPO) provide stable and reproducible on-policy training, but they are highly vulnerable to stragglers, a single unusually long rollout can delay reward computation and parameter updates for the entire group.

By Azal Ahmad Khan, Ammar Ahmed, Zeshan Fayyaz, Sheng Di, Mingyi Hong, Ali Anwar

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning

arXiv:2605. 03403v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has recently shown strong performance in post-training large language models and vision-language models.

By Yujun Li, Hongyuan Zhang, Yuan Yuan

llms reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Efficient Diffusion Models under Nonconvex Equality and Inequality constraints via Landing

arXiv:2604. 17838v2 Announce Type: replace Abstract: Generative modeling within constrained sets is essential for scientific and engineering applications involving physical, geometric, or safety requirements (e.

By Kijung Jeon, Michael Muehlebach, Molei Tao

diffusion robotics benchmarks safety

arXiv Machine Learning

Jun 2

Meta-Black-Box Optimization with Ensemble Surrogate Modeling for Robustness-Accuracy Trade-off within SAEA

arXiv:2606. 00862v1 Announce Type: cross Abstract: Surrogate-assisted evolutionary algorithms (SAEAs) have been widely used for expensive black-box optimization problems.

By Xiao Jin, Yongxiong Wang, Haobo Liu, Yudong Du, Yukun Du

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting

arXiv:2502. 18834v3 Announce Type: replace-cross Abstract: Financial time series (FinTS) record the behavior of human-brain-augmented decision-making, capturing valuable historical information that can be leveraged for profitable investment strategies.

By Yifan Hu, Yuante Li, Peiyuan Liu, Yuxia Zhu, Naiqi Li, Tao Dai, Shu-tao Xia, Dawei Cheng, Changjun Jiang

benchmarks data-engineering

arXiv AI

Jun 2

v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound

arXiv:2509. 25773v3 Announce Type: replace-cross Abstract: AI models capable of comprehending humor hold real-world promise -- for example, enhancing engagement in human-machine interactions.

By Zhengpeng Shi, Yanpeng Zhao, Jianqun Zhou, Yuxuan Wang, Qinrong Cui, Wei Bi, Songchun Zhu, Bo Zhao, Zilong Zheng

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Rethinking Amortized Neural Representations for High-Resolution Terrain Elevation Data

arXiv:2606. 00404v1 Announce Type: cross Abstract: Implicit neural representations (INRs) model a signal as a continuous coordinate-to-value function.

By Haoan Feng, Xin Xu, Leila De Floriani

efficiency benchmarks

arXiv AI

Jun 2

CityTrajBench: A Unified Benchmark for City-Scale Vehicle Trajectory Generation

arXiv:2606. 02287v1 Announce Type: cross Abstract: Urban trajectory generation is a fundamental task for transportation simulation, urban planning, and mobility analytics.

By Shibo Zhu, Xiaodan Shi, Dayin Chen, Yuntian Chen, Haoran Zhang, Tianhao Wu, Jinyue Yan

diffusion benchmarks

arXiv AI

Jun 2

From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

arXiv:2606. 00357v1 Announce Type: new Abstract: Training strong large language models (LLMs) requires high-quality supervision, which is often scarce.

By Qi Sun, Siyue Zhang, Yulin Chen, Yuxiang Xue, Ru Peng, Chen Zhao

llms agents fine-tuning benchmarks safety

arXiv AI

Jun 2

Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction

arXiv:2602. 08585v2 Announce Type: replace-cross Abstract: Given the quadratic complexity of attention, KV cache eviction is vital to accelerate model inference.

By Ziyao Tang, Pengkun Jiao, Xinhang Chen, Wei Liu, Shiyong Li, Jingjing Chen

efficiency benchmarks

arXiv AI

Jun 2

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

arXiv:2606. 02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.

By Seojeong Park, Jiho Choi, Junyong Kang, Seonho Lee, Jaeyo Shin, Hyunjung Shim

llms reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 2

VMDNet: Temporal Leakage-Free Variational Mode Decomposition for Electricity Demand Forecasting

arXiv:2509. 15394v3 Announce Type: replace Abstract: Accurate electricity demand forecasting is challenging due to the strong multi-periodicity of real-world demand series, which makes effective modeling of recurrent temporal patterns crucial.

By Weibin Feng, Ran Tao, John Cartlidge, Jin Zheng

rag benchmarks

Get the daily briefing