Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,288 stories · RSS feed

arXiv AI

Jun 4

Supportive Token Revealing for Fast Diffusion Language Model Decoding

arXiv:2606. 04236v1 Announce Type: cross Abstract: Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off.

By Giries Abu Ayoub, Mario Barbara, Llu\'is Pastor-P\'erez, Tanja Bien, Aneesh Barthakur, Alaa Maalouf, Loay Mualem

llms diffusion benchmarks

arXiv AI

Jun 4

SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization

arXiv:2505. 11166v3 Announce Type: replace-cross Abstract: Despite advances in pretraining with extended context sizes, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives.

By Huashan Sun, Shengyi Liao, Yansen Han, Yu Bai, Yang Gao, Cheng Fu, Weizhou Shen, Fanqi Wan, Ming Yan, Ji Zhang, Fei Huang

llms benchmarks safety data-engineering

arXiv AI

Jun 4

DetectZoo: A Unified Toolkit for AI-Generated Content Detection Across Text, Audio, and Image Modalities

arXiv:2606. 04205v1 Announce Type: cross Abstract: The growing popularity and capacity of generative models have eroded the distinction between human and machine-generated content, motivating a growing body of work on detection across text, images, and audio.

By Sajad Ebrahimi, Nima Jamali, Bardia Shirsalimian, Kelly McConvey, Wentao Zhang, Jalehsadat Mahdavimoghaddam, Maksym Taranukhin, Maura Grossman, Vered Shwartz, Yuntian Deng, Ebrahim Bagheri

multimodal benchmarks

arXiv AI

Jun 4

GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval

arXiv:2603. 22121v2 Announce Type: replace-cross Abstract: Video Corpus Moment Retrieval (VCMR) aims to retrieve both the correct video and its temporal segment corresponding to a natural-language query, a task that is especially challenging for multi-verb queries where temporal action ordering is critical.

By Yunzhuo Sun, Xinyue Liu, Yanyang Li, Nanding Wu, Linlin Zong, Xianchao Zhang, Wenxin Liang

llms multimodal benchmarks

arXiv AI

Jun 4

Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty

arXiv:2606. 04342v1 Announce Type: cross Abstract: Multi-step time series forecasting (MSF) is commonly evaluated using point-wise error metrics such as mean squared error (MSE), implicitly treating the conditional mean as a sufficient target.

By Riku Green, Zahraa S. Abdallah, Telmo M Silva Filho

benchmarks

arXiv Machine Learning

Jun 4

LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding

arXiv:2606. 04302v1 Announce Type: cross Abstract: Key-value (KV) caching accelerates inference of large language models (LLMs) by reusing past computations for generated tokens.

By Haocheng Xia, Mihir Pamnani, Hanxi Fang, Supawit Chockchowwat, Yongjoo Park

llms rag efficiency benchmarks

arXiv AI

Jun 4

Rethinking Sales Lead Scoring with LLM-based Hierarchical Preference Ranking

arXiv:2606. 04387v1 Announce Type: cross Abstract: Sales lead conversion in high-stakes domains (e.

By Chenyu Zhang, Yiwen Liu, Yin Sun, Xinyuan Zhang, Yuji Cao, Junming Jiao, Juyi Qiao

llms benchmarks safety

arXiv AI

Jun 4

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning

arXiv:2606. 04442v1 Announce Type: cross Abstract: AI systems increasingly need to combine two demanding capabilities: navigating multi-session conversation history and performing deep reading comprehension within long documents.

By Qiyang Xie, Jialun Wu, Xinjie He, Su Liu, Shuai Xiao, Zhiyuan Lin, Weikai Zhou

llms rag benchmarks

arXiv Machine Learning

Jun 4

EpiFormer: Learning Antigen-Antibody Interactions for Epitope Prediction via Geometric Deep Learning

arXiv:2606. 04154v1 Announce Type: cross Abstract: Antibodies neutralize foreign antigens by binding to specific surface regions called epitopes.

By Mansoor Ahmed, Huirong Chai, Haoxin Wang, Hemanth Venkateswara, Murray Patterson

efficiency benchmarks

arXiv Machine Learning

Jun 4

veriFIRE: an Industrial Case Study in Verifying Consistency Properties for a DNN-Based Wildfire Detection System

arXiv:2606. 04121v1 Announce Type: cross Abstract: We present our ongoing work on the veriFIRE project: a collaboration between industry and academia, aimed at applying verification to increase the reliability of a real-world, safety-critical system.

By Idan Refaeli, Maya Swisa, Itay Buchnik, Alon Zada, Guy Amir, Elad Mandelbaum, Ziv Freund, Guy Katz

benchmarks safety

arXiv AI

Jun 4

Self-Evolving Deep Research via Joint Generation and Evaluation

arXiv:2606. 04507v1 Announce Type: cross Abstract: Large Language Models (LLMs) have become increasingly adopted in daily applications, with deep research standing out as a particularly important capability.

By Han Zhu, Chengkun Cai, Yuanfeng Song, Xing Chen, Sirui Han, Yike Guo

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 4

ChessMimic: Per-Rating Transformer Models for Human Move, Clock, and Outcome Prediction in Online Blitz Chess

arXiv:2606. 04473v1 Announce Type: cross Abstract: We present ChessMimic, a system of three small encoder-only transformers - for move, thinking-time, and outcome prediction - conditioned on the position, recent move history, player rating, and clock state.

By Thomas Johnson

llms benchmarks safety

arXiv AI

Jun 4

Optical-Guided Neural Collapse for SAR Few-Shot Class Incremental Learning

arXiv:2606. 04528v1 Announce Type: cross Abstract: Few-shot class-incremental learning (FSCIL) in synthetic aperture radar imagery presents unique challenges due to severe data scarcity and SAR-specific variability.

By Fan Zhang, Sijin Zheng, Fei Ma, Qiang Yin, Yongsheng Zhou, Fei Gao, Xian Sun

benchmarks

arXiv AI

Jun 4

ANN Search: Recall What Matters

arXiv:2606. 04522v1 Announce Type: cross Abstract: Approximate nearest neighbor (ANN) search has become a core primitive in information retrieval and modern machine learning tasks, from classification to retrieval-augmented generation.

By Dimitris Dimitropoulos, Nikos Mamoulis

llms rag efficiency benchmarks

arXiv AI

Jun 4

Curvature-aware dynamic precision approach for physics-informed neural networks

arXiv:2606. 04736v1 Announce Type: cross Abstract: Physics-informed neural networks (PINNs) have become a promising framework for simulating partial differential equations (PDEs) by embedding physical laws directly into neural network training.

By Yingjie Shao, Ioannis N. Athanasiadis, George van Voorn, Taniya Kapoor

rag benchmarks

arXiv AI

Jun 4

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

arXiv:2606. 04244v1 Announce Type: new Abstract: Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output, specifically when they rely on visual aids.

By Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari, Yasamin Medghalchi, Ilker Hacihaliloglu, Mesrob Ohannessian, Lele Wang, Giuseppe Carenini

llms multimodal benchmarks

arXiv Machine Learning

Jun 4

Post-Training Corrections for Improved Time-Series Forecasting

arXiv:2505. 15354v3 Announce Type: replace Abstract: Time-series forecasting is a critical task in various business domains, but it remains inherently challenging.

By Hamza Cherkaoui, Malik Tiomoko, Giuseppe Paolo, Zhang Yili, Yu Meng, Zhang Keli, Hafiz Tiomoko Ali

benchmarks

arXiv Machine Learning

Jun 4

LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction

arXiv:2511. 12581v2 Announce Type: replace Abstract: Static IR drop analysis is a fundamental and critical task in the field of chip design.

By Kai Ma, Zhen Wang, Hongquan He, Qi Xu, Tinghuan Chen, Hao Geng

llms diffusion multimodal benchmarks

arXiv AI

Jun 4

Activation Steering of Video Generation Models via Reduced-Order Linear Optimal Control

arXiv:2606. 04775v1 Announce Type: cross Abstract: Text-to-video (T2V) models trained on large-scale web data can generate undesired content, motivating interventions that reduce harmful outputs without sacrificing visual quality.

By Jihoon Hong, Alice Chan, Qiyue Dai, Julian Skifstad, Glen Chou

diffusion fine-tuning benchmarks safety

arXiv Machine Learning

Jun 4

The Perception-Physics Paradox: Probing Scientific Alignment with TC-Bench

arXiv:2605. 24782v2 Announce Type: replace Abstract: While Vision Foundation Models (VFMs) excel at predictive tasks on satellite imagery, their performance can arise from visual correlations rather than underlying structural invariants, making even perception-based out-of-distribution accuracy a poor proxy for scientific utility.

By Dingling Yao, Andrea Polesello, Adeel Pervez, Caroline Muller, Francesco Locatello

benchmarks safety

Get the daily briefing