Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,071 stories · RSS feed

arXiv Machine Learning

Jun 18

Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

arXiv:2606. 18367v1 Announce Type: new Abstract: Standard benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating regimes.

By Yingshuo Wang, Xian Sun, Lingdong Kong, Wei Gao, Yanhang Li, Zhichao Fan, Zexin Zhuang

benchmarks

arXiv AI

Jun 18

TransitNet: A Compact Attention-Augmented Deep Learning Framework for Low-SNR Transit Blind Searches

arXiv:2606. 18932v1 Announce Type: cross Abstract: Motivated by the observational incompleteness of intermediate-to-long-period Earth-size planets, we present TransitNet, a compact attention-augmented deep-learning framework for low-SNR transit blind searches.

By Xingchen Yan, Jian Ge, Qingtian Liu, Kevin Willis, Quanquan Hu, Jiapeng Zhu

benchmarks

arXiv AI

Jun 18

scGTN: Deep Siamese Graph Transformer Network for Single-cell RNA Sequencing Clustering

arXiv:2606. 18672v1 Announce Type: cross Abstract: Single-cell RNA sequencing (scRNA-seq) serves a pivotal role in characterizing gene expression at the cellular level, enabling the identification of cell types and advancing the understanding of cellular heterogeneity.

By Jinke Wu, Yifan Wang, Siyu Yi, Caiyang Yu, Ziyue Qiao, Nan Yin, Jiancheng Lv, Wei Ju

llms efficiency benchmarks

arXiv AI

Jun 18

InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement

arXiv:2601. 14968v2 Announce Type: replace-cross Abstract: Most existing time series classification methods adopt a discriminative paradigm that maps input sequences directly to one-hot encoded class labels.

By Mingyue Cheng, Xiaoyu Tao, Huajian Zhang, Qi Liu, Zhiding Liu, Yucong Luo, Yiheng Chen, Enhong Chen

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 18

A finite-element-inspired bipartite graph learned simulator for manufacturability assessment in large-deformation sheet forming

arXiv:2605. 22845v2 Announce Type: replace-cross Abstract: Explicit dynamic finite element (FE) simulations are widely used for large deformation engineering analysis, but repeated simulations remain costly during design space exploration and optimisation.

By Yingxue Zhao, Haoran Li, Haosu Zhou, Tobias Pfaff, Nan Li

rag benchmarks

arXiv AI

Jun 18

A Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images from GPT-Image-2

arXiv:2606. 19259v1 Announce Type: cross Abstract: Text-rich images often contain privacy-sensitive, transactional, or decision-relevant information.

By Yijin Wang, Shuyi Wang, Wenhan Zhang, Yuqi Ouyang

llms diffusion multimodal benchmarks safety

arXiv AI

Jun 18

SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval

arXiv:2606. 18801v1 Announce Type: cross Abstract: With the rapid expansion of massive multilingual corpora, Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access.

By Youngjoon Jang, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim

rag benchmarks safety

arXiv AI

Jun 18

G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment

arXiv:2606. 18989v1 Announce Type: cross Abstract: Idioms are difficult to transfer across languages due to their non-compositionality and weak surface-form grounding, making literal mappings unreliable.

By Fengying Ye, Yanming Sun, Runzhe Zhan, Zheqi Zhang, Lidia S. Chao, Derek F. Wong

llms rag benchmarks safety

arXiv AI

Jun 18

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning

arXiv:2606. 18831v1 Announce Type: cross Abstract: Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories.

By Xiaoyue Xu, Sikui Zhang, Xiaorong Wang, Xu Han, Chaojun Xiao

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 18

Test-Time Adaptation in Optical Coherence Tomography Using Trajectory-Aligned Time-Independent Flow

arXiv:2606. 18876v1 Announce Type: cross Abstract: Optical coherence tomography (OCT) is essential in ophthalmology, but inconsistent image quality especially in low-cost devices hinders automated analysis.

By Veit Hucke, Thomas Pinetz, Gregor Reiter, Ursula Schmidt-Erfurth, Hrvoje Bogunovi\'c

benchmarks

arXiv Machine Learning

Jun 18

Formalizing and Mitigating Structural Distortion in LLM Attention for Graph Reasoning

arXiv:2606. 15633v2 Announce Type: replace Abstract: Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs).

By Donald Loveland, Puja Trivedi, Ari Weinstein, Edward W Huang, Danai Koutra

llms rag benchmarks

arXiv Machine Learning

Jun 18

Task-Restricted Symmetries in Recurrent Weight Space

arXiv:2606. 18457v1 Announce Type: new Abstract: Recurrent networks can contain substantial functional redundancy in weight space: changing a recurrent matrix may leave the input-output rollout nearly unchanged on a task distribution, while similar-scale changes can destroy the same behavior.

By Simon Dr\"ager

agents benchmarks

arXiv AI

Jun 18

Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey

arXiv:2602. 15851v2 Announce Type: replace-cross Abstract: Applications of narrative theories using large language models (LLMs) deliver promising methods in automatic story generation and understanding tasks.

By David Y. Liu, Aditya Joshi, Paul Dawson

llms nlp benchmarks

arXiv AI

Jun 18

EffiNav: Fusing Depth and Vision-Language for Efficient Object Goal Navigation

arXiv:2606. 18634v1 Announce Type: cross Abstract: To locate a target object while exploring the unknown environment is a fundamental capability for autonomous agents, with applications ranging from search-and-rescue to field robots.

By Zecheng Yin, Benedict Jun Ma

agents robotics multimodal benchmarks

arXiv Machine Learning

Jun 18

Acceleration of an algebraic multigrid pressure solver using graph neural networks

arXiv:2606. 19251v1 Announce Type: cross Abstract: Solving the pressure-Poisson equation remains the primary computational bottleneck in incompressible unstructured flow solvers primarily due to the inherent sensitivity of traditional linear solvers to mesh irregularities.

By Eric Chill\'on, Artur K. Lidtke, Nguyen Anh Khoa Doan, Bernat Font

benchmarks

arXiv Machine Learning

Jun 18

Lifecycle-Aware Dynamic Analysis for Secure ML Model Execution

arXiv:2606. 19023v1 Announce Type: cross Abstract: The growing reliance on pre-trained Machine Learning (ML) models has introduced new attack surfaces.

By Gabriele Digregorio, Marco Di Gennaro, Francesco Pastore, Stefano Zanero, Stefano Longari, Michele Carminati

benchmarks

arXiv AI

Jun 18

Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging

arXiv:2606. 18521v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT) in eliciting reasoning intelligence and resisting catastrophic forgetting.

By Chenrui Wu, Zexi Li, Jiajun Bu, Jiangchuan Liu, Haishuai Wang

reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 18

InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search

arXiv:2606. 18676v1 Announce Type: new Abstract: Training-free neural architecture search promises efficient discovery of high-performance networks without costly training.

By Qinqin Zhou, Fuhai Chen, Jipeng Wu, Zhiwei Chen, Zhikai Hu, Weiwei Cai

benchmarks

arXiv AI

Jun 18

ThinkDeception: A Progressive Reinforcement Learning Framework for Interpretable Multimodal Deception Detection

arXiv:2606. 18988v1 Announce Type: new Abstract: Multimodal deception detection is critical for identifying fraudulent intentions, yet existing approaches predominantly rely on end to end black--box paradigms.

By Jinhao Song, Shan Liang, Yiqun Yue, Zhuhuayang Zhang, Tianqi Gao

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 18

Do Neural Networks Lose Plasticity in a Gradually Changing World?

arXiv:2602. 09234v2 Announce Type: replace-cross Abstract: Continual learning has become a trending topic in machine learning.

By Tianhui Liu, Lili Mou

benchmarks

Get the daily briefing