Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning

Oct 15

Decoupled Marginal Sharpening for Training-Free Inference-Time Scaling

arXiv:2610.20483v1 Announce Type: new Abstract: We study how to improve LLM reasoning at inference time without updating model weights or using external reward models. Recent reward-free power-sampli...

By Weixin Wang, Wei Deng, Anderson Schneider, Yuriy Nevmyvaka, Pan Xu, Andrew Bennett

llms benchmarks

arXiv Machine Learning

Oct 15

Edgewise Envelopes Between Balanced Forman and Ollivier-Ricci Curvature

arXiv:2603.13535v5 Announce Type: cross Abstract: Evaluating Ollivier-Ricci (OR) curvature on large-scale graphs is computationally prohibitive due to the necessity of solving an optimal transport pr...

By Giorgio Micaletto, Tebe Nigrelli

benchmarks

arXiv Machine Learning

Oct 15

Optimization for One, Feasibility for Most: Decoupling Optimization from Feasibility in Chance-Constrained Programs

arXiv:2610.19027v1 Announce Type: cross Abstract: Chance-constrained stochastic programs (CCSPs) require solutions that satisfy uncertain constraints with high probability. Standard approaches either...

By Justin Dumouchelle, Shunyu Yao, Beste Basciftci, Thiago Serra

benchmarks

arXiv Machine Learning

Oct 15

Target-Token Asymmetry Can Masquerade As Depth Specialization

arXiv:2610.19344v1 Announce Type: cross Abstract: Layer-ablation studies often perturb early or late layers of a language model, compare the damage to prediction on two corpora, and read a larger ear...

By Danlin Li, Miao Li, Xu Han, Jiecheng Lu

llms nlp benchmarks safety

arXiv Machine Learning

Oct 15

Learning to Select Features with Manifold Deep Unfolding PCA

arXiv:2610.19436v1 Announce Type: cross Abstract: Unsupervised feature selection (UFS) is of great significance in image processing and pattern recognition. Nevertheless, existing model-based methods...

By Jingjing Liu, Pei Tai, Xianchao Xiu, Wanquan Liu

benchmarks

arXiv Machine Learning

Oct 15

Articulation Upper Bounds for Language Models

arXiv:2610.19721v1 Announce Type: cross Abstract: Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers. What, then, are the limits of language...

By Alexander Spangher, Daniel E. Ho, Diyi Yang, Noah Goodman, Sanmi Koyejo

llms benchmarks

arXiv Machine Learning

Oct 15

Maximum-Entropy Exploration with Future State-Action Visitation Measures

arXiv:2603.18965v2 Announce Type: replace Abstract: Maximum entropy reinforcement learning motivates agents to explore states and actions to maximize the entropy of some distribution, typically by pr...

By Adrien Bolland, Gaspard Lambrechts, Damien Ernst

agents reinforcement-learning benchmarks

arXiv Machine Learning

Oct 15

From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RL

arXiv:2605.07057v2 Announce Type: replace Abstract: Online reinforcement learning (RL) relies on the Markov property for guaranteed performance, but real-world applications often lack well-defined st...

By Jiamin Xu, Jacqueline Maasch, Kyra Gan

reinforcement-learning benchmarks

arXiv Machine Learning

Oct 15

TimeES: Probabilistic and Deterministic Time Series Forecasting via Evolutionary Spectra

arXiv:2609.32384v2 Announce Type: replace Abstract: Real-world time series are inherently non-stationary, with trends, periodic patterns, and uncertainty evolving over time. While the Fourier domain...

By Weiwei Ye, Renhe Jiang, Hangchen Liu, Dongyuan Li, Yoshihide Sekimoto

efficiency benchmarks safety

arXiv Computation and Language

Oct 15

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

arXiv:2610.19166v1 Announce Type: new Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that co...

By Anubhav Jangra, Adrian de Wynter

llms rag benchmarks

arXiv Computation and Language

Oct 15

MemDuet: Learning Dual-Process Memory for Long-Context Reasoning

arXiv:2610.19928v1 Announce Type: new Abstract: Reasoning over infinitely long documents is becoming a reality with recurrent memory agents. However, increasing context length exacerbates the trade-o...

By Jiangcheng Song, Haodong Jing, Yulin Mu, Lesong Tao, Jixin Wang, Jin Cui, Tianjing Ren, Runxuan Xie, Yongqiang Ma, Nanning Zheng

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Computation and Language

Oct 15

Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

arXiv:2610.20221v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in...

By Zahraa Al Sahili, Ioannis Patras, Matthew Purver

llms benchmarks safety

arXiv Computation and Language

Oct 15

Document-as-Image Representations Fall Short for Scientific Retrieval

arXiv:2604.18508v2 Announce Type: replace-cross Abstract: Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the un...

By Ghazal Khalighinejad, Raghuveer Thirukovalluru, Alexander H. Oh, Bhuwan Dhingra

rag multimodal benchmarks

arXiv Computer Vision

Oct 15

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

arXiv:2610.19063v1 Announce Type: new Abstract: As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requi...

By Nan Zhou, Jianxiong Gao, Qiuyu Wang, Wen Wang, Jiahao Wang, Yufeng Yuan, Shuaiting Li, Hao Ouyang, Yujun Shen, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

benchmarks safety

arXiv Computer Vision

Oct 15

HyrCap: Hybrid Rank-Calibration of Action Proposals for Temporal Event Understanding

arXiv:2610.19526v1 Announce Type: new Abstract: Temporal action detection in long untrimmed videos suffers from inaccurate localization and low-quality proposals, especially with sparse annotations a...

By Rui Chen, Xingyu Chen, Pengxin Xu, Kai Chen, Junzhi Yu

llms computer-vision benchmarks

arXiv Computer Vision

Oct 15

RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation

arXiv:2610.20091v1 Announce Type: new Abstract: Video scene graph generation (VidSGG) aims to capture objects and their evolving relationships in videos. This requires accounting for both the persist...

By Jaeyeon Heo, Geuntaek Lim, Yukyung Choi

llms benchmarks safety

arXiv Computer Vision

Oct 15

PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

arXiv:2412.12877v3 Announce Type: replace Abstract: Recent Artificial Intelligence (AI)-based video editing has enabled users to edit videos through simple text prompts, significantly simplifying the...

By Samuel Teodoro, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim

benchmarks

arXiv Machine Learning

Oct 15

Njord: A Probabilistic Graph Neural Network for Ensemble Ocean Forecasting

arXiv:2605.15470v3 Announce Type: replace Abstract: Ocean dynamics are inherently chaotic, yet existing machine learning ocean models produce only deterministic forecasts. We introduce Njord, a proba...

By Daniel Holmberg, Joel Oskarsson, Erik Wikingsson, Fredrik Lindsten, Teemu Roos

benchmarks

arXiv Machine Learning

Oct 15

Parameter-Efficient Temporal Domain Generalization

arXiv:2602.11965v2 Announce Type: replace Abstract: Temporal domain generalization (TDG) learns from a sequence of observed domains to predict models for unseen future domains. Existing parameter-dyn...

By Zekun Cai, Xinyuan Song, Yiheng Yao, Xuan Song, Ryosuke Shibasaki, Liang Zhao

fine-tuning benchmarks

arXiv AI

Oct 15

LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

arXiv:2610.18843v1 Announce Type: cross Abstract: Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted,...

By Andrea Yaoyun Cui, Hanyu Wang, Xinyi Wang, Bangzheng Li, Lin Zhang, Zhenggang Tang, Shyamal Buch, Dejia Xu, Matthew Tancik

robotics multimodal benchmarks

Get the daily briefing