Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI

Oct 27

Neural Information Causality: A Random-Access Capacity Law for Representation Learning

arXiv:2605.09316v2 Announce Type: replace-cross Abstract: How can a learned representation remain useful when the question it must answer is chosen only after encoding? Preserving information is esse...

By Jeongho Bang, Marcin Paw{\l}owski

benchmarks

arXiv AI

Oct 27

GraphDancer: Training LLMs to Explore and Reason over Graphs via Curriculum Reinforcement Learning

arXiv:2602.02518v3 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly rely on external knowledge to improve factuality, yet many knowledge sources on the Web, such as sc...

By Yuyang Bai, Zhuofeng Li, Ping Nie, Jianwen Xie, Yu Wang, Yu Zhang

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 27

Divide et Calibra: Multiclass Local Calibration via Vector Quantization

arXiv:2605.21060v2 Announce Type: replace-cross Abstract: Accurate and well-calibrated Machine Learning (ML) models are mandatory in high-stakes settings, yet effective multiclass calibration remains...

By Cesare Barbera, Lorenzo Perini, Giovanni De Toni, Andrea Passerini, Andrea Pugnana

diffusion efficiency benchmarks

arXiv AI

Oct 27

Content-Aware Adaptive Token Pruning for Video-Language Models

arXiv:2608.03112v2 Announce Type: replace-cross Abstract: Vision-language models excel at image and video understanding but suffer from high inference latency due to the need to process thousands of...

By Paribesh Regmi, Qingshuang Chen, Chi Zhang, Heba Aly, Yelin Kim, Hongda Mao

llms rag fine-tuning efficiency multimodal benchmarks

arXiv AI

Oct 27

A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Services

arXiv:2609.31877v2 Announce Type: replace-cross Abstract: Cloud-based Large language model (LLM) services create a network-level traffic side channel that can expose model, prompt, and task behavior...

By Shahrooz Pouryousef, Jesus Lopez, Saeefa Rubaiyet Nowmi, Md Mahmuduzzaman Kamol, Moinul Hossain, Muoi Tran, Mohammad Saidur Rahman

llms agents benchmarks

arXiv AI

Oct 27

Mathematical Reasoning via Intervention-Based Time-Series Causal Discovery Using LLMs as Concept Mastery Simulators

arXiv:2605.07600v2 Announce Type: replace-cross Abstract: Recent methods for improving LLM mathematical reasoning, whether through MCTS-based test-time search or causal graph-guided knowledge injecti...

By Tsuyoshi Okita

llms benchmarks

arXiv AI

Oct 27

What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

arXiv:2610.05533v2 Announce Type: replace-cross Abstract: Harness search keeps a change to the prompts, token budgets or parsers around a frozen model if the change raises a score. Such a gain can co...

By Bowen Xu, Boyu Chen

fine-tuning benchmarks

arXiv AI

Oct 27

Reasoning-Aware Multicalibrated Reliability Predictors for Language Models

arXiv:2610.26645v2 Announce Type: replace-cross Abstract: As people increasingly rely on large language models (LLMs) for information, model providers need to communicate meaningful estimates of whet...

By Ekaterina Fedorova, Alessandro Morosini, Chara Podimata

llms benchmarks safety

arXiv Machine Learning

Oct 27

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

arXiv:2610.31731v1 Announce Type: new Abstract: Multimodal sentiment analysis (MSA) aims to predict human sentiments by integrating signals from different modalities such as text, video, and audio. H...

By Pan Wang, Lipeng Ke, Huajun Ying, Pritish Mohapatra, Rohan Sarkar, Suresh Lakhani, Sankar Venkataraman, Jingtong Hu

nlp multimodal benchmarks

arXiv Machine Learning

Oct 27

PRISM: Promoting Sampling Diversity of Large Language Models with Implicit Stochastic Modulation

arXiv:2610.32101v1 Announce Type: new Abstract: Large language models (LLMs) exhibit mode collapse, where their output distribution concentrates on familiar semantic modes. When diversity disappears,...

By Qixin Xu, Zihan Zhu, Xiaojiang Liu, Bohan Lyu

llms rag reinforcement-learning benchmarks

arXiv Machine Learning

Oct 27

Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information

arXiv:2610.32175v1 Announce Type: new Abstract: Differentially Private Stochastic Gradient Descent (DPSGD) has become the de facto standard for training deep neural networks with rigorous privacy gua...

By Xincheng Xu, Thilina Ranbaduge, Thierry Rakotoarivelo, David Smith

benchmarks safety

arXiv Machine Learning

Oct 27

MS-TTRL: Breaking the Frequency Conflict in Multiple-Solution Test-Time Reinforcement Learning

arXiv:2610.32234v1 Announce Type: new Abstract: Scientific, engineering, and mathematical reasoning tasks can admit multiple valid solutions, while exhaustive annotation of the full solution set is o...

By Zhixia Zhang, Binqian Xu, Huaiyang Wang, Yikun Ban, Dianbo Liu

reinforcement-learning benchmarks

arXiv Machine Learning

Oct 27

When to Trust Memory: Retrieval-Guided Probabilistic Spatiotemporal Forecasting under Distribution Shift

arXiv:2610.32329v1 Announce Type: new Abstract: Spatiotemporal forecasting plays an important role in real-world applications, where probabilistic approaches provide uncertainty estimates for decisio...

By Haochen Lv, Jianhao Zhang, Zhichen Lei, Yang Jiang, Xiaowei Mao, Shengnan Guo, Youfang Lin, Huaiyu Wan

benchmarks

arXiv Machine Learning

Oct 27

DT-JEPA: Learning to Predict Future Representations for Decision-Making

arXiv:2610.32657v1 Announce Type: new Abstract: Decision Transformer (DT) has demonstrated strong performance in various offline reinforcement learning (RL) tasks by conditioning action generation on...

By Changxu Ni, Yiyang Ni, Jun Li, Feiran You, Hongyang Du, Haitao Zhao, Shi Jin

llms rag diffusion reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Oct 27

U-NeXt: General-purpose event detection for neonatal EEG

arXiv:2610.32811v1 Announce Type: new Abstract: We developed U-NeXt, a general framework for neonatal electroencephalography (EEG) event detection based on semantic segmentation. Its convolutional en...

By Robert Hogan, Sean R. Mathieson, Aurel Luca, Soraia Ventura, Sean Griffin, Geraldine B. Boylan, John M. O'Toole

computer-vision benchmarks

arXiv Machine Learning

Oct 27

Sustainability of TSFM Deployment and Pretraining: From Carbon Footprint Analysis to Mitigation via Pruning

arXiv:2610.32859v1 Announce Type: new Abstract: Time Series Foundation Models (TSFMs) offer strong zero-shot forecasting capabilities, but their high pretraining cost raises a practical sustainabilit...

By C\'ecile Luc, Nathana\"el Lemonnier, Ievgen Redko

efficiency benchmarks

arXiv Machine Learning

Oct 27

EMG Tokenization Preserves Movement Information at a Fraction of the Bitrate

arXiv:2610.32674v1 Announce Type: cross Abstract: Wireless electromyography (EMG) interfaces must balance electrode coverage against transmission bandwidth. Here, we show that compact EMG tokens subs...

By Raul C. S\^impetru, Amin Olamazadeh, Devon Rohlf, Vlad Cnejevici, Alessandro Del Vecchio

nlp efficiency benchmarks

arXiv Machine Learning

Oct 27

Knowledge Distillation across Item and Code Spaces for Parallel Generative Recommendation

arXiv:2610.32943v1 Announce Type: cross Abstract: Generative recommendation with semantic IDs represents each item as a sequence of discrete codes derived from its semantic representation and predict...

By Yixuan Xu, Hailin Liu, Linxiang Peng, Can Yang

efficiency benchmarks

arXiv Machine Learning

Oct 27

FC-PINO: High Precision Physics-Informed Neural Operators via Fourier Continuation

arXiv:2211.15960v4 Announce Type: replace Abstract: The physics-informed neural operator (PINO) is a machine learning paradigm that has demonstrated promising results for learning solutions to partia...

By Adarsh Ganeshram, Valentin Duruisseaux, Haydn Maust, Zongyi Li, Yixuan Wang, Daniel Leibovici, Oscar Bruno, Thomas Hou, Anima Anandkumar

benchmarks

arXiv Machine Learning

Oct 27

U-STS-LLM: A Unified Spatio-Temporal Steered Large Language Model for Traffic Prediction and Imputation

arXiv:2605.11735v2 Announce Type: replace Abstract: Accurate spatio-temporal cellular traffic analysis is fundamental to the reliable operation of modern mobile networks, as it underpins load forecas...

By Yichen Zhang, Yiyang Ni, Pengcheng Xia, Jun Li, Chunguo Li

llms benchmarks safety

Get the daily briefing