Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,528 stories · RSS feed

arXiv Machine Learning

Aug 4

Using Lower-Bound Representations for Trajectory Similarity Learning

arXiv:2608. 01039v1 Announce Type: cross Abstract: Trajectory similarity learning is fundamental to efficient trajectory retrieval under complex distance measures.

By Liwei Deng, Haotian Meng, Yupu Zhang, Yan Zhao, Torben Bach Pedersen, Kai Zheng, Christian S. Jensen

rag benchmarks

arXiv Machine Learning

Aug 4

Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning

arXiv:2608. 02332v1 Announce Type: new Abstract: In offline reinforcement learning (RL), the distribution shift between behavioral data and the learned policy can lead to erroneous \emph{Q}-value estimation, thereby misguiding the direction of policy optimization.

By Botao Dong, Longyang Huang, Ning Pang, Hongtian Chen

diffusion reinforcement-learning benchmarks safety

arXiv Machine Learning

Aug 4

Hybrid Quantum Neural Networks: Theory, Implementations, and Applications

arXiv:2608. 01194v1 Announce Type: cross Abstract: Artificial intelligence has been transformed by deep neural networks, yet the search for new learning architectures continues.

By L\'eo Monbroussou, Maniraman Periyasamy, Viacheslav Kuzmin, Pavel Sekatski, Viktoria Patapovich, Asel Sagingalieva, Alexey Melnikov

benchmarks

arXiv Machine Learning

Aug 4

New York Smells: A Large Multimodal Dataset for Olfaction

arXiv:2511. 20544v2 Announce Type: replace-cross Abstract: While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines.

By Ege Ozguroglu, Junbang Liang, Ruoshi Liu, Mia Chiquier, Michael DeTienne, Wesley Wei Qian, Alexandra Horowitz, Andrew Owens, Carl Vondrick

multimodal benchmarks

arXiv Machine Learning

Aug 4

A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs

arXiv:2603. 07475v4 Announce Type: replace-cross Abstract: Autoregressive (AR) language models build representations incrementally via left-to-right prediction, while diffusion language models (dLLMs) are trained through full-sequence denoising.

By Raghavv Goel, Risheek Garrepalli, Sudhanshu Agrawal, Chris Lott, Mingu Lee, Fatih Porikli

llms diffusion benchmarks safety

arXiv Machine Learning

Aug 4

DynamicManip: Enabling Dynamic Manipulation from a Single Static Demonstration

arXiv:2608. 01452v1 Announce Type: cross Abstract: Dynamic manipulation is a critical capability for robots operating in complex and dynamic environments, where robots must interact with objects that are moving or require rapid adjustments.

By Haoran Liao, Pengyue Wang, Shuoyu Chen, Kehan Cheng, Xuhang Chen, Yuhao Lin, Mu Lin, Zhizhao Liang, Xiaoyi Fan, Chengyi Xing, Dan Niu, Yi-Lin Wei, Wei-Shi Zheng

robotics benchmarks

arXiv Machine Learning

Aug 4

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.

By Leyan Xue, Feng Xiong, Mingjun Ma, Changqing Zhang

llms efficiency multimodal benchmarks

arXiv Machine Learning

Aug 4

GAPSL: A Gradient-Aligned Parallel Split Learning over Data-Heterogeneous Edge Computing Systems

arXiv:2603. 18540v2 Announce Type: replace Abstract: The increasing complexity of neural networks poses significant challenges for democratizing federated learning (FL) on resource-constrained edge devices.

By Zheng Lin, Ons Aouedi, Zihan Fang, Wei Ni, Yue Gao, Symeon Chatzinotas, Xianhao Chen

benchmarks safety

arXiv Machine Learning

Aug 4

Statistical comparisons of time-series feature sets on classification tasks

arXiv:2608. 01586v1 Announce Type: cross Abstract: In recent years, numerous open-source software libraries have been developed for computing sets of features from univariate time series.

By Trent Henderson, Ben D. Fulcher

benchmarks

arXiv Machine Learning

Aug 4

Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer

arXiv:2608. 01585v1 Announce Type: cross Abstract: Language model benchmarking is a difficult task.

By Tyler Ashoff, Jordan Rodu

llms rag multimodal benchmarks safety

arXiv Machine Learning

Aug 4

ReasonCast: Towards Explainable Time Series Forecasting with Reasoning

arXiv:2608. 01875v1 Announce Type: cross Abstract: Most time series (TS) models are specialized for a single task, either understanding (i.

By Seunghan Lee, Jun Seo, Jaehoon Lee, Junhyeok Kang, Sangjun Han, Sungdong Yoo, Minjae Kim, Tae Yoon Lim, Dongwan Kang, Hwanil Choi, Soonyoung Lee, Wonbin Ahn

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 4

Optimizing Minimax Regret in Uncertain MDPs with Small Sets of Policies

arXiv:2608. 02509v1 Announce Type: cross Abstract: Sequential decision-making in real-world applications often involves uncertainty about the environment's model.

By Sterre Lutz, Dani\"el Vos, Matthijs T. J. Spaan, Anna Lukina

reinforcement-learning benchmarks safety

arXiv Machine Learning

Aug 4

Just on Time: Token-Level Early Stopping for Diffusion Language Models

arXiv:2602. 11133v2 Announce Type: replace Abstract: Diffusion language models generate text through iterative refinement, a process that is often computationally inefficient because many tokens reach stability long before the final denoising step.

By Zakhar Kohut, Severyn Shykula, Mykola Vysotskyi, Serhii Dmytryshyn, Dmytro Khamula, Michal Zakrzewski, Damian Rynczak, Jacek Ma{\l}ecki, Taras Rumezhak, Volodymyr Karpiv

llms diffusion nlp fine-tuning benchmarks

arXiv Machine Learning

Aug 4

The Push-Forward Transform for Continuous and Robust Comparison of Dynamic Shapes

arXiv:2608. 02306v1 Announce Type: cross Abstract: We introduce a mathematical framework for shape comparison based on mapping functions from the shape domain to a common reference domain.

By Roua Rouatbi, Juan-Esteban Suarez Cardona, Ivo F. Sbalzarini

benchmarks

arXiv Machine Learning

Aug 4

CAPMix: Robust KPI Anomaly Detection for AIOps in Noisy and Dynamic Environments

arXiv:2509. 06419v2 Announce Type: replace Abstract: Time-series anomaly detection is crucial in AIOps for maintaining large-scale service reliability.

By Xudong Mou, Rui Wang, Tiejun Wang, Zexin Wu, Fangda Guo, Jie Sun, Shiru Chen, Penghao Zhang, Tiezi Zhang, Tianyu Wo, Hao Peng, Chunming Hu, Xudong Liu, Renyu Yang

benchmarks

arXiv Machine Learning

Aug 4

EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

arXiv:2608. 01142v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning, but standard LoRA produces a single deterministic model and does not directly support predictive uncertainty estimation.

By Srinivas Anumasa, Dianbo Liu

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 4

Eigenvalues as a Metric for Memory Dynamics in Sequence Models

arXiv:2510. 09379v2 Announce Type: replace Abstract: While softmax attention drives state-of-the-art performance in sequence modeling, its quadratic complexity motivates linear alternatives such as state space models (SSMs).

By Rahel Rickenbach, Jelena Trisovic, Alexandre Didier, Jerome Sieber, Melanie N. Zeilinger

benchmarks

arXiv Machine Learning

Aug 4

LakeMLB: Data Lake Machine Learning Benchmark

arXiv:2602. 10441v2 Announce Type: replace Abstract: Data lakes have become a fundamental platform for large-scale machine learning by enabling flexible management of heterogeneous data.

By Feiyu Pan, Tianbin Zhang, Aoqian Zhang, Yu Sun, Zheng Wang, Lixing Chen, Li Pan, Jianhua Li

benchmarks

arXiv Machine Learning

Aug 4

Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs

arXiv:2608. 01473v1 Announce Type: cross Abstract: Multimodal large language models (MLLM) for surgical scene understanding typically inject hundreds of dense visual tokens into a language model, leading to costly inference and limited spatial traceability for generated answers.

By Guiqiu Liao, Matjaz Jogan, Daniel A. Hashimoto

llms nlp multimodal benchmarks safety

arXiv Machine Learning

Aug 4

Adaptive Reconstruction of Bosonic Quantum States

arXiv:2608. 02049v1 Announce Type: cross Abstract: Bosonic quantum systems provide a hardware-efficient platform for quantum information processing but remain challenging to characterise due to their large Hilbert space and the high measurement cost of state tomography.

By Vasilisa Usova, Phila Rembold, Ian Yang, Marco Rossignolo, Simone Montangero, Samuele Tosatto, Gerhard Kirchmair

agents benchmarks

Get the daily briefing