Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,494 stories · RSS feed

arXiv Machine Learning

Jun 2

Gradient Preconditioning for Efficient and Reliable Reward-Guided Generation

arXiv:2602. 08646v3 Announce Type: replace Abstract: We propose a gradient preconditioning method that makes reward-guided generation with one-step generative models both efficient and reliable.

By Jisung Hwang, Minhyuk Sung

benchmarks

arXiv Machine Learning

Jun 2

Human in the Loop Adaptive Optimization for Improved Time Series Forecasting

arXiv:2505. 15354v2 Announce Type: replace Abstract: Time series forecasting models often produce systematic, predictable errors even in critical domains such as energy, finance, and healthcare.

By Malik Tiomoko, Hamza Cherkaoui, Giuseppe Paolo, Zhang Yili, Yu Meng, Zhang Keli, Hafiz Tiomoko Ali

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

Scalable Ride-Sourcing Vehicle Rebalancing with Service Accessibility Guarantee: A Constrained Mean-Field Reinforcement Learning Approach

arXiv:2503. 24183v3 Announce Type: replace Abstract: The expansion of ride-sourcing services such as Uber and Lyft has reshaped urban transportation by offering flexible, on-demand mobility via mobile applications.

By Matej Jusup, Kenan Zhang, Zhiyuan Hu, Barna P\'asztor, Andreas Krause, Francesco Corman

agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 2

Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning

arXiv:2412. 04177v2 Announce Type: replace Abstract: Recently, there has been an increasing interest in performing post-hoc uncertainty estimation about the predictions of pre-trained deep neural networks (DNNs).

By Luis A. Ortega, Sim\'on Rodr\'iguez-Santana, Daniel Hern\'andez-Lobato

benchmarks

arXiv AI

Jun 2

OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents

arXiv:2606. 02031v1 Announce Type: cross Abstract: Building capable visual web agents requires long-horizon reasoning, precise grounding, and robust interaction with dynamic real-world websites.

By Rui Yang, Qianhui Wu, Yuxi Chen, Hao Bai, Wenlin Yao, Hao Cheng, Baolin Peng, Huan Zhang, Tong Zhang, Jianfeng Gao

llms agents reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 2

Towards Stable, Globally Expressive Graph Representations with Laplacian Eigenvectors

arXiv:2410. 09737v2 Announce Type: replace Abstract: A popular way to improve the expressive power of graph neural networks (GNNs) is to use Laplacian eigenvectors as additional node features, since they can serve both as structural identifiers and global coordinates of nodes.

By Junru Zhou, Cai Zhou, Xiyuan Wang, Pan Li, Muhan Zhang

benchmarks

arXiv Machine Learning

Jun 2

Benchmark Dataset for Catalysis on 2D MXenes

arXiv:2606. 00794v1 Announce Type: cross Abstract: Merging first-principles calculations with machine learning (ML), we aim to accelerate the exploration of catalytic behaviour in novel materials.

By Pavlo Melnyk, Anmar Karmush, M{\aa}rten Wadenb\"ack, Ania Beatriz Rodr\'iguez-Barrera, Johanna Rosen, Michael Felsberg, Jonas Bj\"ork

benchmarks

arXiv Machine Learning

Jun 2

Quality Audio Prototyping: a prototype system for unified sound retrieval and procedural generation

arXiv:2606. 00629v1 Announce Type: cross Abstract: Sound design workflows frequently oscillate between time-consuming library searches and the complexity of procedural synthesis, with practitioners typically relying on disconnected tools to address each challenge separately.

By Nelly Garcia, Aditya Bhattacharjee, Gabryel Mason-Williams, Israel Mason-Williams, Emmanouil Benetos, Joshua Reiss

benchmarks

arXiv Machine Learning

Jun 2

Evaluating Real-World Generalizability of Algorithm Selection Models

arXiv:2606. 02016v1 Announce Type: new Abstract: Algorithm Selection (AS) aims to automatically identify the most suitable optimization algorithm for a given problem instance by leveraging measurable problem characteristics and historical performance data.

By Gjorgjina Cenikj, Jakub Kudela, Eva Tuba, Tome Eftimov

robotics benchmarks

arXiv Machine Learning

Jun 2

G2LoRA: Gradient Orthogonal Low-Rank Adaptation Framework for Graph Continual Learning on Text-Attributed Graphs

arXiv:2606. 01873v1 Announce Type: new Abstract: LLM-as-Aligner has emerged as a prevalent pre-training paradigm for Text-Attributed Graphs(TAGS), aligning graph and text modalities into a shared embedding space via CLIP-style contrastive learning.

By Yuhan Wang, Yibo Ding, Yutong Ye, Mufan Zhao, Wenbo Zhang, Ruijie Wang, Jianxin Li

llms rag fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

arXiv:2606. 01717v1 Announce Type: new Abstract: Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization.

By Minsik Choi, Geewook Kim

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

ATLAS: Agentic Test-time Learning-to-Allocate Scaling

arXiv:2606. 01667v1 Announce Type: new Abstract: Test-time scaling has become a major way to improve large language model reasoning, but its orchestration has remained designer-engineered: a fixed sample budget, a fixed refinement loop, a fixed scoring rule, or a fixed search policy decides how compute is spent, leaving the model in charge of solving but not of orchestration.

By Peijia Qin, Qi Cao, Pengtao Xie

llms agents nlp multimodal benchmarks

arXiv AI

Jun 2

SkyShield: Occupancy as a Safety Interface for Low-Altitude UAV Autonomy

arXiv:2606. 00747v1 Announce Type: cross Abstract: For low-altitude Unmanned Aerial Vehicle (UAV) autonomy, 3D spatial understanding is not merely a perception objective, but the safety interface between human instructions and physical flight.

By Jie Gao, Jie Ma, Kaihui Lin, Kai Ye, Miaohui Zhang, Pingyang Dai, Liujuan Cao

agents benchmarks safety

arXiv Machine Learning

Jun 2

Safe-Subspace Pseudo-Label Refinement for Source-Free Graph Domain Adaptation

arXiv:2606. 00808v1 Announce Type: new Abstract: Source-free graph domain adaptation (SF-GDA) aims to adapt source-trained graph models to unlabeled target graphs when source graphs are no longer accessible.

By Yingxu Wang, Xinwang Liu, Siyang Gao, Nan Yin

fine-tuning benchmarks

arXiv Machine Learning

Jun 2

OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models

arXiv:2606. 00815v1 Announce Type: new Abstract: Electroencephalography (EEG) supports a variety of brain-computer interface (BCI) tasks ranging from brain-state monitoring to human-LLM interactions.

By Ziling Lu, Zongsheng Li, Xinke Shen, Kexin Lou, Yingyue Xin, Xiaoqi Chen, Shinan Wang, Xiang Chen, Jiahao Fan, Chenyu Huang, Xin Xu, Zhoujie Hou, Chen Wei, Quanying Liu

llms benchmarks

arXiv AI

Jun 2

Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams

arXiv:2606. 01770v1 Announce Type: cross Abstract: Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks.

By Zewen Liu, Zhan Shi, Yisi Sang, Bing He, Minhua Lin, Tianxin Wei, Dakuo Wang, Benoit Dumoulin, Wei Jin, Hanqing Lu

llms agents benchmarks

arXiv Machine Learning

Jun 2

Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing

arXiv:2606. 00686v1 Announce Type: new Abstract: The prevailing paradigm in large language model (LLM) alignment operates via erasure, filtering unsafe data or training models to strictly refuse harmful prompts.

By Maryam Hashemzadeh, Jerry Huang, Minseon Kim, Marc-Alexandre C\^ot\'e, Sarath Chandar

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jun 2

Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization

arXiv:2606. 00544v1 Announce Type: new Abstract: Modern language-model fine-tuning typically pairs each prompt with a single response, even though many prompts admit multiple valid completions.

By Hasan Amin, Kian Ahrabian, Ming Yin, Rajiv Khanna

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

Canonicalized Stable-List Replay for Private Federated Continual Learning over Language-Model Embeddings

arXiv:2606. 00426v1 Announce Type: new Abstract: Federated continual learning (FCL) lets distributed clients adapt language-model heads to evolving NLP tasks without sharing raw text.

By Ibne Farabi Shihab, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Anuj Sharma

rag nlp benchmarks safety

arXiv AI

Jun 2

Learning When to Translate for Multilingual Reasoning

arXiv:2606. 02465v1 Announce Type: cross Abstract: Reasoning language models (RLMs) achieve strong performance on complex reasoning tasks, but still exhibit substantial multilingual reasoning gaps, largely due to language-understanding failures in non-English inputs.

By Deokhyung Kang, Hyounghun Kim, Gary Geunbae Lee

llms reinforcement-learning benchmarks

Get the daily briefing