Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Machine Learning

Sep 30

FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models

arXiv:2609.36202v1 Announce Type: cross Abstract: Gradient-based reward guidance provides a flexible way to use downstream reward models to control masked diffusion language models at inference time....

By Darshan Thaker, Lachlan Ewen MacDonald, Ren\'e Vidal

llms diffusion reinforcement-learning benchmarks

arXiv Machine Learning

Sep 30

Backdoor Mitigation in Decentralized LLM Fine-Tuning

arXiv:2609.37367v1 Announce Type: cross Abstract: Decentralized large language model (LLM) fine-tuning lets organizations collaboratively train a shared LLM on data they cannot pool, without a centra...

By Sayan Biswas, Jade Garcia Bourr\'ee, Rachid Guerraoui, Maxime Jacovella, Anne-Marie Kermarrec, Sathwika Peechara, Martijn de Vos, Milos Vujasinovic

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 30

KT-EGO: A Knowledge Transfer Assisted Efficient Global Optimization Algorithm for Solving High-Dimensional Expensive Black-Box Problems

arXiv:2609.37473v1 Announce Type: cross Abstract: Many engineering problems involve optimizing a high-dimensional expensive black-box (HEB) design space. To solve such problems efficiently, we propos...

By Qineng Wang, Liming Song, Yun Chen, Guangjian Ma, Zhendong Guo, Jun Li

benchmarks

arXiv Machine Learning

Sep 30

FORUM: Frozen Outputs Reconciled Using Model Agreement for Visual Grounding

arXiv:2609.37488v1 Announce Type: cross Abstract: Frozen multimodal large language models (MLLMs) now solve standard referring expression comprehension with a single prompted call, yet on adversarial...

By Taiyo Sato, Takamasa Sanda, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama, Shunya Nagashima

llms multimodal benchmarks safety

arXiv AI

Sep 30

Information Bottleneck-Guided Adaptive Hypergraph Transformer for Brain Disease Diagnosis

arXiv:2609.37220v1 Announce Type: new Abstract: Exploring high-order correlations and long-range dependencies in brain networks holds significant value for both neuroscience research and clinical dia...

By Jingxi Feng, Xudong Chen, Yifan Zhang, Heming Xu, Hongcheng Han, Xijing Wang, Dong Zhang, Shaoyi Du

llms benchmarks

arXiv Machine Learning

Sep 30

SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation

arXiv:2609.37918v1 Announce Type: cross Abstract: Reasoning across videos requires aligning events, matching identities, comparing motion, and integrating partial observations. Evaluating these capab...

By Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 30

Post-Anomaly Detection Inference for Deep SVDD

arXiv:2609.37935v1 Announce Type: cross Abstract: Deep Support Vector Data Description (Deep SVDD) has become a prominent framework for unsupervised anomaly detection by learning latent representatio...

By Cao Le Cong Thanh, Dang Quang Vinh, Vo Nguyen Le Duy

benchmarks safety

arXiv Machine Learning

Sep 30

MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers

arXiv:2508.14925v2 Announce Type: replace-cross Abstract: By providing a standardized interface for LLM agents to interact with external tools, the Model Context Protocol (MCP) is quickly becoming a...

By Zhiqiang Wang, Yichao Gao, Yanting Wang, Suyuan Liu, Haifeng Sun, Haoran Cheng, Guanquan Shi, Haohua Du, Xiangyang Li

llms agents benchmarks safety

arXiv Machine Learning

Sep 30

Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

arXiv:2602.15382v3 Announce Type: replace-cross Abstract: Heterogeneous multi-agent systems combine models with different capabilities through a common communication interface. Exchanging internal st...

By Xiaoze Liu, Ruowang Zhang, Weichen Yu, Siheng Xiong, Liu He, Feijie Wu, Hoin Jung, Matt Fredrikson, Xiaoqian Wang, Jing Gao

llms agents efficiency multimodal benchmarks safety

arXiv Machine Learning

Sep 30

IR-SIM: A Lightweight Declarative Simulator for Navigation Learning and Benchmarking

arXiv:2606.08729v2 Announce Type: replace-cross Abstract: Developing navigation policies requires simulation scenarios that support repeatable training and evaluation. Despite the availability of num...

By Ruihua Han, Shuai Wang, Chengyang Li, Rui Gao, Xinyi Wang, Zhe Liu, Guoliang Li, Yupu Lu, Qi Hao, Jia Pan, Hengshuang Zhao

llms agents reinforcement-learning robotics benchmarks

arXiv Machine Learning

Sep 30

Theoretical Guarantees for SMC-Guided Diffusion Sampling

arXiv:2607.04780v2 Announce Type: replace-cross Abstract: Post-hoc conditioning of pretrained diffusion models can be addressed using Sequential Monte Carlo (SMC) methods. By evolving an interacting...

By Stanislas Strasman (SU, LPSM), Gabriel Victorino Cardoso (STIM), Sylvain Le Corff (LPSM), Vincent Lemaire (LPSM), Antonio Ocello

diffusion benchmarks

arXiv Machine Learning

Sep 30

OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

arXiv:2609.32810v2 Announce Type: replace-cross Abstract: Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions,...

By Anqi Li, Zhixuan Ge, Yixuan Duan, Jiarong Qian, Chi-Yu Chen, MingYu Lu, Huan-Yu Hsu, Yu Gu, Yue Guo, Sheng Wang, Wei Qiu, Hanwen Xu

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Sep 30

Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models

arXiv:2609.33965v2 Announce Type: replace-cross Abstract: We describe a methodology for estimating the per-token energy cost of cloud-hosted large language model (LLM) inference, separating between i...

By Joshua Horswill, Ross Hunter, Matt Clifford, James Hall

llms robotics benchmarks