Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,633 stories · RSS feed

arXiv Machine Learning

Jul 20

From Diffusion to Reaction-Diffusion: A Dynamical-Systems View of Oversmoothing in Hypergraph Neural Networks

arXiv:2607. 15773v1 Announce Type: new Abstract: Higher-order couplings enhance the expressive power of hypergraph neural networks (HGNNs), but they also intensify representation collapse in deep propagation due to strong multi-way feature mixing.

By Zhiheng Zhou, Mengyao Zhou, Yancheng Chen, Dengyi Zhao, Xingqin Qi, Guiying Yan

diffusion benchmarks

arXiv Machine Learning

Jul 20

AutoSpec: Automated Generation of Neural Network Specifications

arXiv:2409. 10897v3 Announce Type: replace Abstract: The increasing adoption of neural networks in learning-augmented systems highlights the growing need for model safety and robustness, especially in safety-critical domains.

By Shuowei Jin, Taobo Liao, Anuj Kalia, Xenofon Foukas, Huan Zhang, Cheng Tan, Z. Morley Mao, Francis Y. Yan

benchmarks safety

arXiv AI

Jul 20

S1-Omni: A Unified Multimodal Reasoning Model for Scientific Understanding, Prediction, and Generation

arXiv:2607. 15686v1 Announce Type: new Abstract: We present S1-Omni, a unified multimodal reasoning model for scientific understanding, prediction, and generation.

By Jiahao Zhao, Junyi Liu, Lifeng Xu, Nan Xu, Qingli Wang, Qingxiao Li, Tianle Chen, Xiaoyu Wu, Yawen Zheng, Zikai Wang, Guanming Liu, Hequn Zhou, Jingyi Wang, Jingyuan Shu, Keqi Wang, Li He, Songyang Diao, Wenhui Xu, Xinyu Ren, Yaqin Fan, Yujin Zhou, Zhanao Yao

llms diffusion multimodal benchmarks safety

arXiv AI

Jul 20

A Formally Grounded ODRL Evaluator: Implementation and Comparison

arXiv:2607. 15987v1 Announce Type: new Abstract: The ODRL policy language is emerging as the de-facto standard for policy modelling data access and usage preferences, AI governance policies and data workflows in European dataspaces.

By Jaime Osvaldo Salas, Paolo Pareti, Adeel Aslam, Christopher Maidens, George Konstantinidis

benchmarks

arXiv Machine Learning

Jul 20

Analysis of Semi-Supervised Learning on Hypergraphs

arXiv:2510. 25354v3 Announce Type: replace Abstract: Hypergraphs provide a natural framework for modeling multiway interactions.

By Adrien Weihs, Andrea L. Bertozzi, Matthew Thorpe

benchmarks

arXiv Machine Learning

Jul 20

Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraints

arXiv:2607. 16027v1 Announce Type: new Abstract: Introduction: Biological systems face anatomical and metabolic constraints, including costly synaptic maintenance and limited connectivity.

By Patrick Inoue, Florian R\"ohrbein, Andreas Knoblauch

rag efficiency benchmarks

arXiv Machine Learning

Jul 20

A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learning

arXiv:2507. 21018v2 Announce Type: replace-cross Abstract: Automated assessment of human motion plays a vital role in rehabilitation, enabling objective evaluation of patient performance and progress.

By Ali Ismail-Fawaz, Maxime Devanne, Stefano Berretti, Jonathan Weber, Germain Forestier

benchmarks

arXiv Machine Learning

Jul 20

Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark

arXiv:2607. 15935v1 Announce Type: cross Abstract: Deep reinforcement learning (DRL) has a longstanding tradition in addressing the reach-avoid task problem, especially for controlling robotic arms.

By Jonas Weihing, Shahram Eivazi

agents reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 20

Adaptive Multi-Step Lookahead Decoding for Diffusion Language Models

arXiv:2607. 15655v1 Announce Type: cross Abstract: Masked diffusion language models (DLMs) enable parallel text generation by iteratively refining masked tokens, offering a promising alternative to autoregressive decoding.

By Yingqian Cui, Wei Deng, Lantao Mei, Hang Li, Charu C. Aggarwal, Hui Liu, Yue Xing

llms diffusion benchmarks

arXiv Machine Learning

Jul 20

Intentional Electromagnetic Interference Attacks on Facial Recognition

arXiv:2607. 15512v1 Announce Type: cross Abstract: Attacks on general computer vision algorithms are often relegated to the digital domain, with the optimization performed purely in the digital world and then translated to physical mediums for implementation.

By Tyler Fitzsimmons, Adam Czajka

computer-vision benchmarks

arXiv AI

Jul 20

Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding

arXiv:2607. 15778v1 Announce Type: cross Abstract: Video Large Language Models (Video LLMs) have made significant advancements in various video understanding tasks.

By Wei Feng, Xin Wang, Yu-Wei Zhan, Yuwei Zhou, Wenwu Zhu

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 20

An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale

arXiv:2602. 14656v2 Announce Type: replace Abstract: Orthogonality constraints are ubiquitous in robust and probabilistic machine learning.

By Adri\'an Javaloy, Antonio Vergari

benchmarks

arXiv AI

Jul 20

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

arXiv:2603. 26556v2 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs.

By Juan Gabriel Kostelec, Qinghai Guo

llms efficiency benchmarks

arXiv Machine Learning

Jul 20

MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation

arXiv:2607. 15299v1 Announce Type: cross Abstract: In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation.

By Zhiyuan Zhao, Bin Wang, Linke Ouyang, Yiqi Lin, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, Conghui He

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 20

An Exam for Active Observers

arXiv:2607. 16165v1 Announce Type: cross Abstract: Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot.

By Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger

llms multimodal benchmarks

arXiv Machine Learning

Jul 20

SODA: Semi On-Policy Black-Box Distillation for Large Language Models

arXiv:2604. 03873v4 Announce Type: replace Abstract: Black-box knowledge distillation for large language models presents a strict trade-off.

By Xiwen Chen, Jingjing Wang, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Yueyue Deng, Hejian Sang, Zhipeng Wang, Alborz Geramifard, Feng Luo

llms efficiency benchmarks safety

arXiv Machine Learning

Jul 20

SloMo-Fast: Slow-Momentum and Fast-Adaptive Teachers for Source-Free Continual Test-Time Adaptation

arXiv:2511. 18468v2 Announce Type: replace Abstract: Continual Test-Time Adaptation (CTTA) is crucial for deploying models in real-world applications with unseen, evolving target domains.

By Md Akil Raihan Iftee, Mir Sazzat Hossain, Rakibul Hasan Rajib, Tariq Iqbal, Md Mofijul Islam, M Ashraful Amin, Amin Ahsan Ali, AKM Mahbubur Rahman

benchmarks safety

arXiv Machine Learning

Jul 20

Discovering Generalizable Governing Equations for Graph Dynamical Systems with Interpretable Neural Networks

arXiv:2508. 18173v2 Announce Type: replace Abstract: The discovery of symbolic governing equations is a central goal in science; yet, it remains challenging particularly for graph dynamical systems, where the network topology further shapes the system behavior.

By Riccardo Cappi, Paolo Frazzetto, Nicol\`o Navarin, Alessandro Sperduti

benchmarks

arXiv AI

Jul 20

DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings

arXiv:2607. 15418v1 Announce Type: new Abstract: We introduce DrawingVQA, the first benchmark designed to evaluate multimodal large language models (MLLMs) on real-world construction drawings -- a core media in architecture, civil, and many other engineering practices.

By Yoonhwa Jung, Junryu Fu, Mani Golparvar-Fard

llms multimodal benchmarks

arXiv AI

Jul 20

NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs

arXiv:2607. 14186v2 Announce Type: replace-cross Abstract: Synthesizing training data to scale agent capabilities in LLM post-training is bottlenecked by substrate-bound task synthesis: tasks are generated from fixed tools, repositories, or skill graphs, so expanding coverage requires manual substrate engineering, transferring to a new domain demands bespoke infrastructure, and the resulting distributions inherit substrate biases rather than reflecting real-world demand.

By Jiarong Zhao, Zhikai Lei, Zhiheng Xi, Rui Zheng, Hang Yan, Jie Zhou, Qin Chen, Liang He

llms agents fine-tuning benchmarks

Get the daily briefing