Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,521 stories · RSS feed

arXiv AI

Jul 8

HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment

arXiv:2604. 08435v2 Announce Type: replace-cross Abstract: It remains challenging to assess driver fatigue from untrimmed videos under constrained computational budgets, due to the difficulty of modeling long-range temporal dependencies in subtle facial expressions.

By Changdao Chen, Qinqiuhong Ye, Hao Chen, Jinyu Wang

efficiency multimodal benchmarks

arXiv Machine Learning

Jul 8

Improving TabPFN's Synthetic Data Generation by Integrating Causal Structure

arXiv:2603. 10254v2 Announce Type: replace Abstract: Synthetic tabular data generation addresses data scarcity and privacy constraints in a variety of domains.

By Davide Tugnoli, Andrea De Lorenzo, Marco Virgolin, Giovanni Cin\`a

benchmarks safety data-engineering

arXiv Machine Learning

Jul 8

PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

arXiv:2607. 01646v2 Announce Type: replace Abstract: State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack.

By Haotian Xie, Junlin Chen, Mingkai Zheng, Lishan Yang, Zhao Zhang

llms benchmarks

arXiv AI

Jul 8

Quantifying Frontier LLM Capabilities for Container Sandbox Escape

arXiv:2603. 02277v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly act as autonomous agents, using tools to execute code, read and write files, and access networks, creating novel security risks.

By Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz, Harry Coppock

llms agents benchmarks safety

arXiv Machine Learning

Jul 8

AtomBench: A Benchmarking Framework for Generative Crystal Reconstruction Models in Conventional Superconductors

arXiv:2510. 16165v2 Announce Type: replace Abstract: A key question in benchmarking generative crystal reconstruction models is how the amount and type of crystallographic information provided to a generative model affects its ability to reconstruct atomic structures.

By Charles Rhys Campbell, Aldo H. Romero, Kamal Choudhary

benchmarks

arXiv Machine Learning

Jul 8

Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.

By Yao Fu, Chunxia Zhang, Junmin Liu, Yihang Jin, Haishan Ye, Yuanao Yang

benchmarks

arXiv Machine Learning

Jul 8

Channel-wise Retrieval for Multivariate Time Series Forecasting

arXiv:2604. 05543v2 Announce Type: replace Abstract: Multivariate time series forecasting often struggles to capture long-range dependencies due to fixed lookback windows.

By Junhyeok Kang, Jun Seo, Soyeon Park, Sangjun Han, Seohui Bae, Hyeokjun Choe, Soonyoung Lee

rag benchmarks

arXiv AI

Jul 8

From Global to Granular: Revealing IQA Model Performance via Correlation Surface

arXiv:2601. 21738v2 Announce Type: replace-cross Abstract: Evaluation of Image Quality Assessment (IQA) models has long been dominated by global correlation metrics, such as Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank-Order Correlation Coefficient (SRCC).

By Baoliang Chen, Danni Huang, Hanwei Zhu, Lingyu Zhu, Wei Zhou, Shiqi Wang, Yuming Fang, Weisi Lin

benchmarks

arXiv Machine Learning

Jul 8

Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

arXiv:2607. 05861v1 Announce Type: cross Abstract: Large reasoning models (LRMs) improve language model capabilities by generating explicit thinking traces before final answers.

By Kaishen Wang, Tong Zheng, Xuehao Cui, Ruibo Chen, Tianyi Xiong, Heng Huang

llms nlp reinforcement-learning benchmarks safety

arXiv AI

Jul 8

DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising

arXiv:2603. 19216v2 Announce Type: replace-cross Abstract: Understanding and generating 3D objects as compositions of meaningful parts is fundamental to human perception and reasoning.

By Tianjiao Yu, Xinzhuo Li, Muntasir Wahed, Jerry Xiong, Yifan Shen, Ying Shen, Ismini Lourentzou

benchmarks safety

arXiv AI

Jul 8

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

arXiv:2605. 30794v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated significant achievements in general visual question answering (VQA) tasks.

By Qian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing

llms nlp multimodal benchmarks

arXiv Machine Learning

Jul 8

Energy-Efficient GPU DVFS for Fine-Tuning of SLMs on Resource-constrained Embedded Devices

arXiv:2607. 05933v1 Announce Type: cross Abstract: Dynamic Voltage Frequency Scaling (DVFS) on resource-constrained embedded GPU platforms is essential for energy-efficient small language model (SLM) fine-tuning, as privacy- and personalization-driven adaptation increasingly requires local execution and involves repeated forward-backward optimization over many mini-batches, making it substantially more time- and energy-intensive than single-pass inference.

By Jurn-Gyu Park, Sanzhar Zholdybayev, Aidar Amangeldi, Ademi Zhanuzakova

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 8

On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

arXiv:2607. 05836v1 Announce Type: cross Abstract: The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs.

By Don Li

benchmarks

arXiv Machine Learning

Jul 8

Label Hierarchy Transition: Delving into Class Hierarchies to Enhance Deep Classifiers

arXiv:2112. 02353v3 Announce Type: replace-cross Abstract: Hierarchical classification aims to sort the object into a hierarchical structure of categories.

By Renzhen Wang, De cai, Kaiwen Xiao, Xixi Jia, Xiao Han, Deyu Meng

benchmarks

arXiv AI

Jul 8

eCREAM-MedCorpus A Large-Scale Corpus of Clinical Notes for Italian

arXiv:2606. 12569v3 Announce Type: replace-cross Abstract: We present eCREAM-MedCorpus, a new and unique large-scale dataset of clinical notes produced in Emergency Departments of Italian hospitals.

By Tiziano Labruna, Guido Bertolini, Pietro Ferrazzi, Bernardo Magnini

llms benchmarks

arXiv AI

Jul 8

Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

arXiv:2607. 05399v1 Announce Type: cross Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks.

By Nikita Agrawal, Ruben Mayer

llms nlp efficiency benchmarks

arXiv Machine Learning

Jul 8

InvWeaver: Deductive Feedback for Invariant Synthesis in Interacting-Loop Programs

arXiv:2607. 05478v1 Announce Type: new Abstract: Loop invariant inference is a fundamental yet challenging problem in program verification.

By Guangyuan Wu, Weining Cao, Zehui Tan, Yuan Yao, Hengfeng Wei, Taolue Chen, Xiaoxing Ma

llms benchmarks

arXiv AI

Jul 8

To Retain or to Adapt? Generalizing Continual Learning

arXiv:2607. 05609v1 Announce Type: cross Abstract: The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting.

By Giulia Lanzillotta, Mandana Samiei, Doina Precup, Razvan Pascanu, Claire Vernade

computer-vision reinforcement-learning benchmarks safety

arXiv AI

Jul 8

From Graphs to Gradients: Physics-Inspired Structural Attribution for Cyber-Physical IoT Systems and Beyond

arXiv:2607. 05563v1 Announce Type: new Abstract: Interpretable explanation methods in Artificial Intelligence aim to uncover the underlying causes and their effects, enabling a deeper understanding of why a system behaves in a certain way under different inputs.

By Spyridon Evangelatos, Christos Diou, Georgios Th. Papadopoulos, Evangelos Markakis, Panagiotis Sarigiannidis

benchmarks

arXiv Machine Learning

Jul 8

Exogenous Dropout: A Simple, Strong Baseline for Corruption-Robust Time Series Forecasting with Covariates

arXiv:2607. 05452v1 Announce Type: new Abstract: Time series forecasters that use exogenous covariates are fragile in deployment: when those covariates are noised, temporally misaligned, or missing, strong exogenous-fusion and exogenous-adapted models can degrade far above the endogenous-only floor.

By Hao Hu, Xue-shan Ai

benchmarks

Get the daily briefing