Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv Machine Learning

Jul 21

The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture

arXiv:2607. 17146v1 Announce Type: cross Abstract: We present a continuous geometric framework that models the discrete algebraic operations of the Transformer architecture as an integro-differential equation (IDE) on a semantic fiber bundle $\calE = \calM \times \R^d$.

By Zhihua Liang

llms diffusion benchmarks

arXiv AI

Jul 21

Otap:Structure-Aware Optimal Transport for Evaluating Planning and Execution in Agent Trajectories

arXiv:2607. 17082v1 Announce Type: new Abstract: Large language model agents solve tasks by generating trajectories that interleave planning, tool calls, and intermediate results.

By Babak Barazandeh, Subhabrata Majumdar, George Michailidis

llms agents benchmarks

arXiv Machine Learning

Jul 21

Bridging battery design and health assessment through virtual sensing and physics-informed learning

arXiv:2607. 16864v1 Announce Type: new Abstract: Supercharging of lithium-ion batteries (LiBs) requires robust health monitoring to ensure durability, safety, and user confidence, particularly for emerging vehicle-to-grid applications with bidirectional energy flows.

By Wendi Guo, S{\o}ren Byg Vilsen, Daniel Ioan Stroe, Yaqi Li, Yicun Huang, Ashima Verma, Daniel Brandell

rag diffusion benchmarks safety

arXiv Machine Learning

Jul 21

Interpretable Anomaly and Drift Detection with Gaussian Mixture Models

arXiv:2607. 16811v1 Announce Type: new Abstract: We revisit Gaussian Mixture Models (GMMs) as a lightweight, interpretable tool for anomaly detection and, in particular, for detecting distributional drift in data streams.

By Behnam Asadi

benchmarks

arXiv AI

Jul 21

The Shared Discovery Paradox: How a One-Answer Rule Turns Better Information into Worse Search

arXiv:2607. 18045v1 Announce Type: new Abstract: Organizations often pool dispersed information into one ranking and then allow many agents to act on that shared view.

By Yohei Nakajima

agents benchmarks

arXiv AI

Jul 21

Do Maps Still Matter for Machines: Revisiting the Role of Choropleth Maps in Foundation Model Spatial Understanding

arXiv:2607. 17999v1 Announce Type: new Abstract: Spatial understanding is crucial for foundation models (FMs), and maps have long helped humans organize and reason about geographic information.

By Zhiwei Wei, Yonghe Sun, Zhenjia Liu, Wenjia Xu, Chao He, Weihua Dong, Chunbo Liu, Hua Liao

llms benchmarks

arXiv Machine Learning

Jul 21

HyBDM: Multi-Scale Hybrid Experts for Time Series Forecasting with Bidirectional Dependency Modeling

arXiv:2607. 16882v1 Announce Type: new Abstract: Time series forecasting (TSF) is vital to many applications, yet existing models often struggle to capture the heterogeneous long-range global patterns and short-range local variations in multivariate time series.

By Wenqiang Ma, Chen Cheng, Xue Cheng, Jiarui Ye

llms benchmarks

arXiv Machine Learning

Jul 21

Chameleon: A Multiplier-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data

arXiv:2505. 24852v3 Announce Type: replace-cross Abstract: On-device learning at the edge enables low-latency, private personalization with improved long-term robustness and reduced maintenance costs.

By Douwe den Blanken, Charlotte Frenkel

benchmarks

arXiv Machine Learning

Jul 21

A Benchmark of Generative Methods for Zero-Shot Environmental Sound Classification

arXiv:2412. 03771v4 Announce Type: replace-cross Abstract: Zero-shot learning enables models to generalise to unseen classes using semantic information, bridging the gap between training classes and previously unseen test classes.

By Ysobel Sims, Alexandre Mendes, Stephan Chalup

rag diffusion computer-vision benchmarks safety

arXiv AI

Jul 21

Exploratory and Assimilating Reflection: Reflective Recall Cycle for Long-term Memory

arXiv:2607. 17879v1 Announce Type: new Abstract: LLM-based autonomous agents require external memory to overcome their statelessness and limited context window for long-term interaction and dynamic knowledge reasoning.

By Ganesh Senrayan, Moyuru Yamada, Ishan Jindal, Kiran Purohit

llms agents benchmarks

arXiv Machine Learning

Jul 21

Time-Aware Prior Fitted Networks for Zero-Shot Forecasting with Exogenous Variables

arXiv:2603. 15802v2 Announce Type: replace Abstract: In many time series forecasting settings, the target time series is accompanied by exogenous covariates, such as promotions and prices in retail demand; temperature in energy load; calendar and holiday indicators for traffic or sales; and grid load or fuel costs in electricity pricing.

By Andres Potapczynski, Ravi Kiran Selvam, Tatiana Konstantinova, Malcolm Wolff, Kin G. Olivares, Ruijun Ma, Michael W. Mahoney, Andrew Gordon Wilson, Boris N. Oreshkin, Dmitry Efimov

benchmarks

arXiv AI

Jul 21

Mechanistic Attention Guidance for Agent Memory Refinement

arXiv:2607. 17621v1 Announce Type: new Abstract: Existing self-evolving memory systems mainly improve agent memory based on textual outputs, such as task trajectories and reflections.

By Yechao Hong, Haiquan Qiu, Yaqing Wang, Quanming Yao

agents benchmarks

arXiv AI

Jul 21

Coordinated Disentanglement with Iterative Mode Discovery Under Hidden Correlations

arXiv:2607. 17264v1 Announce Type: new Abstract: Disentangled representation learning is a powerful paradigm for robust attribute prediction.

By Rong Hu, Ling Chen

benchmarks

arXiv Machine Learning

Jul 21

SpecXMaster Technical Report

arXiv:2603. 23101v3 Announce Type: replace Abstract: Intelligent spectroscopy serves as a pivotal element in AI-driven closed-loop scientific discovery, functioning as the critical bridge between matter structure and artificial intelligence.

By Yutang Ge, Yaning Cui, Hanzheng Li, Jun-Jie Wang, Fanjie Xu, Jinhan Dong, Yongqi Jin, Dongxu Cui, Peng Jin, Guojiang Zhao, Hengxing Cai, Tianci Yangfeng, Xueqing Chen, Hongshuai Wang, Rong Zhu, Linfeng Zhang, Xiaohong Ji, Zhifeng Gao

agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 21

When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

arXiv:2602. 14161v2 Announce Type: replace Abstract: Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization.

By Max Fomin

llms agents benchmarks safety

arXiv AI

Jul 21

Fourier Geometric Wind Power Forecasting with Numerical Weather Prediction

arXiv:2607. 17095v1 Announce Type: new Abstract: Accurate short-term wind power forecasting is essential for grid stability and operational planning, yet remains challenging due to the complex interactions between atmospheric conditions and turbine dynamics.

By Shiyuan Piao, Fan Zehui, Yang Liu, Hong Cheng, Juepeng Zheng, Jie Zhou, Fugee Tsung

multimodal benchmarks

arXiv AI

Jul 21

AEC-DS: Adaptive Erasure Coding with PDP-Triggered Reputation and QoS-Aware Migration for Decentralized Storage

arXiv:2607. 17460v1 Announce Type: new Abstract: In decentralized storage systems, audit results are often not used directly to guide later redundancy and shard-placement decisions, which can lead to inefficient resource allocation and delayed recovery.

By Shuaiwen Li, Weihang Yu, Ke Wang, Meng Han

benchmarks

arXiv AI

Jul 21

Learning Structural Manipulability in Gate-Level Netlists Using Graph Neural Networks

arXiv:2607. 16245v1 Announce Type: cross Abstract: Gate-level netlists exhibit intrinsic structural properties that influence signal propagation independently of functional simulation.

By Rupesh Raj Karn, Ozgur Sinanoglu

rag benchmarks

arXiv AI

Jul 21

MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

arXiv:2607. 18006v1 Announce Type: cross Abstract: Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 \, \mathrm{B}$ parameters) trained under limited budgets.

By Martino M. L. Pulici, Cuong Xuan Chu, Evgeny Kharlamov, Zifeng Ding, Volker Tresp, Yunpu Ma

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 21

GenSyn10: A Multi-Generative AI Dataset For Benchmarking Image Classification

arXiv:2607. 16283v1 Announce Type: cross Abstract: The rapid advancement of generative AI has outpaced our ability to reliably detect its outputs, particularly when detectors encounter generators they have not seen before.

By Md Faraz Kabir Khan, Saeed Anwar, Ghulam Mubashar Hassan

llms diffusion computer-vision fine-tuning multimodal benchmarks

Get the daily briefing