Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Computer Vision
Sep 11

Measuring Browser Webcam Gaze Honestly: A Capture-Clock Methodology and Open Reference Implementation

The paper addresses the problem of inaccurate latency reporting in browser-based webcam gaze trackers, which often timestamp samples at emission rather than capture time. It introduces a method that recovers a per-frame capture clock using the browser’s requestVideoFrameCallback API, enabling precise pairing of source frames with inference results or providing a verifiable lower bound when the engine does not expose its pipeline. An open TypeScript implementation and benchmark harness are released, tested on WebGazer and a FaceMesh+KRR pipeline.

By Chi-Sheng Chen, Gabriel A. Brat
arXiv Statistics ML
Sep 11

A Mean Field Games Perspective on Evolutionary Clustering

The paper introduces a control‑theoretic framework for evolutionary clustering using quasi‑stationary Mean Field Games. Each cluster is modeled as a probability density whose dynamics follow a Fokker–Planck equation, while a stationary Hamilton–Jacobi equation determines the velocity field. In a Gaussian specialization, affine dynamics replicate the mean and covariance trajectories of the classical Expectation–Maximization algorithm, and the authors propose causal and non‑causal time‑averaged log‑likelihood objectives to enhance temporal coherence, along with a fully density‑based numerical implementation for non‑Gaussian components. The method is evaluated on synthetic and real time‑dependent datasets, compared to snapshot Expectation–Maximization, temporally smoothed observations, and an evolutionary k‑means baseline.

By Alessio Basti, Fabio Camilli, Adriano Festa
arXiv AI
Sep 11

Fortunate Recall: Ontology-Driven Memory Lifecycle Management for Persistent Coherence in LLMs

Fortunate Recall (FR) introduces an ontology-driven policy layer that categorizes personal facts into over ten behavioral types and applies tailored lifecycle rules—such as differential decay, supersession, and event-time validity—to manage memory persistence in large language models. The FR-Bank implementation, independent of underlying infrastructure, achieves a 76.9% pass rate on the new LifecycleBench benchmark and improves LongMemEval-S performance, while significantly reducing confabulation rates compared to prior systems. Ablation studies show that the generic lifecycle metadata drives correctness, whereas the behavioral ontology enhances calibration and reduces downstream hallucinations.

By Ansuman Mullick, Eray T\"uz\"un
arXiv AI
Sep 11

Can AI Agents Detect and Repair Artifact Drift in Network Experiments?

The paper introduces NetArtifactBench, a benchmark designed to evaluate whether AI agents can detect and repair inconsistencies in network experiment records while preserving supported claims. It tests 23 agent configurations on 52 instances with injected inconsistencies, finding an average pass rate of 65.3 % but no runtime exceeding 30 % for complex repairs that require recovering implicit relations and propagating changes across artifacts. The results highlight a clear distinction between local corrections and full record-level repair, leading the authors to argue that artifact integrity should be a primary design and evaluation criterion for AI agents in network systems.

By Tianzhu Zhang, Weichen Tao, Changgang Zheng, Yusheng Zheng, Long Chen, Xiaoyi Fan, Meikang Qiu
arXiv Computer Vision
Sep 11

MC-DeTra: Motion-Consistent Joint Object Detection and Socially-Aware Trajectory Forecasting in Bird's-Eye-View Images

MC-DeTra is a reimplementation of the DeTra model that jointly performs object detection and socially-aware trajectory forecasting in bird's-eye-view images. It introduces motion-consistency mechanisms that add supervision from each actor’s past motion, surrounding traffic occupancy, and a consistency constraint aligning predicted heading with motion direction. The added losses are train‑only and inference‑safe, improving dynamic trajectory forecasting on the Waymo Open Dataset while maintaining or enhancing detection accuracy.

By Vladislav Diuzhev, Dmitry Yudin
arXiv Machine Learning
Sep 11

AdamX: Cosine similarity meets gradient descent

AdamX is a new first‑order optimizer that uses cosine similarity to adaptively control update magnitudes, making it scalable, model‑agnostic, and easy to add to existing training pipelines. It also includes a variance rectification scheme that smooths optimization early in training. Empirical results show AdamX achieves competitive convergence rates across various benchmark datasets and architectures, measured by the number of epochs needed to hit predefined performance thresholds under a fixed hyperparameter budget.

By Francisco Caldas, Ruben Belo, Cl\'audia Soares
arXiv AI
Sep 11

Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations

The paper examines whether internal representations of agentic systems can better indicate task success than traditional confidence measures. It introduces two methods—Latent Trajectory Dynamics (LTD) and Action Representation Probe (ARP)—that analyze changes in residual-stream representations and action-level representations, respectively. Experiments on Bash, SQL, and Python benchmarks with Qwen and DeepSeek models show these methods outperform conventional surface-level and sequence-based calibration baselines, offering a zero‑overhead reliability monitor without prompt changes or multiple rollouts.

By Priyanka Mary Mammen, Emil Joswin, Srujananjali Medicherla
arXiv Machine Learning
Sep 11

A Station-Based Evaluation of Machine Learning-based Weather Forecasting Models in Northern Norway

The study evaluates three machine‑learning weather‑prediction models—FourCastNet3, GraphCast, and the ECMWF High‑Resolution Forecast—for wind‑speed forecasting in Northern Norway using multi‑year station data. Results show the ECMWF model slightly outperforms the ML models (RMSE 2.89 m s⁻¹ vs. 2.96 m s⁻¹ for FourCastNet3 and 2.94 m s⁻¹ for GraphCast), yet all models maintain comparable performance beyond their training periods and underestimate strong winds. FourCastNet3 performs best under high‑wind conditions, indicating ML models are competitive with traditional numerical weather prediction but still need improvement for complex terrain.

By Siyan Chen, Lars Uebbing, Eirik Mikal Samuelsen, Georgios Leontidis, Arnt-B{\o}rre Salberg, S\'ebastien Lef\`evre, Robert Jenssen, Kristoffer Wickstr{\o}m
arXiv AI
Sep 11

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

AgentAudit is an open, extensible framework that evaluates the full lifecycle of AI agents, assessing planning, tool selection, execution, memory, and reasoning across ten dimensions such as instruction integrity, security, and alignment. Unlike existing benchmarks that focus on single aspects, AgentAudit analyzes the entire execution trace to attribute failures to specific stages. The framework was tested on five large language models, revealing significant differences in trustworthiness even among models with similar task‑completion performance.

By Shrey Nag, Sachita, Abhishek Kumar Singh, Lipi Goel, Rajeshwar Singh Janwar
arXiv Machine Learning
Sep 11

Studying Without a Syllabus: Task-Agnostic Environment Preprocessing

The paper investigates whether a language‑model agent can autonomously study an unfamiliar environment without prior task instructions or examples, and decide how to prepare for future tasks. It formalizes task‑agnostic environment preprocessing, where a studying system explores under a budget to produce reusable artifacts for a later solver. Experiments on six diverse benchmarks show that a meta‑agent variant often outperforms fixed methods, though larger budgets do not consistently boost downstream reward, yet the artifacts still reduce test‑time sampling needed to achieve a target score.

By Vinay Samuel, Varun Ursekar, Vijay S. Kalmath, Apaar Shanker, Veronica Chatrath, Yuan Xue
arXiv Computer Vision
Sep 11

Beyond Benchmarks: Using VLMs to Reveal Systematic Classification Failures Under Real World Conditions

The paper investigates using Vision Language Models (VLMs) to accelerate verification and validation (V&V) of classification models by automatically detecting systematic errors. It introduces a VLM-based error slice detection (ESD) method that groups and labels errors, demonstrating its ability to identify perturbations in a non-military dataset and to cluster images by surroundings in a military context. The study highlights challenges such as underrepresentation of defence data in VLM training and limited contextual diversity, and suggests that while fully automated V&V is not yet feasible, VLMs could speed up the process in the future.

By Dieuwertje Alblas, Alma M. Liezenga, Jan Erik van Woerden, Fedor Taggenbrock, Dalia Aljawaheri, Klamer Schutte
arXiv Machine Learning
Sep 11

Structural priors for data-efficient language learning

The paper explores structural transfer, where models are first trained on non-language data such as music, probabilistic grammars, and cellular automata to induce priors for natural language tasks. This pretraining acts as a weight initialization for multilingual language modeling and leads to lower next-token prediction loss and smaller weight shifts during subsequent language training. However, the improved loss does not consistently translate into better downstream linguistic performance, and the efficiency of non-language data is lower than that of additional language data.

By Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager, Lisa Beinborn
arXiv Machine Learning
Sep 11

SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

SG-Blend introduces a per‑layer adaptive activation that interpolates between a bias‑corrected, parametric Swish variant (SSwish) and GELU, using a learnable blend coefficient, sharpness, and zero‑centering bias. The method adds only three scalars per feed‑forward block and, on BERT‑style IMDB classification, matches peak accuracy while reducing seed‑to‑seed variance by 42 %. It also achieves the lowest validation perplexity on WikiText103 and generalizes to computer vision and other domains.

By Gaurav Sarkar, Syed Affan Daimi, Jay Gala, Subarna Tripathi
arXiv Computation and Language
Sep 11

Probing for Knowledge Attribution in Large Language Models

The paper introduces a method for identifying the dominant knowledge source behind large language model (LLM) outputs, distinguishing between faithfulness violations (misuse of provided context) and factuality violations (errors in internal knowledge). A simple linear probe trained on hidden representations can reliably classify this source, and the authors present AttriWiki, a self‑supervised pipeline that generates labeled training data by prompting models to recall withheld entities or read them from context. Probes trained on AttriWiki achieve high Macro‑F1 scores across several models and datasets, generalize zero‑shot to a benchmark, and show that attribution mismatches can increase error rates by up to 70%. "whyItMatters":"The study demonstrates that knowing the source of an LLM’s answer is crucial for effective mitigation of hallucinations, as attribution mismatches significantly raise error rates."

By Ivo Brink, Alexander Boer, Dennis Ulmer
arXiv Machine Learning
Sep 11

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention proposes a two-stage hierarchical indexer that replaces the flat token scan used in token-level sparse attention mechanisms like DeepSeek Sparse Attention. The method first performs block-level coarse filtering to discard irrelevant regions, then applies the original token-level indexer only within the retained candidate blocks, preserving the same top-sparse pattern for downstream attention. Benchmarks show HISA achieves significant speedups at 64K context and matches the quality of DeepSeek-V3.2 and GLM-5 without additional training.

By Yufei Xu, Fanxu Meng, Fan Jiang, Yuxuan Wang, Ruijie Zhou, Zhaohui Wang, Jiexi Wu, Zhixin Pan, Xiaojuan Tang, Wenjie Pei, Tongxuan Liu, Di Yin, Xing Sun, Muhan Zhang
arXiv Computer Vision
Sep 11

Domain Elastic Transform: Bayesian Function Registration for High-Dimensional Scientific Data

Domain Elastic Transform (DET) is a grid‑free, probabilistic framework that jointly aligns geometry and high‑dimensional vector‑valued functions on irregular sparse manifolds, such as those found in spatial transcriptomics. By treating data as functions rather than voxelized images, DET performs unsupervised, scalable registration through sampled point alignment and displacement interpolation, guided by a joint spatial‑functional Bayesian likelihood. Evaluations on MERFISH mouse‑brain slices and Stereo‑seq mouse‑embryo atlases show DET achieving superior spatial overlap and topology compared to existing pipelines, with an accelerated variant delivering high label‑transfer accuracy.

By Osamu Hirose, Emanuele Rodola
arXiv Computation and Language
Sep 11

MindTopo: Can Foundation Models Reason in Topological Space?

MindTopo is a benchmark that tests foundation models on topological reasoning, covering five cognitive properties—continuity, separation, order, enclosure, and knots—across two cognitive levels: reasoning and planning. It contains 11,030 instances from 13 procedurally generated task types, and evaluates 14 multimodal large language models, including agent configurations with image and video generation. Results show that models perform better on reasoning than planning, and even the best model lags far behind human performance, with fine‑tuning and reinforcement learning improving reasoning more than planning.

By Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica, Jianwen Lyu, Zihan Wang, Reuben Tan, Jianfeng Gao, Ruohan Zhang, Yining Hong, Jiajun Wu, Manling Li
arXiv Machine Learning
Sep 11

Robust Multimodal Sentiment Analysis with Incomplete Modalities via Semantic-aware Completeness based Reconstruction

The paper presents a method for robust multimodal sentiment analysis that handles incomplete or noisy modalities. It introduces a completeness estimation technique to measure how much sentiment-relevant information remains in partial data, guiding the reconstruction of missing semantics. A joint training strategy stabilizes multi-task learning for sentiment prediction and completeness estimation, and experiments on three benchmark datasets show improved semantic reconstruction and sentiment accuracy.

By Han-Jun Choi, Byunggill Joe, Saim Shin, Jin Yea Jang
arXiv Machine Learning
Sep 11

New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models

The paper introduces a new benchmark for vision‑language models that tests their ability to decide whether to answer a physics question immediately or to request additional experimental evidence. Each problem presents one measurement image and four possible physical worlds defined by two masses and two values of another property; the model must either stop and answer or choose the cheapest experiment that resolves the question. Across six open models and 144 parameter sets, the models almost always repeat the same action even when the optimal choice changes, and only a single model gets both decisions correct on 5.9% of cases.

By Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar, Shaswati Saha, Yuxuan Jiang, Siyuan Li, Qiheng Wang