Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 9

Elucidating the Space of Enzymatic Reaction: A Unified Benchmark and Pretrained Model

The paper introduces VenusRX-Bench, a unified benchmark that evaluates forward reaction prediction, single‑step retrosynthesis, and EC‑number prediction for enzymatic reactions, integrating data from multiple biochemical databases with standardized curation and leakage‑controlled splits. It also presents VenusRX, a T5‑style sequence‑to‑sequence model that jointly learns these tasks, leveraging two‑stage training on millions of template‑expanded reactions and real biochemical data, with optional EC conditioning and molecule‑library‑constrained decoding to improve complex biomolecule generation. Across benchmark tasks and challenging generalization splits, VenusRX outperforms or matches existing chemical and enzymatic baselines, demonstrating that incorporating EC information and learned reaction representations yields a bidirectional relationship between reaction structure and catalytic function.

By Yutong Hu, Tianming Huang, Yanbo Zhao, Qiongyu Zhang, Shixiang Tang, Lei Bai, Ziyi Zhou, Liang Hong, Pan Tan
arXiv AI
Oct 9

From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

The paper evaluates eight open‑source lightweight Vision‑Language Models (up to 7 B parameters) for converting document images into structured JSON, focusing on heritage collections with historical handwriting and non‑standard layouts. It compares zero‑shot, few‑shot, and fine‑tuned settings, measuring performance with CER, ANLS*, and mAP‑F1, and examines the effects of hyperparameter tuning, image preprocessing, and multi‑stage training. The study demonstrates that adapted VLMs can serve as a sustainable, private alternative to commercial OCR systems for institutional archives.

By Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen
arXiv AI
Oct 9

Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency

The paper introduces Spatial Alignment via Geometric Evolution (SAGE), a self‑evolving framework designed to strengthen robust spatial reasoning in Vision‑Language Models (VLMs). SAGE leverages geometric and linguistic duality operations, incorporating duality consistency into training to ensure coherent answers across original and transformed inputs. The method dynamically updates its operation pool, focusing on challenging dualities while retiring mastered ones, and is model‑agnostic, data‑efficient, and applicable as a lightweight adaptation stage to existing VLMs, yielding consistent improvements on video and spatial reasoning benchmarks.

By Junming Liu, Yuqi Li, Yifei Sun, Maonan Wang, Yiming Cheng, Rui Qian, Piotr Koniusz, Yirong Chen, Ding Wang
arXiv AI
Oct 9

An Explainable Header-Centric Framework for Large-Scale Semantic Table Interpretation and Data Quality Assessment

The paper introduces an explainable, header‑centric framework for semantic table interpretation and data quality assessment that operates solely on column headers. It maps headers to 39 interpretable types using curated lexical resources, applies validation rules to detect data quality issues, and aggregates these into a lightweight quality metric called HeadersIQ. The framework was evaluated on diverse benchmarks, demonstrating broad coverage of noisy real‑world metadata and providing a workflow for metadata‑driven semantic annotation, quality monitoring, and benchmark diagnosis.

By Marcelo Valentim Silva, Hannes Herrmann, Valerie Maxville
arXiv Machine Learning
Oct 9

Sample-Efficiency of Kolmogorov-Arnold Networks

The paper investigates the sample‑efficiency of Kolmogorov‑Arnold Networks (KANs) in reinforcement learning. Using experiments on the Feynman dataset and Gymnasium benchmarks, the authors find that KANs achieve comparable performance to standard Multi‑Layer Perceptrons while requiring 40% fewer samples, with up to 50% relative performance gains during training. These improvements hold even when reward noise varies, underscoring KANs’ potential for more sample‑efficient RL.

By Kevin Riehl, Shaimaa K. El-Baklish, Fan Wu, Anastasios Kouvelas
arXiv Machine Learning
Oct 9

Controlled Acquisition and Abstention in Three-Channel Score Conflicts

The paper introduces a controlled benchmark for deciding whether to acquire a third modality or abstain when audio, video, and text scores conflict. A threshold policy that observes two signed scores can request the third at a cost and abstain, achieving higher targeted decision accuracy and utility than always-answer fusion under certain reward settings. Experiments on synthetic and real data show that the policy’s advantage depends on the ambiguity mechanism, budget constraints, and the ranking of observed pairs, highlighting the trade‑off between full-modality accuracy and pre‑request selection value.

By Mengzhe Geng
arXiv Machine Learning
Oct 9

MotherTree: Meta-learning on synthetic data improves decision tree training

MotherTree is a tabular transformer that meta‑learns to produce hard, axis‑aligned decision trees in a single forward pass. It is pre‑trained on a synthetic prior distribution using stochastic gradient descent, without needing reference trees for supervision. On controlled benchmarks, MotherTree matches or surpasses size‑matched trees from traditional algorithms and serves as a strong initializer for task‑specific fine‑tuning.

By Ziyuan Wang, Fredrik D. Johansson
arXiv Machine Learning
Oct 9

KDFP: A first-principles approach to knowledge distillation in large language models

KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.

By Ryan Swift, Konstantinos Psounis
arXiv Machine Learning
Oct 9

ASPIRE: Saddle-Point Discovery through Set Prediction and Physical Refinement

ASPIRE (Atomistic Saddle-Point Inference with Refinement for Events) is a new framework that predicts multiple saddle-point candidates from a single atomic environment and refines them using Dimer searches on the original interatomic potential. Its equivariant set predictor, Ev-Quiformer, generates proposals via geometry-conditioned scalar‑vector event slots and decodes them into full atomic displacement fields. The authors provide two new datasets—BCCFE4VACAV‑4000 and BCCFE‑1TO4VAC—and demonstrate that ASPIRE achieves 77.20 % reference‑event coverage, outperforming a conventional Dimer baseline while halving the number of force evaluations and reducing wall‑time per configuration from 478.8 s to 176.3 s.

By Yucheng Zhao, Quanyou Zhang, Shaoxiang Qin, Haixuan Xu, Xiongye Xiao
arXiv Machine Learning
Oct 9

CityDeploy-Bench: Benchmarking Physics-Grounded Spatial Set Planning for Multi-Transmitter Network Deployment

CityDeploy-Bench is a new benchmark that frames multi‑transmitter wireless network deployment as a physics‑grounded spatial set planning problem, verified by a unified ray‑tracing engine. It decouples utility representation from planning dynamics, allowing systematic comparison of direct scalar rewards, relational models, and higher‑order interaction structures across different planners. Experiments show that as physical coupling increases, deployment quality hinges on capturing collective transmitter interactions; stronger search alone cannot compensate for missing relational structure, highlighting the coordination nature of the problem.

By Chenyang Yuan, Xiaoyuan Cheng
arXiv Machine Learning
Oct 9

DaCe-DT: Data-Centric Offline Multi-Task Reinforcement Learning via Adaptive Prompts and Trajectory Correction for Heterogeneous Tasks

DaCe-DT is an offline multi‑task reinforcement learning framework that tackles data‑level bottlenecks by introducing length‑gated prompt masking, retrieval‑augmented prompt construction, and value‑adaptive return calibration. These mechanisms adapt prompts to diverse task complexities, ensure semantic relevance of prompt segments, and correct misleading supervision from fragmented trajectories. Experiments on Meta‑World show that DaCe‑DT outperforms state‑of‑the‑art methods, improving performance by 11.73% on optimal datasets and 13.34% on suboptimal datasets.

By Xinfei Wang, Shanchen Pang, Chenhao Zhang, Shudong Wang, Wenhao Ji, Haiyuan Gui, Meng Han, Xiaojian Liao
arXiv AI
Oct 9

AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda

The paper introduces AyurParam-2.9B, a bilingual language model fine‑tuned for Ayurveda using a curated dataset of classical texts and clinical guidance in English and Hindi. It outperforms other open‑source instruction‑tuned models of similar size on the BhashaBench‑Ayur benchmark and rivals larger models in performance. The work demonstrates that authentic domain adaptation and high‑quality supervision are essential for reliable, culturally congruent AI in specialized medical fields.

By Sravan Gorugantu, Mohd Nauman, Vijay Devane, Shyam Pawar, Viraj Thakur, Kundeshwar Pundalik, Piyush Sawarkar, Rohit Saluja, Maunendra Desarkar, Ganesh Ramakrishnan
arXiv AI
Oct 9

GPF-Net: Gated Progressive Fusion Learning for Polyp Re-Identification

The paper introduces GPF‑Net, a Gated Progressive Fusion Network designed for colonoscopic polyp re‑identification. It fuses multimodal features from multiple levels using fully connected gating mechanisms, allowing layer‑wise refinement of semantic information. Experiments on standard benchmarks show that this multimodal, progressive fusion approach outperforms state‑of‑the‑art unimodal ReID models, especially for small polyps where fine‑grained details are crucial.

By Suncheng Xiang, Xiaoyang Wang, Junjie Jiang, Hejia Wang, Dahong Qian
arXiv Machine Learning
Oct 9

Correlational Training of Morphological Neural Networks

The paper introduces a new weight update method for morphological neural networks based on the Multiplicative Weights Update scheme. It treats each morphological perceptron as a learning-from-experts problem in logarithmic space and uses a correlation-based reward that favors inputs aligned with the desired output change, even when gradient signals are weak. Experiments on fully connected layers and transformer components across nine benchmarks show that correlational training improves performance on eight tasks, up to 32.84 percentage points, and reduces run‑to‑run variability.

By Konstantinos Fotopoulos, Petros Maragos
arXiv AI
Oct 9

Imperative Interference: Social Register Shapes Instruction Topology in Large Language Models

The paper investigates how the imperative mood in system prompt instructions influences instruction topology across languages. Experiments on four languages and four models show that imperative instructions can invert cooperative and competitive interaction patterns, and that rewriting these instructions to declarative form reduces cross‑linguistic variance by 81%. The study suggests that language‑dependent social register shapes how large language models interpret instructions, implying that imperative‑mood directives may lead to language‑specific alignment during training and inference.

By Tony Mason
arXiv Machine Learning
Oct 9

Is Real-World Training Data Necessary for Generalist Graph Anomaly Detection?

The paper investigates whether synthetic data can replace real-world training data for generalist graph anomaly detection (GAD). It introduces AG-FORGE, a framework that automatically generates anomalous graphs, and shows that synthetic data can match real-world performance but does not surpass it due to method limitations. To overcome this, the authors propose TS-GGAD, a model that jointly learns topological and semantic anomaly cues with a curriculum learning strategy, achieving superior results on 14 real-world datasets when trained on AG-FORGE-generated data.

By Yujing Liu, Yixin Liu, Yue Tan, Xiaofeng Cao, Alan Wee-Chung Liew, Heng Tao Shen, Shirui Pan
arXiv Machine Learning
Oct 9

Just Weather Scoring: Efficient End-to-end Nowcasting with Distributional Diffusion

Just Weather Scoring (JWS) is a single‑stage, end‑to‑end diffusion model that forecasts precipitation directly in radar space, eliminating the need for separate compression or deterministic components. By combining Masked Asynchronous Diffusion with a scoring‑rule objective, JWS adapts diffusion training to high‑dimensional spatio‑temporal data and enables few‑step generation. On SEVIR and MeteoNet benchmarks, JWS achieves state‑of‑the‑art probabilistic forecasting while using fewer parameters and delivering over 17× faster inference than existing methods.

By Jannik Wiese, Johannes Schusterbauer, Tommaso Martorella, Bj\"orn Ommer
arXiv Machine Learning
Oct 9

DataSense-Bench: The First Step Toward an AI Scientist

DataSense-Bench is a new benchmark that tests whether advanced AI models can choose and rank training data subsets that will improve a small language model’s performance. The benchmark lets agents inspect data, run analysis code, and perform forward passes, but not train the model or access evaluation tasks. After agents select subsets, the base model is fine‑tuned on each subset and evaluated, measuring both the performance of the best subset and the accuracy of the agents’ rankings. Experiments show limited gains over random selection and inconsistent ranking ability across tasks, with agents often using similar data signals but interpreting their value differently.

By Yudi Zhang, Mingyu Cao, Lu Yin, Mykola Pechenizkiy, Shiwei Liu
arXiv AI
Oct 9

Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition

The paper introduces Adversarial Heuristic Learning (AHL), a framework that uses AI agents to refine game policies without changing model weights. It presents AAArena, a benchmark of 12 adversarial games and 1,920 human programs, where agents analyze replays, select opponents, and revise policies to maximize ranking within limited budgets. Experiments show that certain configurations achieve gold medals, while performance drops in games with complex rules, highlighting challenges in game understanding and long‑horizon policy development.

By Kaisen Yang, Qingle Liu, Kejin Wang, Yicheng Zhao, Jieming Li, Shenghan Zheng, Ruize Yang, Bojun Yang, Heng Gong, Xiang Gao, Lanyue Zhang, Kaiyu Zhong, Zhuo Liu, Shaoxuan Li, Chengxi Li, Yong Yan, Weixuan Zhang, Tianwei Luo, Situ Wang, Youjie Zheng, Sihan Zhao, Shengyuan Wang, Huan-ang Gao, Jiazheng Xu, Xiaohui Xie, Wentao Han, Hongning Wang
arXiv AI
Oct 9

Strategic Governance of AI Models in Earth Science

The article discusses how AI foundation models trained on weather and climate data are increasingly fine‑tuned for a wide range of Earth science tasks beyond traditional weather forecasting. It highlights that current evaluations rely mainly on benchmark skill metrics, which assess forecast accuracy but not the models’ representation of underlying physical processes, a gap that becomes critical under changing climate conditions. The authors propose five priorities for physical evaluation—covering training data, fine‑tuning, behavioral testing, mechanistic interpretability, and output validation—and recommend three actions for the next decade: open AI‑ready evaluation datasets, a shared physics‑based reporting standard, and a dedicated research program on model safety.

By Makoto Kelp, Amirhossein Arzani, Patricia Castellanos, Paul Griffiths, Ivan Higuera-Mendieta, Manuel Perez-Carrasco, Viral Shah, Patrick Obin Sturm, James Weber