Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 28

Factorized axis convolutional gated recurrent unit with dynamic adaptive pooling for remaining useful life prediction of rolling bearings

The paper presents a factorized-axis convolutional gated recurrent unit (GRU) with dynamic adaptive pooling (DAP) for predicting the remaining useful life (RUL) of rolling bearings from time‑frequency representations (TFRs). It introduces multiscale anisotropic convolution, a dual‑axis convolution block attention module, and Monte Carlo dropout for uncertainty estimation, addressing the directional structure challenges in TFRs. Experiments on two public bearing datasets show that this approach outperforms existing RUL prediction methods and that the factorized axis design and adaptive pooling contribute to lower mean errors.

By Hanbyeol Park, Jungho Choo, Hyerim Bae
arXiv AI
Sep 28

Cartograph: Federated Tool Discovery with Operator-Attested Retrieval for AI Agents

Cartograph is a federated Model Context Protocol (MCP) proxy that reduces AI agent tool discovery from linear catalog traversal to progressive disclosure, exposing only a few proxy tools instead of all definitions. It uses operator-attested capability cards, a three-layer confusable-cluster analysis called Rift, and a two-stage retrieval process to rank servers before tools. In a 22-server, 374-tool deployment, Cartograph achieves higher recall (R@5 = 0.816 vs. 0.592) and drastically fewer tokens (475 vs. 42,450) for discovery exchanges, with minimal latency overhead.

By Justice Owusu Agyemang, Michael Agyare, Kwame Opuni-Boachie Obour Agyekum, Kwame Agyeman-Prempeh Agyekum, Francisca Adoma Acheampong, Jerry John Kponyo
arXiv Computer Vision
Sep 28

MM-VeriAgent: Learning to Use Extensive Tools to Verify Multimodal Misinformation with Reinforcement Learning

MM-VeriAgent is a reinforcement‑learning framework that learns to verify multimodal misinformation by leveraging a specialized toolkit called MM-VeriTools. The toolkit encapsulates the strongest models for textual, visual, and cross‑modal forgery analysis as callable tools with a unified interface. To improve training efficiency, the authors introduce a Tool‑Execution Cache that pre‑executes candidate tool calls and reuses cached outputs, resulting in substantial accuracy gains on MMFakeBench and reduced online tool executions during training.

By Peipei Li, Shuhan Xia, Shengyang Liu, Zekun Li, Ran He
arXiv AI
Sep 28

Spackle: Completing Large View Single Image NVS with Adaptive Gaussians

Spackle is a lightweight residual learning framework designed to improve large-view single-image novel view synthesis (NVS) by mitigating capacity competition in hybrid decoupled systems that combine 3D Gaussian Splatting (3DGS) and diffusion models. It operates in three stages: predicting base 3DGS attributes, automatically identifying poorly reconstructed regions, and learning a residual 3DGS focused on those areas. During inference, Spackle merges the baseline and augmented Gaussians to produce high-fidelity novel views, achieving state‑of‑the‑art performance on large-view-deviation cases.

By Xuanzhi Liu, Yuhe Zhou, Xinyi Wu, Zhenyao Wu, Jinghao Chen, Ruize Han, Song Wang
arXiv AI
Sep 28

From Shortcut Learning to Discrete Neural Insertion Sort

The paper investigates how neural networks can learn to execute the insertion‑sort algorithm faithfully. Analysis of the CLRS30 baseline reveals that the model often learns shortcuts, producing correct final outputs without following the intended intermediate steps. To address this, the authors propose Discrete Neural Insertion Sort, which represents the sequence as a chain, separates exchanges from control transitions, and discretizes node states after each step. Trained on sequences of length 16, the model attains perfect accuracy on lengths 64 and 128, but ablation studies show that discretization and graph structure alone are insufficient without additional supervision of the global inner‑loop state.

By Konstantinos Mylonas, Thrasyvoulos Spyropoulos
arXiv AI
Sep 28

FedHisto-PAST: Parameter-Efficient Stain-Aware Federated Learning for Cross-Site Lung Histopathology Classification

FedHisto-PAST v2 is a parameter‑efficient, stain‑aware federated learning framework for cross‑site lung histopathology classification, combining a frozen HIBOU‑B foundation model with techniques such as paired‑view prediction, feature consistency, prototype learning, and adaptive aggregation. In a five‑client, non‑IID simulation and an exploratory LungHist700 cohort, the method achieved a Macro‑F1 of 0.7286 and a balanced accuracy of 0.7305, with the prediction‑level consistency component providing the most clear independent benefit. The framework updated only about 1.25% of the model parameters, demonstrating efficient adaptation while acknowledging limitations in privacy guarantees and clinical validation.

By Muhammad Muhtasim Shahriar, M. M. Golam Hafiz, Saad Aloteibi, Mohammad Ali Moni
arXiv AI
Sep 28

Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design

The paper examines how evaluation design can cause significant fluctuations in the benchmark results of reasoning models, particularly the Deepseek‑R1‑Distill series. It shows that subtle changes in evaluation conditions lead to large variations in reported performance, a phenomenon also seen in other open‑source models fine‑tuned from Deepseek‑R1‑Distill and in the QwQ‑32B model. The authors call for a more rigorous evaluation paradigm and provide empirical assessments of the Deepseek‑R1‑Distill models.

By Yongfu Zhu, Lin Sun, Jinzhu Wu, Weihong Lin, Xiaoqi Jian, Guangxiang Zhao, Change Jia, Linglin Zhang, Sai-er Hu, Yuhan Wu, Xiangzheng Zhang
arXiv AI
Sep 28

Attribution Bias in Large Language Models

The paper introduces AttriBench, a benchmark dataset that balances author fame and demographics to study quote attribution in large language models (LLMs). Using AttriBench, the authors evaluate 11 popular LLMs and find that accurate attribution remains difficult, with significant disparities across race, gender, and intersectional groups. They also identify a new failure mode—suppression—where models omit attribution entirely, which is unevenly distributed across demographics and not reflected by standard accuracy metrics.

By Eliza Berman, Bella Chang, Daniel B. Neill, Emily Black
arXiv AI
Sep 28

ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

ReasonAudio is a new benchmark designed to evaluate reasoning capabilities in text‑audio retrieval, addressing the gap left by existing semantic‑matching focused datasets. It tests four logical abilities—negation, temporal order, sound co‑occurrence, and sound duration—across five synthetic subtasks (1,000 queries over 10,000 composite clips) and one natural subtask (100 queries over 1,000 real‑world clips). Evaluation of 11 state‑of‑the‑art systems shows significant limitations, with the best model, OmniEmbed‑7B, scoring only 20.7 overall and 53.8% in a controlled setting, compared to 70.6% for its generative backbone and 95.6% for humans.

By Honglei Zhang, Yuting Chen, Chenpeng Hu, Pengfei Zhou, Siyue Zhang, Yilei Shi
arXiv AI
Sep 28

SLMFix: Leveraging Small Language Models for Domain Specific Language Error Fixing with Reinforcement Learning

SLMFix is a code‑generation pipeline that uses a small language model fine‑tuned with reinforcement learning to correct syntactic errors in programs produced by large language models for domain‑specific languages. The approach relies on interpreter feedback to guide the error‑fixing process. Experiments show that SLMFix improves validator pass rates by 40% on low‑resource programming languages and removes over 50% of syntactic errors on high‑resource DSLs, outperforming supervised fine‑tuning even for 7B models.

By David Jiahao Fu, Aryan Gupta, Aaron Councilman, Yu-Xiong Wang, Vikram Adve
arXiv AI
Sep 28

CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentrating

The paper introduces CaC, a coarse‑to‑fine anomaly reward model that uses Vision‑Language Models to first scan globally for anomalous time windows, then ground anomalies spatially, and finally reason with structured spatiotemporal Chain‑of‑Thought. It builds the first large‑scale generated video anomaly dataset with detailed annotations and trains the model through a three‑stage progressive paradigm, including reinforcement learning with Group Relative Policy Optimization. Experiments show CaC improves fine‑grained anomaly detection by 25.7% and reduces generated‑video anomalies by 11.7% while enhancing overall video quality.

By Jiyuan Wang, Huan Ouyang, Jiuzhou Lin, Chunyu Lin, Dewen Fan, Boheng Zhang, Haonan Fan, Honglie Wang, Yiyang Fan, Zhenlong Yuan, Zijun Li, Yongrui Heng, Guosheng Lin, Fan Yang
arXiv AI
Sep 28

Smooth Piecewise Cutting for Neural Operator to Handle Discontinuities and Sharp Transitions

The paper introduces Cut-DeepONet, a two‑stage training framework that explicitly models discontinuities in partial differential equations by partitioning the domain into smooth subregions and representing discontinuities as boundaries in a higher‑dimensional space. This lifting strategy aligns the operator learning task with neural network inductive biases, allowing the model to avoid directly approximating discontinuities. Experiments on benchmark PDEs demonstrate that Cut‑DeepONet outperforms state‑of‑the‑art methods, even with low‑resolution data, while using fewer trainable parameters.

By Ha Dang, Sebastian Schmidt, Juergen Hesser
arXiv AI
Sep 28

Prompt-Based Continual Compositional Zero-Shot Learning

The paper introduces PromptCCZSL, a framework that enables vision‑language models to continually learn new attributes, objects, and their unique compositions while avoiding forgetting. It uses a frozen VLM backbone with prompt‑based techniques, recency‑weighted multi‑teacher distillation, and several loss functions (CAL, OPL, IDL) to maintain prior knowledge and promote diverse, distinct embeddings. Experiments on UT‑Zappos and C‑GQA show significant performance gains over existing VLM‑based and non‑VLM baselines, establishing a new benchmark for continual compositional zero‑shot learning.

By Sauda Maryam, Sara Nadeem, Faisal Qureshi, Mohsen Ali
arXiv Computer Vision
Sep 28

LensDesigner: A Self-Improving Agent for Optical Lens Design

LensDesigner is an autonomous agent framework that emulates expert opticians to tackle the complex, non‑convex problem of optical lens design. It uses a large lens library (LensLib100K) and optics‑aware retrieval to provide valid structural seeds, then iteratively improves through a curriculum agent that learns and reuses design heuristics. The system is evaluated on LensArena, a benchmark of 120 diverse optical tasks, where it outperforms existing baseline algorithms in success rate and optimization efficiency.

By Lei Sun, Haoran Liang, Dannong Xu, Yao Gao, Yuyu Geng, Jinjin Gu, Kaiwei Wang, Danda Pani Paudel, Luc Van Gool
arXiv Computer Vision
Sep 28

MDSkin-Net: Multi-Task Skin Lesion Analysis Driven by Pattern Analysis Priors and Spatial Alignment Regularization

MDSkin-Net is a multi‑task skin lesion analysis framework that integrates Pattern Analysis priors into a hybrid CNN‑Transformer architecture. It introduces a Pattern Analysis‑Guided Attention Module (PAGAM) with improved Efficient Channel Attention, Multi‑Scale Spatial Attention, and Biased Asymmetry Attention, along with a multi‑scale spatial alignment regularization that uses segmentation masks as soft supervision. Trained only on the ISIC 2017 training split, the model achieves high segmentation and classification performance on multiple datasets, demonstrating strong zero‑shot generalization across different cohorts.

By Yijian Li, Saad Bedros, Paul Bigliardi, Mei Bigliardi Qi, Vassilios Morellas, Nikolaos Papanikolopoulos
arXiv Computer Vision
Sep 28

STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence

STORM-Bench is a new benchmark for online video question answering that evaluates models’ ability to track state transitions and selectively abstain when visual evidence is insufficient. It contains 5,736 questions across 630 short, change‑dense episodes in five egocentric domains and two simulation subsets, with questions stratified by change intensity and answerability. The benchmark introduces STORM‑BR, a harmonic metric that reveals abstention failures and overconfidence on uncertain queries, showing that traditional accuracy masks gaps in epistemic reliability and state tracking.

By Siru Zhong, Shenghan Tan, Rihong Yan, Xiaohui Lv, Yuzheng Zhuang, Shuai Tao, Wulong Liu, Haohuan Fu, Yuxuan Liang
arXiv Computer Vision
Sep 28

Forensic Twins: Self-Supervised Residual Learning for AI-Generated Image Forensics

Forensic Twins introduces a Self‑Supervised Residual Learning (SSRL) framework that trains on real images only, using a frozen forensic residual extractor to generate two disjoint crops per image. The pretext task suppresses semantic content, focusing the model on the stationary fingerprint of the image acquisition pipeline, and achieves 56.61% accuracy in attributing AI‑generator sources, outperforming prior zero‑shot methods. When combined with an offline Gaussian Mixture Model, the approach reaches 97.99% AUC across 27 unseen AI generators, including GANs, diffusion models, and commercial systems.

By Javier Mu\~noz-Haro, Ruben Tolosana, Ruben Vera-Rodriguez, Aythami Morales, Julian Fierrez
arXiv Computer Vision
Sep 28

Structured Reasoning Agentic Framework for Interpretable Critical View of Safety Assessment

The paper introduces ReasonCVS, a structured reasoning framework for assessing the Critical View of Safety in laparoscopic cholecystectomy. It uses a Vision‑Language Model to build an Anatomical Scene Graph Abstraction and a Large Language Model–based Rationale‑Aware Reasoning Agent to verify sub‑criteria, producing a final verdict with traceable clinical rationale. Experiments on the Endoscapes‑CVS201 benchmark show ReasonCVS outperforms existing methods with a 68.1% mAP while offering interpretable, criterion‑level explanations.

By Qing Xu, Yuxiang Luo, Zhen Chen
arXiv Computer Vision
Sep 28

InternW0-$\Delta$: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data

InternW0-Δ is a unified World Action Model that integrates pretrained visual dynamics, scene semantics, 4D geometry, and motion priors within a Mixture-of-Transformers framework to generate robot actions. It leverages a frozen VLM for semantic guidance, a 4D foundation model for geometric priors, and introduces Causal Imprint to learn future-relevant scene changes without future-video rollout. The model is pretrained on a newly curated 20K‑hour heterogeneous corpus of robot and human demonstrations, achieving superior performance on simulation benchmarks and real‑robot platforms.

By Xingyu Miao, Zizun Li, Baole Fang, Kaiwen Song, Tenghui Wang, Hanxue Zhang, Yating Wang, Xudong Li, Yuping He, Xueyuan Wei, Chao Gao, Xijie Yang, Yingxiang Xu, Kerui Ren, Wenqi Guo, Jianjun Zhou, Xinzhe Wang, Weiguang Zhao, Ni Yang, Zetao Cai, Yufei Xue, Hengjie Li, Zeyu He, Yuanzhen Zhou, Rong Fu, Jianyang Zhang, Siwei Cui, Fuxian Huang, Yunsong Zhou, Xing Gao, Yifei Yao, Qiaojun Yu, Kailin Li, Ming Zhou, Mu Huang, Xinyue Li, Wenze Cui, Bingqi Jiang, Xueyue Zhu, Junting Dong, Haoyu Guo, Tao Lu, Mulin Yu, Bowen Zhou, Bin Zhao, Tianfan Xue, Weinan Zhang, Chunhua Shen
arXiv Statistics ML
Sep 28

Brenier Meets Adversarial Training: Optimal Transport Geometry for Robust Learning

The paper introduces a penalized distributionally robust optimization framework that allows an adversary to choose any distribution while incurring a Wasserstein penalty for deviating from the empirical distribution. It shows that the adversary’s problem can be reformulated as optimizing transport maps that push empirical samples to adversarial ones, proving that optimal maps are cyclically monotone. The authors argue that standard per-sample adversarial training violates this property and propose two remedies—multi-start particle ascent and input-convex neural network parameterization—to enforce cyclical monotonicity, demonstrating improved robustness and generalization in experiments on regression, image classification, and control tasks.

By Alireza Abdollahpoorrostam, Ehsan Sharifian, Buse \c{S}en, Marco Cuturi, Daniel Kuhn