Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computer Vision
Oct 28

Visual Autoregressive Generation with Next Token Imagination

The paper introduces Next Token Imagination (NTI), a technique for visual autoregressive generation that allows a model to predict beyond the ground-truth next token while staying within a defined visual distribution. By enabling the model to explore plausible but unseen visual realizations, NTI addresses the limitation of training only on observed data. Experiments on ImageNet demonstrate that NTI consistently improves image fidelity, approaching the performance of state‑of‑the‑art diffusion and autoregressive models with a smaller model and lower training cost.

By Ruihang Jiang, Zongyao Zhao, Zhaolin Wang, Yuanwei Liu, Xiatian Zhu
arXiv Computer Vision
Oct 28

uTHCD-C: A Corruption-Robustness Benchmark for Tamil Handwritten Character Recognition

The paper introduces uTHCD-C, a benchmark for assessing corruption robustness in Tamil handwritten character recognition (HCR). It defines ten corruption types—including a stroke‑thinning corruption—across five severity levels and evaluates nine convolutional and transformer models using cross‑validation and writer‑independent testing. The study reports mean Corruption Error (mCE) scores, showing ConvNeXt and Swin as most robust, while also comparing computational efficiency metrics.

By Preethi S, Meeradevi T, Arun Prakash A
arXiv Statistics ML
Oct 28

Data-driven Discovery of Lie Symmetries and First Integrals with Latent ODEs

The paper introduces LatentLieODE, a neural ODE framework that learns linear Lie symmetries and first integrals from time‑series data such as videos. It uses an autoencoder to obtain a low‑dimensional latent space, fits linear infinitesimal generators to a Lie‑bracket residual, and trains a latent ODE to reconstruct observations. The method successfully identifies symmetries and conserved quantities in six benchmark systems, achieving high correlation with ground‑truth invariants and recovering a non‑abelian σ(3) algebra on the sphere.

By Matt L. Wiemman, Peter Melchior
arXiv Statistics ML
Oct 28

Is Your LLM Relay Serving What It Claims? A low-cost and high-power test for LLM Relay Auditing

The paper introduces Distributional Fingerprint Auditing (DFA), a statistical method for auditing large language model (LLM) relay services. DFA uses only textual responses from suspect and trusted endpoints, repeatedly querying simple finite-choice probes to build categorical response distributions that serve as fingerprints. By comparing these fingerprints probe‑by‑probe and aggregating evidence with the Cauchy combination test, DFA produces an approximate endpoint‑level p‑value for audit decisions at a chosen significance level, and further supports model attribution and substitution ratio estimation through a likelihood‑based framework.

By Fangran Miao, Shengjie Niu, Jian Huang, Ting Li
arXiv Computer Vision
Oct 28

TriDifNet: A Unified Three-Task Diffusion Network for End-to-End RGB-T Perception

TriDifNet is a diffusion-based network that jointly performs registration, fusion, and detection for misaligned RGB-T pedestrian perception. It includes three modules—RegDM for aligning infrared to visible images, FusDM for generating a detection-oriented fused representation, and DetDM for iterative bounding‑box denoising. The system uses closed‑loop optimization and a Nash‑MTL refinement to balance the three tasks, achieving state‑of‑the‑art detection metrics on the KAIST multispectral benchmark.

By Qianchao He, Jing Liu, Yunhao Li, Xiaoqing Tian
arXiv Computer Vision
Oct 28

Zhengrong-Spatial-Aware 1.0: Advancing 3D Multimodal Intelligence

The paper introduces two end‑to‑end multimodal adaptation pipelines for 3D‑aware visual question answering on a custom spatial dataset. One pipeline fine‑tunes HiSpatial‑3B, adding metric‑scale 3D point maps to RGB inputs, while the other adapts Qwen3‑VL using memory‑efficient QLoRA supervised fine‑tuning. The engineering stack includes dataset conversion, image‑annotation alignment, monocular geometry preprocessing, instruction construction, mixed‑domain training, checkpoint management, batch inference, and automated evaluation, achieving significant accuracy gains on both the target spatial benchmark and a styled split.

By Yu Zhang, Xiaogang Wang, Yuhang Cheng, Jingrui Li, Wenbo Chen
arXiv Computer Vision
Oct 28

TopoField: Technical Report for ECCV 2026 SceneFun3D Challenge

The paper introduces TopoField, a deterministic decoder that bridges the gap between continuous semantic probability fields and discrete instance segmentation for functional parts in indoor point clouds. It aggregates multiple forward passes into a consensus field, decodes high‑confidence cores, constructs a pruned point graph, and applies a multi‑source graph‑geodesic partition to separate overlapping support regions. On the ECCV 2026 SceneFun3D Challenge, TopoField achieved second place with an AP of 28.286%, AP50 of 41.383%, and AP25 of 47.558%.

By Zhiwei Chen, Zhiheng Fu, Zixu Li, Mingyu Zhang, Jiale Huang, Yupeng Hu
arXiv Computer Vision
Oct 28

LEVIRDet: A Million-Scale 159-Category Dataset and Foundation Model for Universal Remote Sensing Object Detection

LEVIRDet-159 is the largest remote sensing object detection dataset, featuring 159 categories, 2.56 million bounding boxes, and 700k fine‑grained annotations across a multi‑level taxonomy. It surpasses existing datasets by roughly 7× more images, 6× more object instances, and 4× more categories. Leveraging this dataset, the authors propose LEVIRDetNet, a scale‑hierarchy‑aware foundation model that predicts ground sampling distance, modulates queries accordingly, and employs a hierarchy‑aware detection head, achieving state‑of‑the‑art performance on nine external benchmarks without target‑domain training or fine‑tuning.

By Qinzhe Yang, Dongyu Wang, Haohan Niu, Jia Xu, Zhenwei Shi, Zhengxia Zou
arXiv AI
Oct 28

The Trinity:Capability-Driven Recursive Self-Improvement for Multimodal Agents

The paper introduces The Trinity, a recursive self‑improvement framework for multimodal agents that couples world generation with harness evolution through three roles: Creator, Explorer, and Evolver. By shifting task organization from surface categories to shared capability requirements, it builds a hierarchical capability tree and a reusable harness tree, enabling efficient transfer learning across heterogeneous tasks. Experiments show the evolved harnesses improve mean performance by 34 points on 28 training benchmarks and 26 points on 18 unseen benchmarks, including two real‑world embodied tasks.

By Weidong Tang, Ruiheng Wang, Jingxuan Zhang, Hexin Wang, Haoyuan Chen, Zhengfeiyang Zhang, Yixing Ma, Zichen Jiang, Ziyu Guo, Haodong Li, Zhengbo Jiao, Zhiqing Cui, Xinxiang Yin, Shuo Liang, Zhenglin Wan, Hongzhan Lin, Wangbo Zhao Kaipeng Zhang, Pengfei Zhou, Odin Zhang, Yang You
arXiv AI
Oct 28

OA-RAG: Ontology-based Agentic RAG for Faithful LLM Reasoning

OA‑RAG is an agent that combines formal ontology reasoning with large language model (LLM) reasoning to produce more faithful answers. For each response it lists the ontology observations used as evidence and assigns a grounding level—entailed, supported, related, or unrelated—to indicate the strength of that evidence. On benchmarks from three large ontologies and six real‑world medical QA tasks, OA‑RAG outperforms other ontology‑fine‑tuned and popular RAG methods, especially on reasoning‑intensive questions, and its grounding levels correlate with higher accuracy.

By Hui Yang, Jiaoyan Chen, Yongsheng Gao
arXiv AI
Oct 28

Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure

The study investigates whether large language model (LLM) agents can distinguish between user‑initiated corrections and manipulative pressure when deciding to concede. Across seven models from four vendors, researchers found that while citations increased willingness to concede, they also reduced discrimination between genuine corrections and manipulation. Additionally, concession on answers did not predict policy violations in a customer‑support scenario, indicating that mere concession rates are insufficient for evaluating agent reliability.

By Nardine Basta, Venus Haghighi, Dali Kaafar, Amin Beheshti
arXiv AI
Oct 28

CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments

CogArena is a benchmark that evaluates multimodal AI agents on ten social and behavioral science experiments, requiring agents to interact via a web browser with the same visual stimuli and response interfaces used by human participants. The benchmark employs a three-level scoring system to assess task completion, accuracy, and the presence of characteristic behavioral signatures. Results from four multimodal models show that agents often fail to finish experiments and inconsistently reproduce human-like behavioral patterns.

By Kiant\'e Fernandez, Caitlin Chen, Jialu Zhou, Bartek Sadowski, Anthony C. Miceli, Ian Krajbich
arXiv AI
Oct 28

God Does Not Play Dice: Enforcing Batch Invariance in Agentic Kernel Generation

The paper introduces ool, a plug‑in for kernel generation agents that enforces batch invariance (BI) in GPU kernels used by large language model serving systems. By combining static dependency analysis with dynamic bitwise consistency checks, ool rejects kernels that are structurally risky or non‑invariant before performance benchmarking. Experiments on three agent frameworks show that ool raises the BI pass rate from 67.6 % to 91.9 % while preserving 91.8 % of the baseline speedup.

By Youwei Bao, Chengpeng Wang, Simin Chen
arXiv AI
Oct 28

OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

OpenGray is a new benchmark that evaluates radiotherapy planning agents on their decision-making rather than just dose outcomes. It extends the OpenKBP-Opt head-and-neck dataset with tasks that have known correct decisions, including contradictions with dose goals and a feasible control plan. Four systems—ranging from a rule-based script to full LLM agents—were tested on 30 patients, revealing that high decision accuracy can mask specific failures such as missed hard goals and inadequate stopping actions.

By Humza Nusrat, Kundan Thind
arXiv AI
Oct 28

RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport

RPSD introduces Retrieval-Privileged On-Policy Self-Distillation for sequential recommendation, addressing credit-blind optimization by using token-level self-distillation and collaborative-blind supervision by incorporating cross-user behavioral evidence via temporally aware optimal transport. The method grounds the teacher model in similar users’ interaction trajectories, selecting diverse complementary paths to provide structured privileged signals. Experiments on real-world datasets show that RPSD achieves state‑of‑the‑art performance.

By Dongyue Li, Zebang Cheng, Zhongyi Deng, Hongboxu, Yi Bin, Fei Ma
arXiv AI
Oct 28

Clearing the FoG: Training KG Agents to Retrieve the Evidence Behind Their Answers

The paper introduces FoG, a framework that trains knowledge graph (KG) agents to retrieve the evidence behind their answers. FoG combines evidence‑grounded synthesis, a KG interaction agent, and an evidence‑coverage process reward (ECPR) to ensure that every answer is supported by retrieved triples. Experiments on three KGQA benchmarks show that 3B–4B open‑source agents trained with FoG outperform larger baselines in both answer accuracy and evidence coverage, even with zero‑shot cross‑KG transfer.

By Zongyuan Sun, Shuai Zhao, Yingjun Du, Ke Shu, Ziyue Zhu, Haoran Luo
arXiv AI
Oct 28

HIDRA: Hierarchical Dual-Routing Attention for Replay-Free Lifelong Imitation Learning

HIDRA is a hierarchical dual-routing attention framework designed for replay‑free lifelong imitation learning in robotics. It separates instruction‑level expert retrieval from context‑dependent refinement and adds key‑level geometric regularization to keep tasks aligned yet distinct, thereby reducing routing interference. Experiments on LIBERO benchmarks show that HIDRA outperforms both replay‑free and replay‑based baselines, improving AUC, forward transfer, and maintaining low forgetting, especially under high interference and language variation.

By Fanqi Yu, Matteo Tiezzi, Cigdem Beyan, Tommaso Apicella, Vittorio Murino
arXiv AI
Oct 28

Robust LLM Data Selection via Content-Invariant Gradient Attribution

The paper introduces Content-Invariant Gradient Attribution (CIGA), a method for selecting training data for large language models that mitigates surface-form bias in gradient-based attribution. CIGA identifies layers whose gradients are both influential and stable under semantics-preserving perturbations, then fuses these signals with semantic representations to produce robust data rankings. Experiments show that CIGA yields more stable rankings, maintains downstream performance, and reduces attribution cost.

By Jia Gu, Liang Pang, Huawei Shen
arXiv AI
Oct 28

PerceptRL: Making Visual Perception Rewardable in MLLMs from Answers to Interactions

PerceptRL enhances visual perception in multimodal large language models by employing reinforcement learning that transforms dense image captions into verified QA pairs with rule‑based rewards. It uses a multi‑agent pipeline to link correlated errors, combine unrelated errors, and target high‑error domains, while a specialized multi‑turn inspection process corrects spatial inaccuracies. Across seven benchmarks, PerceptRL improves average accuracy to 71.6%, outperforming both the base model with tools and the leading evaluated VLM by significant margins.

By Tongkun Guan, Zhibo Yang, Mingkun Yang, Haocheng Wang, Wei Shen, Xiaokang Yang
Hugging Face Trending Papers
Oct 27

Timestral

Timestral is a family of four time‑series foundation models (5M, 28M, 123M, and 209M parameters) that achieve state‑of‑the‑art results on GIFT‑Eval, TIME, and fev‑bench, with the 209M and 123M variants topping every metric. The models use Decoupled Horizon Forecasting to separate prediction tokens from context, a covariate framework for known‑future inputs, and a two‑stage training pipeline that first pre‑trains on synthetic data and then post‑trains on a mix of real and synthetic data with extended context. All four models and inference code are publicly available on Hugging Face and GitHub.