Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computer Vision
1d ago

Towards Visual Query Segmentation in the Wild

arXiv:2603.08898v3 Announce Type: replace Abstract: In this paper, we introduce visual query segmentation (VQS), a new paradigm of visual query localization (VQL) that aims to segment all pixel-level...

By Bing Fan, Minghao Li, Hanzhi Zhang, Shaohua Dong, Naga Prudhvi Mareedu, Xiaoqiong Liu, Weishi Shi, Yunhe Feng, Yan Huang, Heng Fan
arXiv Computer Vision
1d ago

ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning

arXiv:2605.20342v3 Announce Type: replace Abstract: Training large multimodal models (LMMs) via reinforcement learning (RL) to natively invoke video-processing tools (e.g., cropping) has become a pro...

By Zuhao Yang, Kaichen Zhang, Sudong Wang, Keming Wu, Zhongyu Yang, Bo Li, Xiaojuan Qi, Shijian Lu, Xingxuan Li, Lidong Bing
arXiv Machine Learning
1d ago

Role-based Effectiveness of an Epistemic Uncertainty Estimate in Training Models for Semantic Segmentation of 3D LiDAR Point Clouds

The paper studies how epistemic uncertainty estimates can be used in different ways—either as an additional input feature or as a per‑sample confidence weight in the loss function—to improve 3D LiDAR point cloud segmentation. Experiments on four large‑scale LiDAR benchmarks (ISPRS Vaihingen, AHN3, Paris‑Lille‑3D, and Toronto‑3D) show that uncertainty‑aware models outperform a baseline ANN in mean Intersection over Union, recover minority classes, and better align prediction error with estimated uncertainty. The results suggest that uncertainty can serve multiple roles in the segmentation pipeline beyond merely boosting accuracy.

By Krishnakumar N, Jaya Sreevalsan-Nair
arXiv Machine Learning
1d ago

Balancing hierarchical representation and quantization: A performance trade-off in text embeddings

The study benchmarks how hierarchical text embeddings and quantization affect performance across 38 tasks using four transformer models. It evaluates 168 configurations combining 1D/2D hierarchical representations with 32‑bit, 8‑bit, and binary quantization, reporting storage and evaluation time separately. Results show int8 quantization is nearly as effective as float32, binary quantization degrades performance, and dimensionality reduction paired with int8 can cut storage by 32× with minimal loss.

By Kai Ponel, Marco Polignano, Erasmo Purificato, Aylin Julia Temizkan, Marco Levantesi, Giovanni Semeraro, Ernesto William De Luca
arXiv Machine Learning
1d ago

Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers

The paper introduces fast and stable methods for triangular matrix inversion used in Delta-Rule linear transformers, a key component of efficient long‑context models such as Qwen3.5/3.6 and RWKV‑7. It systematically compares direct and iterative inversion algorithms, emphasizing numerical stability, computational complexity, and hardware efficiency, and validates the approaches with rigorous experiments in low‑precision settings. Benchmarks on NPUs show up to a 4.3× speed‑up over current SGLang implementations, improving layer performance while preserving model accuracy.

By Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros, Mirko De Vita, Filip Skogh, Anastasios Zouzias, Jiawei Zhuang
arXiv Computer Vision
1d ago

Lesion Overlap Ranks PET Segmentation Models but Does Not Certify Patient-Level Tumor Burden

The study evaluates whether overlap metrics like Dice can reliably indicate metabolic tumor volume (MTV) estimation accuracy in PET segmentation models. Across five FDG PET/CT models, higher Dice scores correlated with lower relative MTV error on average, but at the patient level, 16.8% of cases fell on opposite sides of clinically relevant thresholds. Additionally, many models missed lesions, showing that model ranking alone does not guarantee reliable individual MTV estimates.

By Tushar Kataria, Gerald Sabin, Ponnuswamy Sadayappan, Shireen Y. Elhabian
arXiv Computer Vision
1d ago

FedXCrop: Privacy-Preserving Crop Disease Detection via Federated Learning with Post-Aggregation Explainability under Non-IID Farm Distributions

FedXCrop is a federated learning pipeline for crop disease classification that preserves farmer privacy while providing post‑aggregation explainability. The system uses a MobileNetV2 model and achieves near‑centralized accuracy (98.81%–99.38%) across five clients, even under non‑IID label skew. Grad‑CAM and SmoothGrad explanations are quantitatively evaluated, showing that federated models largely retain explainability characteristics of centralized models, though agreement decreases with increased heterogeneity.

By Kriti Saini, Manikandan P
arXiv Computer Vision
1d ago

Conditional Diffusion-Based Generation of T2-Weighted Prostate MRI for Clinically Significant Prostate Cancer Classification

The study presents a conditional diffusion-based generative model that augments T2-weighted prostate MRI data with clinically significant prostate cancer (csPCa) and transition zone (TZ) lesions. By extending MAISI‑v2 with csPCa class, zonal, and lesion‑mask conditioning and fine‑tuning on 1,303 patients, the model achieves a substantial reduction in Fréchet inception distance (FID) from 13.28 to 2.04. Synthetic augmentation improves csPCa classification AUC across five simulated training scenarios, though the benefit does not significantly transfer to a cross‑institutional dataset.

By Xavier Rafael-Palou (Quantitative Imaging Biomarkers in Medicine), Alejandro Vergara-Richart (Quantitative Imaging Biomarkers in Medicine, Universitat Polit\`ecnica de Val\`encia, Val\`encia, Spain), Ignacio Iborra Roncales (Quantitative Imaging Biomarkers in Medicine), Carmen Prieto-de-la-Lastra (Quantitative Imaging Biomarkers in Medicine), Andreu Antolin-Redondo (Department of Radiology, Institut de Diagn\`ostic per la Imatge, Department of Surgery, Universitat Aut\`onoma de Barcelona, Bellaterra, Spain), Juan Morote (Department of Surgery, Universitat Aut\`onoma de Barcelona, Bellaterra, Spain), Ana Jimenez-Pastor (Quantitative Imaging Biomarkers in Medicine)
arXiv AI
1d ago

AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory

AGORA is an item‑response‑theory aggregator that improves the reliability of large language model judges by learning each judge’s sensitivity and rating thresholds from all items jointly, without needing human labels. It places every item on a latent quality scale, allowing judge reliability to be inferred from the data itself. Across three benchmarks, AGORA matches or surpasses the Dawid‑Skene method, especially when evaluation sets are small or judge pools are uneven.

By Junhyung Ahn
arXiv AI
1d ago

ModalSearcher: Weaving Visual Evidence into Autonomous Cross-Modal Deep Search

ModalSearcher introduces an autonomous cross‑modal search framework that lets an agent dynamically choose among direct reasoning, text‑to‑text search, text‑to‑image search, and reverse‑image search, actively incorporating visual evidence as needed. The system builds an interleaved multimodal knowledge graph via a scalable pipeline that samples diverse paths and uses backward question construction to prevent shortcut reasoning. Trained with supervised fine‑tuning and reinforcement learning, ModalSearcher‑8B outperforms Qwen3‑VL‑8B by 16.2 accuracy points across five multimodal knowledge‑intensive QA benchmarks and shows consistent gains on active visual‑search tasks.

By Zhihan Yin, Ruyi Li, Chengyu Fang, Yifeng Yao, Nonghai Zhang, Pengfei Xiong, Terry Zhang, Huishuai Zhang, Dongyan Zhao
arXiv AI
1d ago

D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding

The paper introduces D3D, a decoupled diffusion drafting method that eliminates the need for target-model hidden states during speculative decoding. By combining blockwise denoising with auxiliary autoregressive supervision, D3D can draft successor blocks asynchronously, achieving significant speedups—up to 7.48× on GSM8K with Qwen3-32B. The authors also prune layers to maintain performance while reducing computational cost, demonstrating over 99% of mean acceptance length after removing five of 36 layers.

By Baoyou Chen, Ruiqiao Mei, Guancong Lin, Tao Song, Weihao Yuan, Qun Liu, Siyu Zhu
arXiv AI
1d ago

Time-Division Multiplexing: The Operational Rhythm and Temporal Organization Layer of Digital Life

The paper proposes a temporal organization layer for continual agents, introducing phase‑tagged micro‑epochs, a duty cycle, and nested clocks to separate exploration from commitment. It presents deterministic and stochastic guarantees for duty‑cycle updates, defines busy/idle execution contracts, and evaluates the design through synthetic experiments and a Qwen3‑8B prototype. The results demonstrate a testable scheduling design and clarify its conditions of validity.

By Haibin Tong, Luming Guo, Nana Cui, Jiang Yu
arXiv AI
1d ago

ConceptARC at Scale: Studying Conceptual Understanding in Reasoning Models

ConceptARC at Scale introduces a generator that produces thousands of ARC-style tasks focused on specific spatial and semantic concepts, expanding the original benchmark by tenfold. The new dataset, ConceptARC-100, contains 1,600 manually reviewed tasks across 16 concepts, each with 100 tasks. Evaluation of large reasoning models on this dataset reveals that they reliably solve certain concepts, struggle with others, and develop internal representations that cluster by transformation operations such as recoloring, removal, or translation.

By Philipp Mondorf, Adrien Deli\`ege, Solim LeGris, Claas Beger, Ryan Yi, Barbara Plank, Brenden M. Lake, Todd M. Gureckis
arXiv AI
1d ago

Decision Models as Scalable Economic Surrogates that Capture Human Variation

Decision Models (DMs) are a new class of AI models that provide fast responses and output a probability distribution over a fixed set of options. In a first evaluation, 10 DMs were compared with 8 Large Language Models (LLMs) across three economic settings—a household spending survey, classic economic games, and a labour‑market field experiment. Frontier DMs, those scoring highest on general decision‑making benchmarks, reproduced human response variation more closely than LLMs in all settings, matched LLMs on modal responses in the spending survey, and were roughly ten times faster and over a hundred times cheaper than the fastest LLM.

By Maxime Saxena, Fabio Caccioli, R. Maria del Rio-Chanona
arXiv AI
1d ago

BAM: Bivariate Additive Models for Interpretable Tabular Prediction

Bivariate Additive Models (BAMs) are a new class of Generalized Additive Models with pairwise interactions that constrain each shape function to a simple asymmetric bivariate Gaussian form, producing unimodal contour maps that are easier to interpret. The model automatically selects feature pairs during training and achieves predictive accuracy comparable to standard GAMs while approaching the performance of more complex GA$^2$Ms on six tabular benchmarks. Experiments also illustrate the trade‑off between accuracy and the number of components that need to be inspected.

By Atsushi Koike
arXiv AI
1d ago

Intra-Architecture Source Attribution for Stable Diffusion: A New Task with Benchmark Dataset and Baseline

The paper introduces a new task, Intra-Architecture Source Attribution for Stable Diffusion (IntrAA‑SD), which focuses on distinguishing between fine‑tuned versions of the same Stable Diffusion XL architecture. It presents a 140K‑image benchmark dataset with in‑domain and out‑of‑distribution prompts, and proposes a baseline method using a style‑conditioned Mixture‑of‑Experts network that leverages a frozen CLIP backbone and a LoRA‑adapted DINOv2 network. The authors evaluate their approach on attribution accuracy and conduct ablation studies, providing code and data publicly on GitHub.

By Ashutosh Dayal, Mallika Garg, Debashis Sen