Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computation and Language
Sep 23

Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs

Flash-dLLM is a training‑free inference acceleration framework that improves the speed and memory efficiency of Diffusion Large Language Models (dLLMs). It tackles GPU memory I/O bottlenecks by introducing an I/O‑aware fused KV‑cache kernel and then employs a draft‑and‑verify decoding strategy that uses the dLLM itself as both drafter and verifier. Experiments on mathematical reasoning and code‑generation tasks show Flash‑dLLM outperforms existing acceleration methods, achieving up to 11.0× speedups over the Elastic‑Cache baseline.

By Quan Nguyen-Tri, Mukul Ranjan, Zhiqiang Shen
arXiv Computation and Language
Sep 23

From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health

The article surveys how large language models (LLMs) are being applied to mental health, outlining a three‑phase evolution: Phase I uses LLMs as passive information tools and pattern recognizers for assessment; Phase II employs them as empathetic conversationalists for stateless, in‑the‑moment interactions; Phase III aims to create longitudinal, personalized companions that act as stateful cognitive agents. It systematically reviews core technologies, agent architectures (Profile, Memory, Reasoning, Planning), and the datasets and benchmarks that support this progression, offering a coherent narrative and roadmap for future research. The survey also provides a curated resource list at https://github.com/Emo-gml/Awesome-Mental-Health-LLMs.

By He Hu, Yucheng Zhou, Qianning Wang, Yingjian Zou, Chiyuan Ma, Juzheng Si, Jianzhuang Liu, Zitong Yu, Laizhong Cui, Fei Ma, Qi Tian
arXiv Computation and Language
Sep 23

WatchPoint: Executable User Feedback for Real-World Agentic Web Development

WatchPoint is a simulated‑user system that generates and runs diagnostic scripts against a live web application, producing structured observations to guide coding model retries. Unlike prior methods that rely on screenshots or non‑executable metrics, WatchPoint operates on Web‑Bench—a benchmark of 50 multi‑file web projects with 1,000 sequential tasks verified by deterministic end‑to‑end tests. It recovers 57.6% of diagnosed tasks, matching a human tester’s 54.5% recovery rate, and identifies when such simulated feedback is beneficial or should be withheld.

By Guanqun Yang, Wei Yang, Xueqing Liu
arXiv Computation and Language
Sep 23

CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding

CausalEmbed is an auto‑regressive method for generating compact multi‑vector embeddings in visual document retrieval. By using iterative margin loss during contrastive training, it reduces the number of visual tokens needed by 30‑155× while keeping performance competitive across different backbones and benchmarks. The approach offers efficient training, scalable test‑time performance, and a flexible scaling strategy for multi‑vector representations.

By Jiahao Huo, Yu Huang, Yibo Yan, Ye Pan, Kening Zheng, Wei-Chieh Huang, Yi Cao, Mingdong Ou, Philip S. Yu, Xuming Hu
arXiv Computation and Language
Sep 23

Calibrated Confidence Expression for Radiology Report Generation

The paper introduces ConRad, a reinforcement learning framework that fine‑tunes large vision‑language models to generate calibrated verbalized confidence estimates for radiology reports. ConRad offers both a single report‑level confidence score and a sentence‑level variant, trained with the GRPO algorithm and logarithmic scoring rewards to encourage truthful self‑assessment. Experiments show significant calibration improvements over existing methods, and clinical evaluation indicates that report‑level scores align well with clinicians’ judgments, enabling targeted review of low‑confidence statements.

By David Bani-Harouni, Chantal Pellegrini, Julian L\"uers, Su Hwan Kim, Markus Baalmann, Benedikt Wiestler, Rickmer Braren, Nassir Navab, Matthias Keicher
arXiv Computation and Language
Sep 23

Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding

Hy‑MultiTurn is a Chinese benchmark designed to evaluate deep multi‑turn dialogue understanding over long interactions. It introduces six controlled evaluation modes—constraint memory, precise execution, constraint synthesis, object localization, action suppression, and reference resolution—across 209 tasks ranging from 12 to 76 turns, incorporating dialogue length, irrelevant distractions, and colloquial phrasing. Testing 22 state‑of‑the‑art models shows the benchmark is highly challenging, with even the best model meeting all criteria only 41.1% of the time and no model excelling in every mode.

By Eileen Ye, Jiawen Tao, Yaoming Li, Chenxu Liu, Wenhan Yu, Yaxin Fan, Xiaokun Yuan, Mengzhou Wu, Yanbing Jiang, Maxm Pan
arXiv Computer Vision
Sep 23

MirrorDistill: Illumination-Aware Latent Distillation for Efficient Low-Light Restoration

MirrorDistill introduces an illumination‑aware latent distillation framework for low‑light image enhancement. It trains a lightweight student encoder‑decoder by aligning its intermediate features with clean‑domain targets generated by a teacher decoder, using feature mirroring and illumination‑aware weighting to emphasize underexposed regions. The method achieves state‑of‑the‑art performance on the LOL‑v2‑Real benchmark while maintaining the lowest computational complexity, and the code is released as open source.

By Farida Mohsen, Tala Zaim, Nurul Izni Rusli, Ali Al-Zawqari, Ali Safa, Samir Brahim Belhaouari
arXiv Computer Vision
Sep 23

Combinatorial Network-Based Manifold Topological Deep Learning for Image Analysis

Combinatorial Network-Based Manifold Topological Deep Learning (CNMTDL) is a new framework that represents medical images as discrete manifolds and decomposes them into three Hodge components. Features from these components are concatenated and fed into a combinatorial complex architecture, enabling higher‑order message passing between 0‑cells and 2‑cells via attention‑based blocks. CNMTDL was evaluated on six 2D and 3D datasets from the MedMNIST v2 benchmark, showing improved performance for medical image analysis.

By Alice Wachira, Xiang Liu, Zhe Su, Yiying Tong, Ge Wang, Guo-Wei Wei
arXiv Computer Vision
Sep 23

RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models

RGSQ introduces a Riemannian geometry‑aware post‑training quantization method for large vision‑language models, treating quantization as a reconstruction problem under a Fisher‑Riemannian metric. It identifies modality‑specific sensitive directions via manifold mappings and applies geometry‑aligned rotations and whitening to steer low‑bit perturbations toward loss‑insensitive axes. Experiments on diverse VLM benchmarks show RGSQ delivers the best accuracy and stability in extremely low‑bit settings, outperforming existing VLM‑aware baselines by up to 5.9% and single‑modality methods by up to 8.6%.

By Zhiping Wu, Dongdong Ren, Yangchengyu Zhou, Zhengjie Zhang, Wenbin Li, Hongbing Pan, Yang Gao
arXiv Computer Vision
Sep 23

SBMVTrack: Spike-Budgeted Multi-View Learning for Energy-Efficient UAV Tracking

SBMVTrack is a fully spiking neural network framework designed for energy-efficient UAV visual tracking. It introduces Energy-Weighted Spike Budgeting (EWSB) to constrain spike activity based on computational cost, and Masked Multi-View Target Modeling (MVTM) to enhance target representation by leveraging correlated temporal views. Experiments on multiple benchmarks show that SBMVTrack reduces spike firing rates and theoretical energy consumption while maintaining competitive tracking accuracy.

By Pengzhi Zhong, Jiwei Mo, Haolun Li, Ge Zheng, Jingqi Wang, Xinyi Bo, Shuiwang Li
arXiv Computer Vision
Sep 23

Real-World Perception for Autonomous Driving in Adverse Weather: Enhancing Standard Detectors via Foundation-Guided Auto-Annotation

The paper presents a foundation-guided auto‑annotation pipeline that improves standard autonomous driving object detectors in adverse weather. By benchmarking YOLOv8, Co‑DETR, and SAM3 on a custom dataset of 25 operational scenarios, the authors find SAM3 to be the most robust and use it offline to generate pseudo‑labels. Fine‑tuning YOLOv8 on these labels boosts overall mAP by 16.04% and yields significant gains in specific conditions such as Residential Direct Sunlight (32.73%) and Highway Fog (28.65%).

By Sepideh Gohari, Goodarz Mehr, Azim Eskandarian
arXiv Computer Vision
Sep 23

Point Diffusion Mamba: Unified Diffusion-State-Space Modeling for Single-View 3D Reconstruction under Data Scarcity

Point Diffusion Mamba (PDM) is a new method that fuses diffusion models with state‑space modeling to perform single‑view 3D reconstruction when training data are scarce. It uses a lightweight reconstruction module for unordered point‑clouds, a Local Geometric Aggregation module combined with Mamba blocks to capture both global geometry and local detail, and a Hierarchical Feature Integration Network to merge high‑level semantic and local geometric features for each point. A Dynamic Weighted Sampling strategy further improves reconstruction quality by integrating generative priors, and experiments on ShapeNet and Pix3D show that PDM outperforms existing state‑of‑the‑art approaches.

By Wei Zhou, Xinzhe Shi, Xingxing Hao, Xing Hao, Kang Li, Jinye Peng, Ying He
arXiv Computer Vision
Sep 23

Decoupling Disease, Covariates, and Individual Variability: A Unified Disentanglement Framework for Medical Image Classification

The paper introduces MedIDL, a Medical Imaging Disentanglement Learning framework that separates disease-related features from confounding covariates and individual variability in medical images. It achieves this by projecting image features into three orthogonal latent spaces—disease classification, covariate alignment, and a Gaussian head for individual variation—using specialized disentanglement heads. Across seven diverse imaging datasets, MedIDL surpasses state‑of‑the‑art supervised and self‑supervised methods in classification accuracy, and its latent representations and gradient‑based visualizations align with known clinical patterns.

By Shengjie Zhang, Jinglin Zhang, Zhuangzhuang Jiang, Ziqi Yu, Yipin Zhang, Qi Zhang, Xiang Chen, Haibo Yang, Fei Gao, Longbiao Cui, Yuan Zhou, Xiao-Yong Zhang, Alzheimer's Disease Neuroimaging Initiative
arXiv Computer Vision
Sep 23

Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs

The paper introduces STD, a hierarchical token pruning framework for Large Vision‑Language Models that aligns pruning strategies with the functional roles of different network stages. By using high‑frequency spectral analysis in shallow layers, Gaussian‑smoothed attention in intermediate layers, and a stability‑adaptive trigger in deep layers, STD preserves essential visual information while aggressively reducing token counts. Experiments demonstrate that STD outperforms existing pruning methods, achieving up to 94.4% token reduction and a 3.9× speed‑up on LLaVA‑NeXT‑7B.

By Shuo Zhang, Jintao Tong, Yixiong Zou, Yuhua Li, Ruixuan Li
arXiv Computer Vision
Sep 23

C2FXNet: Coarse-to-Fine Scene Expert for Unified Object Detection across Adverse Weather

C2FXNet is a unified object detection framework designed for adverse weather conditions. It uses a dual-level guidance mechanism: a Multi-step Reasoning Router (MRR) for coarse scene reasoning and a Fine Scene Refinement (FSR) module for fine-grained semantic adjustment. A Scene-aware Mixture-of-Experts (SMoE) dynamically combines scene-specific experts, enabling robust detection across foggy, dark, and clear scenes without scene-specific training.

By Tianle Fang, Zhenbing Liu, Chong Yin, Bolun Li, Haoxiang Lu
arXiv Computer Vision
Sep 23

Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs

Vision‑language models (VLMs) can lose accuracy when images are resized, even with minimal changes. The study shows that such small visual configuration changes—like tiling or token arrangement—cause more correctness flips across multiple checkpoints and benchmarks. Interestingly, in many cases the models still read the correct answer but fail to use it, and attention interventions reveal that configuration shifts weaken the use of readable information. By guiding models with field cues and their own transcriptions, the authors correct 97.2% of these errors.

By Dingyang Lin, Yingfeng Luo, Chenglong Wang, Chenwei Zhu, Anxiang Ma, Jingbo Zhu, Tong Xiao
arXiv Computer Vision
Sep 23

MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models

MorphoSHAP is a model‑agnostic post‑hoc explanation method that uses morphological shapes—derived from the Tree of Shapes—as the units of attribution in a Shapley game. Each shape is characterized by its scale, geometry, and signed contribution, enabling explanations that reveal where evidence lies, what structural type carries it, and how strongly it influences the prediction. The approach offers spatial, textual, and global class‑level explanations, surpasses existing attribution methods on multiple benchmarks, and is preferred by users in a study.

By Anirudh Prabhakaran, Alexandre Rocchi, Gianni Franchi
arXiv Computer Vision
Sep 23

Delving into Asymmetric Information Dynamics for High-Fidelity Virtual Try-On

The paper introduces RealFit, a virtual try‑on framework that addresses texture degradation and structural drift caused by symmetric bidirectional attention in Diffusion Transformers. By employing Unidirectional Information Flow to isolate garment conditions from stochastic noise and Decoupled Timestep Modulation to strengthen the conditional signal, RealFit achieves high‑fidelity garment rendering. The method also enables a time‑invariant condition branch and a conditional KV cache, cutting inference time by about 75%.

By Zishu Qin, Zhiyu Jin, Pipei Huang, Hao Zhou
arXiv Computer Vision
Sep 23

NaCR: Visual Localization via NeRF-aided Camera Ray Regression

NaCR: Visual Localization via NeRF-aided Camera Ray Regression proposes a unified framework that integrates Neural Radiance Fields (NeRF) with Camera Ray Regression (CRR) to improve visual localization accuracy. The method enhances the CRR baseline with three simple improvements, augments training data by synthesizing novel views from a pre‑trained NeRF, and employs a closed‑loop supervision pipeline that back‑propagates photometric rendering errors to refine predicted camera rays. A two‑stage training curriculum ensures stable convergence, and experiments on indoor and outdoor benchmarks show competitive accuracy with validated component efficacy.

By Yesheng Zhang, Xiang Dai, Xu Zhao, Chongyang Zhang
arXiv Computer Vision
Sep 23

Calibrating Retrieval Geometry: Reliability-Guided Training-Free Aggregation for Visual Place Recognition

The paper introduces TFA, a training‑free aggregation technique that calibrates frozen visual foundation models for visual place recognition. TFA uses cross‑codebook agreement, retrieval coverage, and spectral statistics to adjust residual assignment, spectral shaping, and global‑feature fusion without requiring place labels or task‑specific weights. Experiments with a DINOv2‑B backbone show significant Recall@1 gains over existing training‑free methods across multiple benchmarks, demonstrating that reliability‑guided aggregation can unlock additional retrieval performance from frozen representations.

By Xin Li, Zhimin Mao, Shang Wang, Siyuan Duan, Geng Zhang