Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Statistics ML
Sep 28

SAGE: A sampling-aware global evaluation benchmark for species distribution modeling

The paper introduces SAGE, a Sampling‑Aware Global Evaluation benchmark for species distribution modeling that uses GBIF records for training and sPlotOpen vegetation plots for presence‑absence evaluation across 5,771 plant species. It groups species by sampling effort and relative prevalence to assess how well single‑species and multi‑species deep‑learning SDMs perform under different data conditions. The study finds that Random Forests and DeepSDMs perform best overall, with DeepSDMs excelling for infrequently recorded species only when bias‑correction techniques are applied.

By Emilia Arens, Nina van Tiel, Robin Zbinden, Damien Robert, Lukas Drees, Chiara Vanalli, Benjamin Kellenberger, Niklaus E. Zimmermann, Lo\"ic Pellissier, Devis Tuia, Jan Dirk Wegner
arXiv Computer Vision
Sep 28

Query-Conditioned Prototype Adaptation for Cross-Domain Few-Shot Learning: Single-Query Inference, Controlled Comparisons, and Failure Modes

The paper introduces the Within-Instance Prototypical Transformer (WIPT), a method that performs single-query test-time prototype adaptation by jointly transforming an unlabelled query with labelled support embeddings to form query-specific class means. Using a frozen ViT-S/16 encoder trained on miniImageNet and evaluated on CUB, EuroSAT, and ISIC, WIPT improves 1‑shot performance on CUB and EuroSAT but not on ISIC, while in 5‑shot settings it outperforms a support‑only Transformer on ISIC but remains weaker than ProtoNet overall. The study also explores multi‑query processing, memory and latency trade‑offs, and analyzes how WIPT alters uncertain versus confident predictions, concluding that the method offers a streaming‑compatible test‑time adaptation that can enhance low‑shot cross‑domain decisions without target‑time optimization.

By Rushab Rasik Karania, Tomas Maul
arXiv Computation and Language
Sep 28

KuaFu: Compressing Long User Behavior into Understanding at Billion Scale

KuaFu is a unified behavior‑compression layer that reduces each user behavior item to 2–4 tokens, dramatically shrinking per‑item cache size while preserving fidelity through a four‑stage training process. In production across four profiling tasks, it matches or outperforms uncompressed single‑task models, boosts GPU throughput by 37–350%, and saves 190 GPUs. On public benchmarks it consistently beats prior compressors at the same compression ratio, and on RecBench a 4B KuaFu model outperforms its 8B counterpart by 1.90 points, contributing to a 1.37% lift in overall GMV on Tencent’s advertising and recommendation platform.

By Jiahao Hui, Lin Zhu, Yishen Hu, Jingdong Shu, Zetai Jiang, Xining Ran, Ben Tan, Yeshou Cai, Gong Chen, Haijie Gu, Jie Jiang
arXiv Computation and Language
Sep 28

TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

TRACE (Temporal Audit and Condition-aware Evaluation) is a new benchmark and evaluation framework for streaming video understanding that explicitly records when evidence becomes valid, how visual history is maintained, and how responses are triggered. It combines temporally audited visual tasks, evidence timing, instruction-dependent trigger annotations, a unified causal Core–Adapter protocol, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. Using 1,240 records from 517 videos, TRACE evaluated eight publicly available models in eight configurations, revealing that similar QA accuracy can hide significant differences in completion, answer validity, generation workload, and proactive performance metrics such as response delay, false alarms, and missed target windows.

By Yibo Ma, Qianqian Zhang, Peng Liu, Tiancheng Zhao
arXiv Computer Vision
Sep 28

Double-stream registration with pyramid fusion for HDR video with alternating exposures

The paper introduces a new HDR video reconstruction framework that uses a double-stream registration approach combined with pyramid fusion. It processes three consecutive frames by computing optical flow relative to the central frame and applying a midpoint displacement strategy to address severe overexposure. The resulting radiance and low‑dynamic‑range images are merged in a pyramid fusion stage to produce the final HDR output, and experiments show the method outperforms existing state‑of‑the‑art techniques.

By Onofre Martorell, Ivan Pereira-S\'anchez, Antoni Fuentes, Antoni Buades
arXiv Computer Vision
Sep 28

Gauss What You Need: Compact Gaussian Splatting Across Scene Scales

Gauss What You Need: Compact Gaussian Splatting Across Scene Scales introduces TangoGS, a method that automatically selects the number of Gaussian primitives for 3D Gaussian Splatting by combining capture-derived model sizing with training-based adaptation. The approach first estimates a learning allowance based on the capture’s total pixels, then adjusts the number of Gaussians during training according to reconstruction quality. On standard benchmarks, TangoGS matches the best baseline’s PSNR while using 48% fewer Gaussians, and on larger captures it scales automatically to achieve the highest mean PSNR with 2.3× more Gaussians.

By Afif Boudaoud, Jiayi Liu, Alexandru Calotoiu, Torsten Hoefler
arXiv Computer Vision
Sep 28

A Multi-Stage Framework for Kuzushiji Character Recognition in Japanese Historical Documents

The paper presents a multi‑stage framework for recognizing Kuzushiji characters in Japanese historical documents. It combines character detection, cropping, classification, reading‑order reconstruction via adaptive column clustering, and large‑language‑model‑based post‑OCR correction. The authors also augment data synthetically, correct dataset annotations, and introduce new test sets, achieving significant character error rate reductions on real, synthetic, and out‑of‑domain data.

By Rui-Yang Ju, Kohei Yamashita, Hirotaka Kameko, Shinsuke Mori
arXiv Computation and Language
Sep 28

All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation

The paper introduces FAST-CAP, a causality‑aware framework for simultaneous speech‑to‑speech translation that combines a factorized S2ST architecture, an adaptive policy, and a new latency metric. It employs a novel data pipeline to generate high‑fidelity, causally aligned segments, improving voice transfer and reducing the need for large training datasets. Experiments on Spanish, German, and French demonstrate that FAST‑CAP outperforms fixed‑policy baselines, achieving up to +1.2 BLEU, 26% lower latency, and a 38.8% relative latency reduction while maintaining speaker fidelity.

By Amir Hussein, Enas Albasiri, Travis M. Bartley, Nourchene Ferchichi, Ke Hu, Harishchandra Dubey, Myungjong Kim, Zhehuai Chen, Oluwatobi Olabiyi, Sanjeev Khudanpur
arXiv Computation and Language
Sep 28

Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations

The paper introduces ARGUS, a language‑model pipeline that audits evidence for identification assumptions in difference‑in‑differences studies of climate policy. ARGUS evaluates reported evidence against an eleven‑dimension rubric, abstaining when evidence cannot be retrieved. In tests, ARGUS detects 73% of injected flaws versus 18% for a keyword approach, abstains on about 40% of assessments in 26 economics papers, and often assigns higher risk than human labels in a five‑paper pilot.

By Yonghong Zhang, Yong Xie, Isabel M. Parra, Ricardo Correia
arXiv Computation and Language
Sep 28

FAVoR: Measuring and Mitigating Author-Style Homogenization in Federated Personalized Generation

The paper introduces FAVoR, a method for federated personalized generation that mitigates author‑style homogenization caused by standard aggregation in parameter‑efficient fine‑tuning. Using the BlogText benchmark and ASCE diagnostics, the authors show that common federated PEFT baselines preserve semantic utility but blur author‑specific style. FAVoR employs a shared‑private adapter design, where clients upload shared updates while keeping author‑specific residual corrections locally, leading to improved style retention with minimal utility loss.

By Lu Han, Jingyao Zhang, Katy Ilonka Gero, Nguyen H. Tran