Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,260 stories · RSS feed

arXiv Machine Learning

Aug 4

Nonparametric Distribution Regression Re-calibration

arXiv:2602. 13362v2 Announce Type: replace-cross Abstract: A key challenge in probabilistic regression is ensuring that predictive distributions accurately reflect true empirical uncertainty.

By \'Ad\'am Jung, Domokos M. Kelen, Andr\'as A. Bencz\'ur

rag benchmarks safety

arXiv Machine Learning

Aug 4

MAPLE: Metadata Augmented Private Language Evolution

arXiv:2603. 19258v2 Announce Type: replace-cross Abstract: Differentially private (DP) fine-tuning of large language models (LLMs) requires massive compute and full model access, which rules out state-of-the-art proprietary APIs for general users.

By Eli Chien, Yuzheng Hu, Ryan McKenna, Shanshan Wu, Zheng Xu, Peter Kairouz

llms fine-tuning benchmarks safety

arXiv Machine Learning

Aug 4

DRIFT: Drift-Resilient Invariant-Feature Transformer for DGA Detection

arXiv:2605. 10436v2 Announce Type: replace-cross Abstract: Domain Generation Algorithms (DGAs) evolve continuously to evade botnet detection, posing a persistent challenge for dependable network defense.

By Chaeyoung Lee, Chaeri Jung, Seonghoon Jeong

llms nlp fine-tuning benchmarks

arXiv Machine Learning

Aug 4

Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

arXiv:2605. 15607v2 Announce Type: replace-cross Abstract: Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood.

By Vinayshekhar Bannihatti Kumar, Disha Makhija, Manoj Ghuhan Arivazhagan, Rashmi Gangadharaiah

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 4

Physen-Noise2Noise: Physics-Guided Self-Supervised Defocus Deblurring with Bias Correction under Low-Light Conditions

arXiv:2605. 24590v2 Announce Type: replace-cross Abstract: Low-light, long-exposure defocus deblurring remains a challenging problem due to the simultaneous presence of severe blur and complex biased noise.

By Ziyan Huang, Lang Wu, Hongji Wang, Yifei Liu, Dongliang Tang, Hongqiao Wang

benchmarks safety

arXiv Machine Learning

Aug 4

QWRF-Net: A Quantum-Wavelet Framework with Rectified Flow for Short-Term Precipitation Nowcasting

arXiv:2608. 01626v1 Announce Type: new Abstract: Short-term precipitation nowcasting is important for hydrometeorological early warning, especially when intense convective rainfall may trigger urban flooding, flash floods, and other high-impact hazards.

By Zhuo Wang, Chaorong Li, Wenjie Luo, Chuanhu Deng

diffusion benchmarks

arXiv Machine Learning

Aug 4

Understanding and Correcting Low-Frequency Bias in EEG Foundation Model

arXiv:2608. 01898v1 Announce Type: new Abstract: Increasing EEG pretraining data scale or model capacity does not consistently improve downstream performance.

By Junjie Yu, Zihan Deng, Jianyu Zhang, Junrong Mu, Jiahui An, Wenxiao Ma, Ziling Lu, Yue Wang, Yan Zhu, Kexin Lou, Quanying Liu

benchmarks safety

arXiv Machine Learning

Aug 4

Domain-Generalized Adaptive Semantic Communication for Collaborative Perception

arXiv:2608. 00056v1 Announce Type: cross Abstract: We propose RSTA, a domain-generalized semantic communication framework enabling source-free V2X collaborative perception under both observation-domain shift and unseen wireless channel conditions.

By Fan Gao, Youzheng Wang, Ning Ge

agents benchmarks safety

arXiv Machine Learning

Aug 4

Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget

arXiv:2608. 00916v1 Announce Type: cross Abstract: Modern reranking recipes---billion-scale cross-encoders, mixture-of-experts (MoE) backbones, and distillation against strong teachers---have outpaced the training infrastructure available to most academic groups.

By Zhichao Xu, Xueguang Ma, Shengyao Zhuang, Luyu Gao, Wenqian Ye, Yu Wang, Jamie Callan, Jimmy Lin

llms rag fine-tuning efficiency benchmarks data-engineering

arXiv Machine Learning

Aug 4

Fruit-HSNet: A Machine Learning Approach for Hyperspectral Image-Based Fruit Ripeness Prediction

arXiv:2608. 01202v1 Announce Type: cross Abstract: Fruit ripeness prediction (FRP) is a classification-based agricultural computer vision task that has attracted much attention, thanks to its wide-ranging advantages in agriculture field for both pre-harvest and post-harvest management.

By Ahmed Baha Ben Jmaa, Faten Chaieb, Anna Fabija\'nska

computer-vision benchmarks

arXiv Machine Learning

Aug 4

DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models

arXiv:2608. 01821v1 Announce Type: cross Abstract: Diffusion vision-language models (dVLMs) iteratively denoise masked responses while conditioning each denoising step on visual evidence, making visual conditioning a substantial recurring inference cost.

By Yongkang Zhou, Xiang Xia, Cheng Yan, Fan Xu, Wuyang Zhang

llms diffusion efficiency multimodal benchmarks

arXiv Machine Learning

Aug 4

Hard Constraints, Smooth Gradients: Learning Feasible Inventory Policies via Differentiable Projection

arXiv:2608. 02343v1 Announce Type: cross Abstract: Many operational problems are constrained sequential decision processes with large, combinatorial action spaces and interdependent feasibility constraints.

By Patrick Helm, Jan-Niklas Doerr, Joren Gijsbrechts, Stefan Minner

rag reinforcement-learning benchmarks

arXiv Machine Learning

Aug 4

Learning Graph-Indexed Trajectory Patterns for Stochastic On-Time Arrival Routing

arXiv:2508. 17218v4 Announce Type: replace Abstract: Correlated link travel times create decision-relevant patterns in partial route histories.

By Yuanhang Wang, Xing Wei, Duoxiang Zhao, Zezhou Zhang, Hao Qin, Yuqi Ouyang

llms benchmarks

arXiv Machine Learning

Aug 4

Amortized Inference of Multi-Modal Posteriors using Likelihood-Weighted Normalizing Flows

arXiv:2512. 04954v3 Announce Type: replace Abstract: We present a novel technique for amortized posterior estimation using Normalizing Flows trained with likelihood-weighted importance sampling.

By Rajneil Baruah

multimodal benchmarks

arXiv Machine Learning

Aug 4

ARMOR: A Robust Self-Supervised Framework for Root Cause Analysis in Microservices under Missing Modality

arXiv:2603. 25538v3 Announce Type: replace Abstract: Automated incident management is critical for microservice reliability.

By Wenzhuo Qian, Hailiang Zhao, Ziqi Wang, Zhipeng Gao, Jiayi Chen, Zhiwei Ling, Shuiguang Deng

agents multimodal benchmarks safety

arXiv Machine Learning

Aug 4

From Vessel Trajectories to Safety-Critical Encounter Scenarios: A Generative AI Framework for Autonomous Ship Digital Testing

arXiv:2603. 28067v2 Announce Type: replace Abstract: Digital testing has emerged as a key paradigm for the development and verification of autonomous maritime navigation systems, yet the availability of realistic and diverse safety-critical encounter scenarios remains limited.

By Sijin Sun, Liangbin Zhao, Xiuju Fu

agents benchmarks safety

arXiv Machine Learning

Aug 4

GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding

arXiv:2605. 13352v2 Announce Type: replace Abstract: Standard dual-encoder vision-language models that map images and text to deterministic points on a shared unit hypersphere through $\ell_2$ normalization typically expose neither \emph{aleatoric} uncertainty (cross-modal ambiguity) nor \emph{epistemic} uncertainty (lack of training-distribution support).

By Mayank Nautiyal, Li Ju, Andreas Hellander, Ekta Vats, Prashant Singh

llms rag diffusion multimodal benchmarks

arXiv Machine Learning

Aug 4

A 2-Block Architecture for Real-Time EEG Gait Decoding: A Pilot Study

arXiv:2608. 02083v1 Announce Type: new Abstract: Closed-loop lower-limb exoskeleton control via Electroencephalography (EEG) remains limited by motion artifacts, low signal-to-noise ratio, and binary gait formulations that fail to capture full cortical gait complexity.

By Shantanu Sarkar, Saurabh Prasad, Jose L. Contreras-Vidal

benchmarks

arXiv Machine Learning

Aug 4

Posterior Variance Is a Constraint Map, Not an Error Map: Closed-Form Uncertainty for Radiative Gaussian Splatting in Sparse-View CT

arXiv:2607. 13682v2 Announce Type: cross Abstract: Radiative Gaussian splatting reconstructs sparse-view CT fast and accurately, and recent work attaches per-Gaussian posteriors to yield per-voxel uncertainty maps.

By Chulin Zhao, Yiran Xu, Shu Liu

benchmarks safety

arXiv Machine Learning

Aug 4

CountTRuCoLa: Rule Learning for Interpretable Temporal Knowledge Graph Forecasting

arXiv:2509. 09474v2 Announce Type: replace Abstract: We address the task of temporal knowledge graph forecasting with an inherently interpretable method based on symbolic rules.

By Julia Gastinger, Christian Meilicke, Heiner Stuckenschmidt

benchmarks

Get the daily briefing