Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,528 stories · RSS feed

arXiv Machine Learning

Aug 4

HP-JEPA: Hierarchical Partitioning for Multi-Resolution Graph Joint-Embedding Predictive Learning

arXiv:2608. 00491v1 Announce Type: new Abstract: Graph self-supervised learning aims to learn transferable representations from large-scale unlabeled graph data.

By Ruichen Xu, Jingxiang Qu, Wenhan Gao, Jiaxing Zhang, Linsey Pang, Ravid Shwartz-Ziv, Yann LeCun, Yuefan Deng

rag diffusion benchmarks

arXiv Machine Learning

Aug 4

Uncertainty-Aware Simulation-Based Inference for Operations Research with Large Language Models

arXiv:2608. 00019v1 Announce Type: new Abstract: Deploying large language models (LLMs) for operations research (OR) tasks remains challenging because correctness depends on a coherent modeling process, not merely a correct final answer.

By Liang Guo, Lin Shaochong, Shen Zuo-Jun Max, Zhang Kun

llms benchmarks

arXiv Machine Learning

Aug 4

From field-scale to large-scale spectral libraries: Tabular foundation models in soil spectroscopy

arXiv:2608. 00608v1 Announce Type: new Abstract: Visible and near-infrared (vis-NIR) and mid-infrared (MIR) spectroscopy enable rapid, cost-effective prediction of soil properties.

By Viacheslav Barkov, Jonas Schmidinger, Robin Gebbers, Martin Atzmueller

llms benchmarks

arXiv Machine Learning

Aug 4

Mitigating Backdoors via Decoy Shortcuts and Knowledge Decoupling

arXiv:2608. 00732v1 Announce Type: new Abstract: Backdoor attacks pose a serious threat to deep neural networks, especially when training relies on third-party data, allowing adversaries to inject malicious behaviors through data poisoning.

By Zixuan Zhu, Rui Wang, Lihua Jing, Jinwen Zhong

benchmarks

arXiv Machine Learning

Aug 4

HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

arXiv:2606. 21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle.

By Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee, Ion Stoica, Jae W. Lee

llms diffusion efficiency benchmarks

arXiv Machine Learning

Aug 4

Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA

arXiv:2607. 26052v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts $k$.

By Tom Saliencro, Rohan Desai, Priya Nair, Maya Lindqvist, Daniel Whitmore

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 4

LOCUS-DT: Localization via Observation-Conditioned Uncertainty Scoring with Digital Twins

arXiv:2608. 00406v1 Announce Type: cross Abstract: Accurate indoor localization is essential for emerging applications in robotic navigation and search and rescue.

By Haozhe Lei, Roberto Bomfin, Marwa Chafii, Sundeep Rangan

robotics multimodal benchmarks

arXiv Machine Learning

Aug 4

NORi: An ML-Augmented Ocean Boundary Layer Parameterization

arXiv:2512. 04452v3 Announce Type: replace-cross Abstract: NORi is a machine learning (ML) parameterization of ocean boundary layer turbulence that is physics-based and augmented with neural networks.

By Xin Kai Lee, Ali Ramadhan, Andre Souza, Gregory LeClaire Wagner, Simone Silvestri, John Marshall, Raffaele Ferrari

benchmarks

arXiv Machine Learning

Aug 4

Nonparametric Distribution Regression Re-calibration

arXiv:2602. 13362v2 Announce Type: replace-cross Abstract: A key challenge in probabilistic regression is ensuring that predictive distributions accurately reflect true empirical uncertainty.

By \'Ad\'am Jung, Domokos M. Kelen, Andr\'as A. Bencz\'ur

rag benchmarks safety

arXiv Machine Learning

Aug 4

MAPLE: Metadata Augmented Private Language Evolution

arXiv:2603. 19258v2 Announce Type: replace-cross Abstract: Differentially private (DP) fine-tuning of large language models (LLMs) requires massive compute and full model access, which rules out state-of-the-art proprietary APIs for general users.

By Eli Chien, Yuzheng Hu, Ryan McKenna, Shanshan Wu, Zheng Xu, Peter Kairouz

llms fine-tuning benchmarks safety

arXiv Machine Learning

Aug 4

DRIFT: Drift-Resilient Invariant-Feature Transformer for DGA Detection

arXiv:2605. 10436v2 Announce Type: replace-cross Abstract: Domain Generation Algorithms (DGAs) evolve continuously to evade botnet detection, posing a persistent challenge for dependable network defense.

By Chaeyoung Lee, Chaeri Jung, Seonghoon Jeong

llms nlp fine-tuning benchmarks

arXiv Machine Learning

Aug 4

Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

arXiv:2605. 15607v2 Announce Type: replace-cross Abstract: Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood.

By Vinayshekhar Bannihatti Kumar, Disha Makhija, Manoj Ghuhan Arivazhagan, Rashmi Gangadharaiah

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 4

Physen-Noise2Noise: Physics-Guided Self-Supervised Defocus Deblurring with Bias Correction under Low-Light Conditions

arXiv:2605. 24590v2 Announce Type: replace-cross Abstract: Low-light, long-exposure defocus deblurring remains a challenging problem due to the simultaneous presence of severe blur and complex biased noise.

By Ziyan Huang, Lang Wu, Hongji Wang, Yifei Liu, Dongliang Tang, Hongqiao Wang

benchmarks safety

arXiv Machine Learning

Aug 4

QWRF-Net: A Quantum-Wavelet Framework with Rectified Flow for Short-Term Precipitation Nowcasting

arXiv:2608. 01626v1 Announce Type: new Abstract: Short-term precipitation nowcasting is important for hydrometeorological early warning, especially when intense convective rainfall may trigger urban flooding, flash floods, and other high-impact hazards.

By Zhuo Wang, Chaorong Li, Wenjie Luo, Chuanhu Deng

diffusion benchmarks

arXiv Machine Learning

Aug 4

Understanding and Correcting Low-Frequency Bias in EEG Foundation Model

arXiv:2608. 01898v1 Announce Type: new Abstract: Increasing EEG pretraining data scale or model capacity does not consistently improve downstream performance.

By Junjie Yu, Zihan Deng, Jianyu Zhang, Junrong Mu, Jiahui An, Wenxiao Ma, Ziling Lu, Yue Wang, Yan Zhu, Kexin Lou, Quanying Liu

benchmarks safety

arXiv Machine Learning

Aug 4

Domain-Generalized Adaptive Semantic Communication for Collaborative Perception

arXiv:2608. 00056v1 Announce Type: cross Abstract: We propose RSTA, a domain-generalized semantic communication framework enabling source-free V2X collaborative perception under both observation-domain shift and unseen wireless channel conditions.

By Fan Gao, Youzheng Wang, Ning Ge

agents benchmarks safety

arXiv Machine Learning

Aug 4

Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget

arXiv:2608. 00916v1 Announce Type: cross Abstract: Modern reranking recipes---billion-scale cross-encoders, mixture-of-experts (MoE) backbones, and distillation against strong teachers---have outpaced the training infrastructure available to most academic groups.

By Zhichao Xu, Xueguang Ma, Shengyao Zhuang, Luyu Gao, Wenqian Ye, Yu Wang, Jamie Callan, Jimmy Lin

llms rag fine-tuning efficiency benchmarks data-engineering

arXiv Machine Learning

Aug 4

Fruit-HSNet: A Machine Learning Approach for Hyperspectral Image-Based Fruit Ripeness Prediction

arXiv:2608. 01202v1 Announce Type: cross Abstract: Fruit ripeness prediction (FRP) is a classification-based agricultural computer vision task that has attracted much attention, thanks to its wide-ranging advantages in agriculture field for both pre-harvest and post-harvest management.

By Ahmed Baha Ben Jmaa, Faten Chaieb, Anna Fabija\'nska

computer-vision benchmarks

arXiv Machine Learning

Aug 4

DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models

arXiv:2608. 01821v1 Announce Type: cross Abstract: Diffusion vision-language models (dVLMs) iteratively denoise masked responses while conditioning each denoising step on visual evidence, making visual conditioning a substantial recurring inference cost.

By Yongkang Zhou, Xiang Xia, Cheng Yan, Fan Xu, Wuyang Zhang

llms diffusion efficiency multimodal benchmarks

arXiv Machine Learning

Aug 4

Hard Constraints, Smooth Gradients: Learning Feasible Inventory Policies via Differentiable Projection

arXiv:2608. 02343v1 Announce Type: cross Abstract: Many operational problems are constrained sequential decision processes with large, combinatorial action spaces and interdependent feasibility constraints.

By Patrick Helm, Jan-Niklas Doerr, Joren Gijsbrechts, Stefan Minner

rag reinforcement-learning benchmarks

Get the daily briefing