Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,789 stories · RSS feed

arXiv AI

Jun 19

QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.

By Xinyi Zheng, Ling Shi, Tianlong Yu, Yongxin Zhao, Lorenz Goette, Kailong Wang

llms benchmarks

arXiv AI

Jun 19

SL-S4Wave: Self-Supervised Learning of Physiological Waveforms with Structured State Space Models

arXiv:2606. 19888v1 Announce Type: cross Abstract: Modeling long-sequence medical time series data, such as electrocardiograms (ECG), poses significant challenges due to high sampling rates, multichannel signal complexity, inherent noise, and limited labeled data.

By Feng Wu, Harsh Deep, Eric Lehman, Sanyam Kapoor, Guoshuai Zhao, Rahul Krishnan, Gari Clifford, Li-wei H Lehman

benchmarks

arXiv AI

Jun 19

Neural Additive and Basis Models with Feature Selection and Interactions

arXiv:2606. 19850v1 Announce Type: cross Abstract: Deep neural networks (DNNs) exhibit attractive performance in various fields but often suffer from low interpretability.

By Yasutoshi Kishimoto, Kota Yamanishi, Takuya Matsuda, Shinichi Shirakawa

benchmarks safety

arXiv Machine Learning

Jun 19

Model soups need only one ingredient

arXiv:2602. 09689v2 Announce Type: replace Abstract: Fine-tuning large pre-trained models on a target distribution often improves in-distribution (ID) accuracy, but at the cost of out-of-distribution (OOD) robustness as representations specialize to the fine-tuning data.

By Alireza Abdollahpoorrostam, Nikolaos Dimitriadis, Adam Hazimeh, Pascal Frossard

llms fine-tuning benchmarks

arXiv AI

Jun 19

Multi-View Decompilation for LLM-Based Malware Classification

arXiv:2606. 20436v1 Announce Type: cross Abstract: Malware analysts often inspect compiled binaries through decompiled pseudo-C, when source code is unavailable.

By Bercan Turkmen, Vyas Raina

llms benchmarks

arXiv Machine Learning

Jun 19

CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training

arXiv:2510. 18784v3 Announce Type: replace Abstract: Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training.

By Soroush Tabesh, Mher Safaryan, Andrei Panferov, Alexandra Volkova, Dan Alistarh

llms fine-tuning efficiency benchmarks

arXiv AI

Jun 19

TerraMind: Large-Scale Generative Multimodality for Earth Observation

arXiv:2504. 11171v5 Announce Type: replace-cross Abstract: We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO).

By Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, Nicolas Long\'ep\'e

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

FreshRetailNet-50K: A Stockout-Annotated Censored Demand Dataset for Latent Demand Recovery and Forecasting in Fresh Retail

arXiv:2505. 16319v5 Announce Type: replace Abstract: Accurate demand estimation is critical for the retail business in guiding the inventory and pricing policies of perishable products.

By Yangyang Wang, Jiawei Gu, Li Long, Xin Li, Li Shen, Zhouyu Fu, Xiangjun Zhou, Xu Jiang

benchmarks safety

arXiv Machine Learning

Jun 19

DADP: Domain Adaptive Diffusion Policy

arXiv:2602. 04037v3 Announce Type: replace Abstract: Learning domain adaptive policies that can generalize to unseen transition dynamics, remains a fundamental challenge in learning-based control.

By Pengcheng Wang, Qinghang Liu, Haotian Lin, Yiheng Li, Guojian Zhan, Masayoshi Tomizuka, Yixiao Wang

diffusion robotics benchmarks

arXiv AI

Jun 19

BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling

arXiv:2606. 20146v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied to computer-aided design (CAD) to generate design artifacts from textual instructions.

By Bharathi Kannan Nithyanantham, Clemens Kujat, Tobias Sesterhenn, Stefan Telgmann, J\"orn Pl\"onnigs, Stefan L\"udtke, Christian Bartelt

llms benchmarks

arXiv AI

Jun 19

Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation

arXiv:2606. 19636v1 Announce Type: cross Abstract: Math and science reasoning benchmarks rely on pass@k, the fraction of sampled chains that reach gold, as the canonical per-example difficulty signal.

By Luca Zhou, Sajel Shah, Emanuele Rodol\`a, Roberto Dess\`i

reinforcement-learning benchmarks

arXiv AI

Jun 19

CSWinUNETR: Segmentation of Thin Anatomical Structures in Medical Images

arXiv:2606. 19824v1 Announce Type: cross Abstract: Accurate segmentation of thin, tortuous anatomical structures, such as retinal vessels, cerebral vasculature, and facial wrinkles, remains challenging due to low contrast, frequent discontinuities, and severe class imbalance.

By Junho Moon, Haejun Chung, Ikbeom Jang

llms computer-vision benchmarks

arXiv AI

Jun 19

AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Models

arXiv:2509. 25148v2 Announce Type: replace Abstract: Post-training alignment of large language models often combines supervised fine-tuning (SFT) on expert demonstrations with reinforcement learning (RL) from preference or verifiable feedback.

By Faqiang Qian, Kang An, Weikun Zhang, Ziliang Wang, Xuhui Zheng, Liangjian Wen, Yong Dai, Mengya Gao, Yichao Wu

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 19

When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning

arXiv:2606. 19827v1 Announce Type: cross Abstract: Medical tabular data are ubiquitous in clinical research, but deep learning for tables remains underexplored because reliable labels often require costly expert adjudication, even though structured clinical variables are routinely available in tabular form.

By Daehwan Kim, Haejun Chung, Ikbeom Jang

fine-tuning benchmarks safety

arXiv Machine Learning

Jun 19

Adaptive Distance-Aware Trunk Deep Operator Learning for Long-Span Roadway Bridges

arXiv:2606. 20015v1 Announce Type: new Abstract: Long-span roadway bridges exhibit highly localized structural responses under vehicular loading, making repeated FE analysis computationally expensive for applications such as influence surface generation and structural digital twins.

By Bilal Ahmed, Diab W. Abueidda, Waleed El-Sekelly, Tarek Abdoun, Mostafa E. Mobasher

benchmarks

arXiv AI

Jun 19

Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

arXiv:2606. 20502v1 Announce Type: cross Abstract: Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved.

By Arastoo Zibaeirad, Marco Vieira

llms fine-tuning benchmarks

arXiv AI

Jun 19

CogniFold: Always-On Proactive Memory via Cognitive Folding

arXiv:2605. 13438v3 Announce Type: replace Abstract: Existing agent memory remains predominantly reactive and retrieval-based, lacking the capacity to autonomously organize experience into persistent cognitive structure.

By Suli Wang, Yiqun Duan, Yu Deng, Rundong Zhao, Dai Shi, Minghua Deng, Chen Chen, Xinliang Zhou

agents benchmarks

arXiv Machine Learning

Jun 19

A Model-Driven Approach for Developing Families of Reinforcement Learning Environments

arXiv:2606. 20324v1 Announce Type: cross Abstract: Virtual training environments are software-intensive systems in which reinforcement learning (RL) agents learn, adapt, and demonstrate meaningful behavior.

By Xiaoran Liu, Istvan David

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 19

We Need to Rethink Benchmarking in Anomaly Detection

arXiv:2507. 15584v2 Announce Type: replace Abstract: Despite the continuous proposal of new anomaly detection algorithms and extensive benchmarking efforts, progress seems to stagnate, with only minor performance differences between established baselines and new algorithms.

By Philipp R\"ochner, Simon Kl\"uttermann, Kevin Kammler, Franz Rothlauf, Emmanuel M\"uller, Daniel Schl\"or

benchmarks

arXiv Machine Learning

Jun 19

Prompt, Plan, Extract: Zero-Shot Agentic LLMs Workflows for Lung Pathology Extraction from Clinical Narratives

arXiv:2606. 19852v1 Announce Type: cross Abstract: Information extraction from pathology reports is essential for cancer staging, tumor registry population.

By Aman Pathak, Cheng Peng, Mengxian Lyu, Ziyi Chen, Reema Solan, Sankalp Talankar, Yasir Khan, Hiren Mehta, Aokun Chen, Yi Guo, Yonghui Wu

llms agents nlp benchmarks

Get the daily briefing