Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning

Oct 5

Post-Training Quantization of Autoregressive Weather Models

arXiv:2610.02511v1 Announce Type: new Abstract: Advancements in high-resolution numerical weather prediction (NWP) and data assimilation (DA) have shaped the developments in deep learning (DL) archit...

By Ananyo Bhattacharya, Swastik Bhattacharya, Christiane Jablonowski

efficiency benchmarks

arXiv Machine Learning

Oct 5

Learning the Latent Structure: A Feature-Centric Approach to Graph Data Augmentation

arXiv:2610.02517v1 Announce Type: new Abstract: Graph-structured data plays a pivotal role in modeling complex relationships. However, real-world graphs are often incomplete due to data collection an...

By Yu Song, Zhigang Hua, Yan Xie, Bingheng Li, Jingzhe Liu, Bo Long, Jiliang Tang, Hui Liu

rag benchmarks

arXiv Machine Learning

Oct 5

Autoregressive Differentiable Method for Integer Programming

arXiv:2610.02528v1 Announce Type: new Abstract: We introduce an autoregressive differentiable method to solve 0-1 integer programs. We fix an arbitrary order of the binary variables and we train a tr...

By Ouns El Harzli, Yudong Cao

llms benchmarks

arXiv Machine Learning

Oct 5

DISSOLVR: An Interpretable and Fast Framework for Aqueous and Organic Solubility Prediction

arXiv:2610.02574v1 Announce Type: new Abstract: High-fidelity solubility prediction is fundamental to pharmaceutical development and environmental partitioning, where accurate modeling must couple mo...

By Vansh Ramani, Har Ashish Arora, Dhairya Kuchhal, Sayan Ranu, Tarak Karmakar

llms benchmarks safety

arXiv AI

Oct 5

D2K-Bench: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?

arXiv:2610.03226v1 Announce Type: cross Abstract: GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal h...

By Daifeng Li, Huiqiang Jiang, Chengruidong Zhang, Wei Wu, Xudong Guo, Jianhong Tu, Jianwei Zhang, Binhang Yuan, Dayiheng Liu

llms agents benchmarks

arXiv Machine Learning

Oct 5

Bifidelity Karhunen-Lo\`eve Expansion Surrogate with Active Learning for Random Fields

arXiv:2511.03756v2 Announce Type: replace-cross Abstract: We present a bifidelity Karhunen--Lo\`{e}ve expansion (KLE) surrogate model for field-valued quantities of interest (QoIs) under uncertain in...

By Aniket Jivani, Cosmin Safta, Beckett Y. Zhou, Xun Huan

diffusion benchmarks safety

arXiv AI

Oct 5

FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training

arXiv:2610.02828v1 Announce Type: new Abstract: Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL r...

By Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Daren Zha, Jun Xiao

llms reinforcement-learning benchmarks

arXiv Computer Vision

Oct 5

FlowHMR: Physically Plausible Motion Capture from Video

arXiv:2610.03691v1 Announce Type: new Abstract: We present FlowHMR, a framework for recovering physically plausible global 3D human motion from monocular video. Previous learning-based methods typica...

By Zhanke Wang, Chengfeng Zhao, Qing Shuai, Jingzhong Lin, Heng Li, Zeyu Ling, Yuxin Wen, Jing Li, Di Kang, Chunchao Guo, Linchao Bao

diffusion benchmarks

arXiv AI

Oct 5

4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

arXiv:2610.03715v1 Announce Type: cross Abstract: We introduce 4DCodeBench, a benchmark for 4D inverse graphics through code generation, in which agents reconstruct dynamic scenes from video as execu...

By Ruihong Shen, \v{Z}iga Kova\v{c}i\v{c}, Peter Kulits, Xingrui Wang, Zizhang Li, Joshua B. Tenenbaum, Alan Yuille, Jieneng Chen, Jiajun Wu

agents benchmarks

arXiv AI

Oct 5

Recoverability Has a Law: The ERR Measure for Tool-Augmented Agents

arXiv:2601.22352v2 Announce Type: replace-cross Abstract: Language model agents often appear capable of self-recovery after failing tool call executions, yet this behavior lacks a formal explanation....

By Sri Vatsa Vuddanti, Satwik Kumar Chittiprolu

llms agents benchmarks

arXiv AI

Oct 5

Beyond Trained Models: Compiling GNNs for a Sound Explainer Benchmark

arXiv:2610.03526v1 Announce Type: cross Abstract: Explainers for Graph Neural Networks (GNNs) are commonly evaluated by their plausibility, i.e., how well their explanations recover a predefined grou...

By Steve Azzolin, Francesco Paolo Nerini, Stefano Teso, Francesco Bonchi, Bruno Lepri, Andr\'e Panisson, Andrea Passerini

benchmarks

arXiv Computation and Language

Oct 5

Output Language Confusion under Multilingual Prompt Contamination

arXiv:2610.02926v1 Announce Type: new Abstract: Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multiling...

By Riju Marwah, Ritvik Garimella, Khusham Bansal, Atishay Jain, Amit Sheth

llms rag benchmarks safety

arXiv Computer Vision

Oct 5

SurGe: Improved Surface Geometry in Point Maps

arXiv:2605.31577v2 Announce Type: replace Abstract: Recent feedforward 3D reconstruction methods predict point maps and estimate global 3D geometry remarkably well. However, their predictions still e...

By Karim Knaebel, Gonzalo Martin Garcia, Christian Schmidt, Ilya Fradlin, Lucas Nunes, Daan de Geus, Bastian Leibe

benchmarks

arXiv AI

Oct 5

A Language Model from 1913: Pretraining on Historical Text

arXiv:2606.02991v2 Announce Type: replace-cross Abstract: While modern language models increasingly rely on ever-larger web corpora, we show that pretraining on historical text (e.g., pre-1913 text)...

By Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber, Yixuan Wang, Junchi Yu, Freda Shi, Philip Torr, Yao Lu

llms fine-tuning benchmarks data-engineering

arXiv AI

Oct 5

Coding Agents with Harness for Safe Robot Control

arXiv:2609.20822v2 Announce Type: replace-cross Abstract: Coding agents have emerged as a promising paradigm for robot manipulation: a language model writes the robot controller as a program, and age...

By Bingxin Xu, Yuzhang Shang, Zhen Dong, Emilio Ferrara

llms agents robotics benchmarks safety

arXiv Computation and Language

Oct 5

AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration

arXiv:2610.02770v1 Announce Type: new Abstract: Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let n...

By Hy Nguyen, Nabi Rezvani, Robin Vujanic

llms agents benchmarks data-engineering

arXiv AI

Oct 5

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

arXiv:2610.02976v1 Announce Type: new Abstract: Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about anoth...

By Hyunjae Ra, Aecheon Jung, Jungin Park, Sungeun Hong

llms multimodal benchmarks safety

arXiv Computation and Language

Oct 5

Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models

arXiv:2610.03077v1 Announce Type: new Abstract: Propaganda detection is an essential task in natural language processing (NLP), particularly in the context of manipulative political communications. H...

By Claudiu Creanga, Ioachim Lihor, Liviu P. Dinu

llms nlp fine-tuning benchmarks

arXiv Machine Learning

Oct 5

Trained Agentic Context Management

arXiv:2610.02404v1 Announce Type: new Abstract: We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simple...

By Bryce Sandlund

llms agents benchmarks

arXiv Machine Learning

Oct 5

ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

arXiv:2610.02363v1 Announce Type: new Abstract: Benchmarks for agent-generated data work grade a pipeline by running it once against a fixed snapshot. ArrivalBench instead re-executes the pipeline an...

By Pranay Kothari

agents benchmarks safety

Get the daily briefing