Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,285 stories · RSS feed

arXiv Machine Learning

Jul 2

IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

arXiv:2603. 16432v3 Announce Type: replace-cross Abstract: Unsupervised physical parameter estimation from video lacks a common benchmark: existing methods evaluate on non-overlapping synthetic data, the sole real-world dataset is restricted to single-body systems, and no established protocol addresses governing-equation identification.

By Rasul Khanbayov, Mohamed Rayan Barhdadi, Erchin Serpedin, Hasan Kurban

llms multimodal benchmarks

arXiv AI

Jul 2

TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification

arXiv:2508. 17519v3 Announce Type: replace-cross Abstract: Handling missing data in time series classification remains a significant challenge in various domains.

By YongKyung Oh, Dong-Young Lim, Sungil Kim, Alex Bui

benchmarks safety

arXiv AI

Jul 2

Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

arXiv:2607. 01002v1 Announce Type: cross Abstract: In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them.

By Aryo Pradipta Gema, Beatrice Alex, Pasquale Minervini

llms benchmarks

arXiv AI

Jul 2

SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing

arXiv:2607. 00208v1 Announce Type: cross Abstract: Reinforcement learning for diffusion large language models (dLLMs) has largely moved to trajectory-aware methods.

By Ruikang Zhao, Zhenting Wang, Han Gao, Ligong Han

llms diffusion reinforcement-learning benchmarks

arXiv AI

Jul 2

AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation

arXiv:2607. 00062v1 Announce Type: cross Abstract: High pass rates on established programming benchmarks such as HumanEval and LiveCodeBench do not always show whether a model can reason about algorithms.

By Xinyuan Song, Zekun Cai, Liang Zhao

llms benchmarks

arXiv AI

Jul 2

AGI Maze as a Benchmark Framework for World-Modeling Agents

arXiv:2607. 00627v1 Announce Type: new Abstract: Large language models (LLMs) are powerful pattern-completion systems, but their default operating mode - predicting the next token from a static context - does not reliably produce persistent, manipulable representations of an external world.

By Alexey Potapov

llms agents benchmarks

arXiv Machine Learning

Jul 2

Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

arXiv:2509. 17314v4 Announce Type: replace-cross Abstract: Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation.

By Juyeon Yoon, Somin Kim, Robert Feldt, Shin Yoo

llms benchmarks

arXiv AI

Jul 2

Multi-Label Node Classification with Label Influence Propagation

arXiv:2607. 00671v1 Announce Type: cross Abstract: Graphs are a complex and versatile data structure used across various domains, with possibly multi-label nodes playing a particularly crucial role.

By Yifei Sun, Zemin Liu, Bryan Hooi, Yang Yang, Rizal Fathony, Jia Chen, Bingsheng He

rag benchmarks

arXiv AI

Jul 2

EchoRisk: A Multicentre Echocardiography Dataset and Benchmark for Cardio-Oncology

arXiv:2607. 01039v1 Announce Type: cross Abstract: Therapy-induced cardiotoxicity is the leading non-oncological cause of treatment interruption in breast cancer patients, yet early, automated risk stratification from routine cardiac imaging remains an unsolved problem.

By Grigorios Kalliatakis, Georgia Karanasiou, Georgios Manikis, Manolis Tsiknakis, Dimitrios Fotiadis, Dorothea Tsekoura, Kalliopi Keramida, Vasileios Bouratzis, Lampros Lakkas, Katerina Naka, Andri Papakonstantinou, Anastasia Constantinidou, Kostas Marias

benchmarks

arXiv AI

Jul 2

Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity

arXiv:2607. 01153v1 Announce Type: cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model has followed an instruction, refused appropriately, complied with a policy, resisted an embedded command, or misreported progress in an agentic task.

By Brett Reynolds

llms agents benchmarks safety

arXiv AI

Jul 2

TRCGL-Net: A Long-Tailed Multi-Label Chest X-Ray Classification Framework with Generative Data Augmentation and Label Co-Occurrence Modeling

arXiv:2607. 00975v1 Announce Type: cross Abstract: Chest X-ray multi-label classification is a core task in intelligent medical imaging diagnosis.

By Tong Shao, Hongshun Ling, Li Zhang, Jinjing Wu, Junke Wang, Yuan Gao, Fang Wang

diffusion benchmarks

arXiv AI

Jul 2

LLM-Guided ODE Discovery and Parameter Inference from Small-Cohort Aggregate Data

arXiv:2607. 00733v1 Announce Type: cross Abstract: Mechanistic modeling via ordinary differential equations (ODEs) provides interpretable descriptions of complex dynamics and enables inference of underlying mechanisms, which is particularly valuable in clinical settings.

By Hanning Yang, Meropi Karakioulaki, Lennart Purucker, Tim Litwin, Cristina Has, Moritz Hess

llms agents benchmarks safety

arXiv Machine Learning

Jul 2

Interpretable vs Learned Encoders for High-Cardinality Fraud Detection

arXiv:2607. 00477v1 Announce Type: new Abstract: A total of seven categorical encoding methods were tested on the IEEE-CIS fraud benchmark dataset (590,540 records, 3.

By Xiao Han, Jingjing Liu, Moxuan Zheng, Zhen Zhang, Chenyu Wu

rag benchmarks

arXiv AI

Jul 2

An LLM-Based Framework for Intent-Driven Network Topology Design

arXiv:2607. 00292v1 Announce Type: cross Abstract: Designing deployable and resilient network topologies from natural language requirements remains a challenging problem in network automation.

By Kholoud El-Habbouli, Fen Zhou, Stephane Huet

llms benchmarks

arXiv Machine Learning

Jul 2

StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

arXiv:2607. 00465v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities.

By Yuan Qing, Chengzhi Mao, Boqing Gong

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 2

EcoGEO: Trajectory-Aware Evidence Ecosystems for Web-Enabled LLM Search Agents

arXiv:2605. 12887v2 Announce Type: replace-cross Abstract: Web-enabled LLM agents are changing how online information influences search outcomes.

By Hengwei Ye, Jiasheng Mao, Zhenhan Guan, Zheng Tian

llms agents benchmarks

arXiv Machine Learning

Jul 2

Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Captures

arXiv:2605. 04035v3 Announce Type: replace-cross Abstract: We propose HeadsUp, a scalable feed-forward method for reconstructing high-quality 3D Gaussian heads from large-scale multi-camera setups.

By Evangelos Ntavelis, Sean Wu, Mohamad Shahbazi, Fabio Maninchedda, Dmitry Kostiaev, Artem Sevastopolsky, Vittorio Megaro, Trevor Phillips, Alejandro Blumentals, Shridhar Ravikumar, Mehak Gupta, Reinhard Knothe, Jeronimo Bayer, Matthias Vestner, Simon Schaefer, Thomas Etterlin, Christian Zimmermann, Alexey Artemov, Mathias Deschler, Peter Kaufmann, Stefan Brugger, Sebastian Martin, Brian Amberg, Tom Runia

diffusion benchmarks

arXiv Machine Learning

Jul 2

Distributionally Robust Linear Regression With Block Lewis Weights

arXiv:2607. 00252v1 Announce Type: new Abstract: We present an algorithm for the group distributionally robust (GDR) least squares problem.

By Naren Sarayu Manoj, Kumar Kshitij Patel

benchmarks

arXiv AI

Jul 2

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

arXiv:2607. 00700v1 Announce Type: cross Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging.

By Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang, Junjie Chen

llms agents benchmarks

arXiv Machine Learning

Jul 2

TallyTrain: Communication-Efficient Federated Distillation

arXiv:2607. 00173v1 Announce Type: new Abstract: Federated learning is bandwidth-bound on two orthogonal axes: model size, which limits how often parameter-averaging methods can afford to merge, and class count, which makes per-probe soft-label distillation prohibitive at large vocabularies.

By Radhakrishna Achanta, Will Reed

efficiency benchmarks

Get the daily briefing