Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,633 stories · RSS feed

arXiv AI

Jul 21

Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Lift

arXiv:2607. 17384v1 Announce Type: new Abstract: This paper provides an experimentally verified formal law for calculating the uplift that diversity of thought provides in Large Language Model (LLM) ensembles.

By Junade Ali

llms agents benchmarks

arXiv AI

Jul 21

Lomekwi: Resource-Bounded Tool Discovery in LLM Agents

arXiv:2607. 16961v1 Announce Type: new Abstract: Existing tool-use benchmarks report a single success rate for complex, multistep tasks.

By Roshan Klein-Seetharaman, Daniel Wang, Andrew Xu

llms agents benchmarks

arXiv AI

Jul 21

Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models

arXiv:2607. 17779v1 Announce Type: new Abstract: Text-to-Image (T2I) generative models have achieved remarkable progress in synthesizing high-quality visual content, yet they remain vulnerable to adversarial misuse, particularly in generating Not-Safe-For-Work (NSFW) images.

By Dongdong Yang, Deyue Zhang, Zhao Liu, Zonghao Ying, Wenzhuo Xu, Jiankai Jin, Xiangzheng Zhang, Quanchen Zou

llms diffusion multimodal benchmarks safety

arXiv AI

Jul 21

Symmetric Behavior Regularized Policy Optimization

arXiv:2508. 04225v4 Announce Type: replace-cross Abstract: Behavior Regularized Policy Optimization (BRPO) leverages asymmetric divergence regularization to mitigate distribution shift in offline reinforcement learning.

By Lingwei Zhu, Haseeb Shah, Zheng Chen, Martha White

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 21

ECG-LLM: Foundation Model for ECG-Based Cardiac Reasoning

arXiv:2607. 16323v1 Announce Type: cross Abstract: Electrocardiography (ECG) is an inexpensive, standard-of-care test for cardiac symptoms, but front-line triage often lacks immediate access to definitive imaging such as echocardiography (ECHO) or cardiac magnetic resonance (CMR).

By Alexander Selivanov, Friederike Jungmann, Jan Kehrer, Karl-Ludwig Laugwitz, Eimo Martens, Daniel Rueckert

llms multimodal benchmarks

arXiv AI

Jul 21

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

arXiv:2607. 16243v1 Announce Type: cross Abstract: Multimodal industrial anomaly inspection assistants are a critical component of next-generation smart factories, enabling interactive vision-language-based querying.

By Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

llms multimodal benchmarks safety

arXiv AI

Jul 21

Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

arXiv:2602. 03061v2 Announce Type: replace-cross Abstract: Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms.

By Zihan Dong, Zhixian Zhang, Yang Zhou, Can Jin, Ruijia Wu, Linjun Zhang

llms benchmarks

arXiv AI

Jul 21

Just A Rather Very Intelligent Spoken Agent

arXiv:2607. 16610v1 Announce Type: new Abstract: Long-horizon AI agents are becoming increasingly capable, yet their interaction with users remains surprisingly thin.

By Chen Chen, Zhehuai Chen

llms agents benchmarks

arXiv AI

Jul 21

Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

arXiv:2607. 17673v1 Announce Type: cross Abstract: Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs.

By Tillmann Rheude, Roland Eils, Benjamin Wild

multimodal benchmarks safety

arXiv AI

Jul 21

Fantastic Adaptive Taxonomies and How to Use Them

arXiv:2607. 16387v1 Announce Type: cross Abstract: An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback.

By Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica

llms agents benchmarks

arXiv AI

Jul 21

AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Language

arXiv:2607. 16372v1 Announce Type: cross Abstract: Interactive theorem proving (ITP) underpins program verification and formalized mathematics, but its manual effort limits scalability.

By Qiyuan Xu, Joshua Ong Jun Leang, Renxi Wang, Wenda Li, Haonan Li, Luke Ong, Conrad Watt

llms agents benchmarks

arXiv AI

Jul 21

Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

arXiv:2607. 16448v1 Announce Type: cross Abstract: Interpretability methods for neural network activations span a wide cost spectrum, from cheap, training-free techniques (such as linear probes, PCA, SVD) to more expensive training-based ones (such as SAEs and activation oracles).

By Sriram Balasubramanian, Soheil Feizi

llms rag agents diffusion benchmarks safety

arXiv AI

Jul 21

Geometry-Enhanced Portion Estimation for Multimodal LLMs

arXiv:2607. 16514v1 Announce Type: cross Abstract: Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker.

By Lin Liao, Peng Li

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

arXiv:2607. 16476v1 Announce Type: cross Abstract: Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade.

By Chao Jiang, Yulong Ye, Tao Chen, Miqing Li

benchmarks

arXiv AI

Jul 21

Building2Building: A Large Scale Benchmark for Generalizable Real-World Reinforcement Learning

arXiv:2607. 16534v1 Announce Type: cross Abstract: Reinforcement learning (RL) has achieved strong results in control, yet learned policies remain brittle to changes in dynamics, action spaces, observation spaces, or goals, a critical limitation for real-world deployment.

By Vincent Taboga, Justin Veilleux, Doseok Jang, Anushree Rankawat, Pierre-Luc Bacon

reinforcement-learning benchmarks

arXiv AI

Jul 21

PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization

arXiv:2607. 16206v1 Announce Type: new Abstract: This paper introduces PPO-HSC (Proximal Policy Optimization with High-order Sampling Coverage), an exploratory reinforcement learning framework designed to address the "Invisible Shackles" of mode collapse in Large Language Model (LLM) fine-tuning.

By Yujie Shen, Haowen Chen

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 21

SCA: Segment-Wise CoT Compression with Answer Alignment

arXiv:2603. 07598v2 Announce Type: replace Abstract: Chain-of-thought (CoT) reasoning improves problem solving, but long think traces increase inference cost.

By Ye Tian, Hongyu Lin

benchmarks safety

arXiv AI

Jul 21

Pediatric Bone Age Prediction Using Deep Learning

arXiv:2607. 16936v1 Announce Type: cross Abstract: Pediatric bone age prediction is a crucial task in clinical practice that can help diagnose endocrine disorders and provide insight into a child's growth and development.

By Al Zadid Sultan Bin Habib, Md. Ekramul Islam, Md Asif Bin Syed, Md Younus Ahamed, Tanpia Tasnim

computer-vision benchmarks

arXiv AI

Jul 21

DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines

arXiv:2607. 16617v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not automatically materialized as persistent, editable platform artifacts.

By Runming He, Zhen Hao Wong, Hao Liang, Zimo Meng, Chengyu Shen, Xiaochen Ma, Wentao Zhang

llms agents benchmarks

arXiv AI

Jul 21

Privacy Cost as Equity Input: A Group Fairness Criterion for Differentially Private Machine Learning

arXiv:2607. 16620v1 Announce Type: cross Abstract: Differential privacy (DP) is increasingly deployed to limit membership inference risk in machine-learning systems.

By Rakshit Naidu

nlp benchmarks safety

Get the daily briefing