Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,930 stories · RSS feed

arXiv AI

Jun 30

Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

arXiv:2601. 05366v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls.

By Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani, Wanpeng Xu, Hua Wei, Xiyang Hu

llms agents benchmarks

arXiv Machine Learning

Jun 30

MALOQ: Massively Accelerated Learning of Operators for Quantum Transport

arXiv:2606. 28911v1 Announce Type: new Abstract: Machine-learned (ML) operator models can be trained to predict density functional theory (DFT) Hamiltonian/density matrices at significantly reduced computational cost, thus extending electronic-structure calculations to previously unfeasible scales.

By Manasa Kaniselvan, Alexander Maeder, Denghui Lu, Alexandros Nikolaos Ziogas, Mathieu Luisier

benchmarks

arXiv AI

Jun 30

On the Spectral Structure and Objective Equivalence of Orthogonal Multilabel Fisher Discriminants

arXiv:2605. 03283v2 Announce Type: replace-cross Abstract: We provide a unified theoretical analysis of Linear Discriminant Analysis with simultaneous multilabel scatter matrix formulations and Stiefel orthogonality constraints.

By Brian Keith-Norambuena, Juan Bekios-Calfa

benchmarks

arXiv AI

Jun 30

Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition

arXiv:2605. 13087v2 Announce Type: replace-cross Abstract: Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance.

By Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon, Arghya Bhattacharya, Kumarmanas Nethil

fine-tuning multimodal benchmarks

arXiv AI

Jun 30

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

arXiv:2603. 29139v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks.

By Kuangshi Ai, Haichao Miao, Kaiyuan Tang, Nathaniel Gorski, Jianxin Sun, Guoxi Liu, Helgi I. Ingolfsson, David Lenz, Hanqi Guo, Hongfeng Yu, Teja Leburu, Michael Molash, Bei Wang, Tom Peterka, Chaoli Wang, Shusen Liu

llms agents multimodal benchmarks

arXiv AI

Jun 30

Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models

arXiv:2603. 22876v2 Announce Type: replace-cross Abstract: Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets.

By Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang, Sheng Xu, Bo Yue, Zhizheng Wu, Guiliang Liu

reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jun 30

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

arXiv:2605. 30295v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent settings remains limited.

By Valentina Bui Muti, Eug\'enie Dulout, Ziquan Fu

llms benchmarks

arXiv AI

Jun 30

Selective Memory Retention for Long-Horizon LLM Agents

arXiv:2606. 29178v1 Announce Type: new Abstract: When does retention matter for memory-augmented LLM agents?

By Pranath Reddy

llms rag agents benchmarks

arXiv Machine Learning

Jun 30

Efficient 3D Gaussian Splatting with Axis-Shared Rasterization and Order-independent Transmittance

arXiv:2506. 07069v2 Announce Type: replace-cross Abstract: 3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, combining high-quality reconstruction with efficient rendering.

By Zhican Wang, Guanghui He, Lingjun Gao, Dantong Liu, Shell Xu Hu, Chen Zhang, Zhuoran Song, Nicholas Lane, Hongxiang Fan

agents robotics benchmarks

arXiv AI

Jun 30

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

arXiv:2606. 29784v1 Announce Type: cross Abstract: Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity.

By Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang

benchmarks safety

arXiv AI

Jun 30

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

arXiv:2606. 28707v1 Announce Type: new Abstract: Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms reinforcement-learning benchmarks

arXiv AI

Jun 30

CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigation

arXiv:2606. 28397v1 Announce Type: cross Abstract: Vision-language navigation (VLN) has recently advanced with large language and multimodal models, enabling agents to follow natural-language instructions in unseen environments without training a task-specific navigation policy.

By Shaoxuan Li, Xiangyu Dong, Xiaoguang Ma, Junfeng Chen, Haoran Zhao, Yaoming Zhou

rag agents multimodal benchmarks

arXiv AI

Jun 30

Assessing the Business Process Modeling Competences of Large Language Models

arXiv:2601. 21787v2 Announce Type: replace-cross Abstract: The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions.

By Chantale Lauer, Peter Pfeiffer, Alexander Rombach, Nijat Mehdiyev

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 30

KrishokChat: A Citation-Grounded Dataset and Benchmark for Bengali Agricultural Advisory

arXiv:2606. 29243v1 Announce Type: new Abstract: We present KrishokChat, the first citation-grounded Bengali agricultural instruction-tuning dataset for crop advisory in low-resource settings.

By Khan Raiyan Ibne Reza, Omar Ibne Shahid

rag fine-tuning benchmarks safety

arXiv AI

Jun 30

How to Train Your Long-Context Visual Document Model

arXiv:2602. 15257v3 Announce Type: replace-cross Abstract: We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text.

By Austin Veselka

llms nlp fine-tuning benchmarks

arXiv AI

Jun 30

IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

arXiv:2606. 28556v1 Announce Type: new Abstract: Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging.

By Maria Xenochristou, Ashutosh Joshi, Korosh Vatanparvar, Mohammad Abuzar Hashemi, Prasad Kasu, Deepak Bansal, Anchal Nema, Nivedita Wadhwa, Prashams S Jain, Rebecca Abraham, Will Kimbrough, Dilek Hakkani-Tur, Wilko Schulz-Mahlendorf

llms multimodal benchmarks safety

arXiv AI

Jun 30

GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes

arXiv:2606. 28514v1 Announce Type: new Abstract: Multimodal models are increasingly deployed to solve tasks collaboratively with humans or other artificial agents.

By Amit Parekh, Sabrina McCallum, Kareem Al-Hasan, Malvina Nikandrou, Alessandro Suglia, Ioannis Konstas

agents multimodal benchmarks

arXiv Machine Learning

Jun 30

Golden Hour Divide: Trauma Care Accessibility and Resource Vulnerability in Sri Lanka

arXiv:2606. 29889v1 Announce Type: new Abstract: Timely intensive care dictates survival, yet emergency infrastructure remains unevenly distributed across Sri Lanka.

By Sonath Kirindage, Vihanga Nimsara, Sakindu Rajapaksa, Kavyanga Hathurusinghe, Lahiru Dilshan, Subavarshana Arumugam, Nathali Athukorala, Sandareka Wickramanayake, Nisansa de Silva

benchmarks

arXiv Machine Learning

Jun 30

ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data

arXiv:2408. 16028v4 Announce Type: replace-cross Abstract: Supervised-learning-based vulnerability detectors often fall short due to limited labelled training data.

By Weizhou Wang, Eric Liu, Xiangyu Guo, Xiao Hu, Ilya Grishchenko, David Lie

llms benchmarks

arXiv Machine Learning

Jun 30

Alternative Graph Neural Networks: Synergizing GEV Models and Deep Learning for Travel Mode Choice Modeling

arXiv:2509. 07123v2 Announce Type: replace-cross Abstract: Generalized extreme value models capture dependence among choice alternatives in discrete choice modeling, but require this dependence to be predefined, symmetric, and shared uniformly across individuals.

By Yuqi Zhou, Zhanhong Cheng, Dingyi Zhuang, Lingqian Hu, Yuheng Bu, Shenhao Wang

benchmarks

Get the daily briefing