Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,285 stories · RSS feed

arXiv AI

Jul 1

Surrogate-Gated Generation and Foundation-Model Embeddings for Bayesian Materials Design

arXiv:2606. 28578v1 Announce Type: cross Abstract: Closed-loop materials discovery iterates between proposing candidate structures and evaluating their properties, and property evaluation dominates the cost.

By Sk Md Ahnaf Akif Alvi, Jan Janssen, Danny Perez, Douglas Allaire, Raymundo Arroyave

rag diffusion reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 1

Cross-Modal Hierarchical Fusion for from Multi-Sensor Ground Observation

arXiv:2606. 30647v1 Announce Type: cross Abstract: Dense volumetric reconstruction of cloud microphysical fields from sparse ground-based instruments remains an open problem, largely because the available measurements are heterogeneous in both modality and spatial coverage.

By Xinze Zhang

multimodal benchmarks

arXiv AI

Jul 1

Qualified Educational Capacity Planning under Heterogeneous Student Support Needs: A Synthetic Benchmark and Decision-Support Framework

arXiv:2606. 30650v1 Announce Type: cross Abstract: Educational support services often face a qualified-capacity problem: staff time is scarce, qualifications decay, new support needs can appear before anyone is prepared for them, and training consumes the same hours needed by current students.

By Carlos Eduardo Sanoja, Oscar Enrique Moreno Mayz

benchmarks safety

arXiv AI

Jul 1

ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection

arXiv:2606. 30646v1 Announce Type: cross Abstract: Speech recruits the same executive, attentional, and working memory processes underlying instrumental activities of daily living, or IADLs, providing a non-invasive proxy for cognitive assessment.

By Chukwuemeka Ugwu, Oluwafemi Richard Oyeleke

llms rag multimodal benchmarks

arXiv AI

Jul 1

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

arXiv:2606. 30686v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulation benchmarks.

By Taozhao Chen, Ian Manchester, Huaming Chen

llms robotics multimodal benchmarks

arXiv AI

Jul 1

FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agents

arXiv:2606. 31522v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly deployed as autonomous financial agents initialized with explicit behavioral mandates such as "preserve capital" or "avoid speculative bets" that are meant to govern every decision throughout deployment.

By Muhammad Usman Safder (Steve), Ayesha Gull (Steve), Rania Elbadry (Steve), Fan Zhang (Steve), Yankai Chen (Steve), Xueqing Peng (Steve), Xue (Steve), Liu, Preslav Nakov, Zhuohan Xie

llms agents benchmarks

arXiv AI

Jul 1

Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning

arXiv:2606. 31599v1 Announce Type: cross Abstract: Vision-language models (VLMs) combining reinforcement learning (RL) ignite remarkable progress in multimodal reasoning, yet still struggle with medical images, which typically exhibit extremely sparse visual evidence to inform clinical decision-making.

By Kaitao Chen, Weiqian Zhao, Jiamin Wu, Qihao Zheng, Shangquan Sun, Chunfeng Song, Xiaosong Wang, Mu Zhou, Mianxin Liu

llms nlp reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jul 1

Sparsity-Inducing Divergence Losses for Biometric Verification

arXiv:2606. 31664v1 Announce Type: cross Abstract: Performance in face and speaker verification is largely driven by margin-penalty softmax losses such as CosFace and ArcFace.

By Dimitrios Koutsianos, Ladislav Mo\v{s}ner, Yannis Panagakis, Themos Stafylakis

rag efficiency benchmarks

arXiv Machine Learning

Jul 1

The Decomposition Is the Fingerprint: Per-Component Identity for Agent Skills

arXiv:2606. 31272v1 Announce Type: cross Abstract: AI agents increasingly acquire and execute skills at runtime: bundles of prompt instructions, executable code, and tool declarations fetched from marketplaces and other agents.

By Hongliang Liu, Yuhao Wu, Tung-Ling Li

rag agents benchmarks safety

arXiv Machine Learning

Jul 1

Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models

arXiv:2606. 31511v1 Announce Type: cross Abstract: In deployment settings where retraining is infeasible, small frozen code models are routinely asked to repair a failed program after seeing their own failing output, usually treated as a retry mechanism.

By Mehmet Iscan

benchmarks

arXiv Machine Learning

Jul 1

An Empirical Study of Security Calibration in Large Language Models for Code

arXiv:2606. 31159v1 Announce Type: cross Abstract: Large Language Models (LLMs) are rapidly transforming software development, yet their use in security-critical contexts raises a key question: do models know when their generated code is insecure?

By Mohammed Latif Siddiq, Md. Nafiu Rahman, Joanna C. S. Santos

llms benchmarks

arXiv AI

Jul 1

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

arXiv:2606. 32032v1 Announce Type: cross Abstract: Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes.

By Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 1

The HydroGym Reinforcement Learning Platform for Fluid Dynamics

arXiv:2512. 17534v2 Announce Type: replace-cross Abstract: Modeling and controlling fluids is critical across science and engineering.

By Christian Lagemann, Sajeda Mokbel, Miro Gondrum, Mario R\"uttgers, Yuning Wang, Pol Su\'arez, Ludger Paehler, Deniz A. Bezgin, Aaron B. Buhendwa, Jared L. Callaham, Samuel Ahnert, Nicholas Zolman, Xiao Shao, Jean-Christophe Loiseau, Nikolaus Adams, Matthias Meinke, Wolfgang Schr\"oder, Kai Lagemann, Esther Lagemann, Ricardo Vinuesa, Steven L. Brunton

agents reinforcement-learning robotics benchmarks

arXiv AI

Jul 1

RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

arXiv:2604. 19047v2 Announce Type: replace-cross Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity.

By Hanjun Cho, Jay-Yoon Lee

llms rag benchmarks

arXiv AI

Jul 1

REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent

arXiv:2511. 17442v3 Announce Type: replace-cross Abstract: Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines.

By Binger Chen, Tacettin Emre B\"ok, Behnood Rasti, Volker Markl, Beg\"um Demir

llms agents computer-vision nlp multimodal benchmarks

arXiv AI

Jul 1

QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

arXiv:2606. 32034v1 Announce Type: cross Abstract: LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions.

By Sergio Hern\'andez-Guti\'errez, Matteo Merler, Ilze Amanda Auzina, Joschka Str\"uber, Ameya Prabhu, Matthias Bethge

llms rag agents efficiency benchmarks

arXiv AI

Jul 1

MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments

arXiv:2606. 31966v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored.

By Qingyun Liu, Jiwen Zhang, Jingyi Hu, Siyuan Wang, Zhongyu Wei

llms agents robotics multimodal benchmarks

arXiv AI

Jul 1

FMA-Net++: Motion- and Exposure-Aware Joint Video Super-Resolution and Deblurring

arXiv:2512. 04390v2 Announce Type: replace-cross Abstract: Joint video super-resolution and deblurring (VSRDB) requires both efficient long-range temporal modeling and robustness to frame-wise exposure-duration variation, which changes the extent of motion blur across video frames.

By Geunhyuk Youk, Jihyong Oh, Munchurl Kim

rag efficiency benchmarks

arXiv Machine Learning

Jul 1

Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again

arXiv:2606. 30995v1 Announce Type: new Abstract: Recent work has shown that well-optimized individual decision trees can match complex black box models in some settings, primarily in noisy domains.

By Zakk Heile, Hayden McTavish, Margo Seltzer, Cynthia Rudin

benchmarks safety

arXiv AI

Jul 1

Visual Prompt Discovery via Semantic Exploration

arXiv:2603. 16250v2 Announce Type: replace-cross Abstract: LVLMs encounter significant challenges in image understanding and visual reasoning, leading to critical perception failures.

By Jaechang Kim, Yotaro Shimose, Zhao Wang, Kuang-Da Wang, Jungseul Ok, Shingo Takamatsu

agents robotics benchmarks

Get the daily briefing