Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 15

Same-Origin Policy for Agentic Browsers

arXiv:2606. 14027v1 Announce Type: cross Abstract: Agentic browsers integrate autonomous AI agents into web browsers, enabling users to accomplish web tasks through natural-language instructions.

By Xilong Wang, Xiaoxing Chen, Patrick Li, Dawn Song, Neil Gong

agents benchmarks

arXiv AI

Jun 15

COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

arXiv:2512. 02318v4 Announce Type: replace-cross Abstract: This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA.

By Junyu Wang, Changjia Zhu, Yuanbo Zhou, Lingyao Li, Xu He, Mingkui Wei, Junjie Xiong

llms multimodal benchmarks

arXiv Machine Learning

Jun 15

Aligned but Stereotypical? How System Prompts Shape Demographic Bias in LLM-Based Text-to-Image Models

arXiv:2512. 04981v2 Announce Type: replace-cross Abstract: Text-to-image (T2I) systems increasingly rely on Large Language Model (LLM)-based text conditioning to interpret and expand user prompts.

By NaHyeon Park, Na Min An, Kunhee Kim, Soyeon Yoon, Jiahao Huo, Hyunjung Shim

llms rag diffusion benchmarks safety

arXiv Machine Learning

Jun 15

Robin-Neumann Coupling of PINN and FEM Solvers: A Steklov-Poincar\'e View, with Application to Fluid-Structure Interaction with Contact

arXiv:2606. 14181v1 Announce Type: cross Abstract: Physics-informed neural networks (PINNs) are meshless and carry moving geometry and topology change through resampling of collocation points; the finite-element method (FEM) is the workhorse for boundary-fitted discretisations.

By Mikel Landajuela

benchmarks

arXiv AI

Jun 15

Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH

arXiv:2606. 13994v1 Announce Type: cross Abstract: LLM-based Agents are becoming increasingly capable and widely deployed, creating growing incentives for adversarial misuse in the real-world.

By Vikhyath Kothamasu, Virginia Smith, Chhavi Yadav

llms agents benchmarks safety

arXiv Machine Learning

Jun 15

Shuttling Compiler for Trapped-Ion Quantum Computers Based on Large Language Models

arXiv:2512. 18021v3 Announce Type: replace-cross Abstract: We present the first shuttling compiler based on large language models (LLMs) for trapped-ion quantum computers, where qubits are shuttled between segments for gate execution and qubit storage.

By Fabian Kreppel, Reza Salkhordeh, Ferdinand Schmidt-Kaler, Andr\'e Brinkmann

llms fine-tuning benchmarks

arXiv AI

Jun 15

How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?

arXiv:2606. 13896v1 Announce Type: cross Abstract: Self-supervised geospatial foundation models (GeoFMs) learn transferable representations from remote sensing data, but their downstream behavior is difficult to characterize.

By Julia Romero, Qin Lv, Morteza Karimzadeh

llms rag computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jun 15

SANA: What Matters for QA Agents over Massive Data Lakes?

arXiv:2606. 13904v1 Announce Type: cross Abstract: Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results.

By Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu

llms agents nlp benchmarks

arXiv AI

Jun 15

From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrails

arXiv:2606. 14517v1 Announce Type: cross Abstract: LLM-based guardrails have emerged as a highly effective defense against prompt injection and jailbreak attacks in autonomous agents.

By Yuguang Zhou, Xunguang Wang, Pingchuan Ma, Zhantong Xue, Zhaoyu Wang, Shuai Wang

llms agents benchmarks safety

arXiv Machine Learning

Jun 15

SpikF-GO: Spiking Fourier Graph Operators for Multivariate Time Series Forecasting

arXiv:2606. 13901v1 Announce Type: new Abstract: Spiking Neural Networks (SNNs) have emerged as an energy-efficient alternative to conventional neural networks, demonstrating strong performance in computer vision and robotics.

By Jafar Bakhshaliyev, Niels Landwehr

rag computer-vision robotics benchmarks

arXiv Machine Learning

Jun 15

EquiDexFlow: Contact-Grounded SE(3)-Equivariant Dexterous Grasp Generative Flows

arXiv:2606. 12728v2 Announce Type: replace-cross Abstract: Most learned dexterous grasp generators relegate contact forces to a downstream verification step, so a kinematically-plausible pose can still violate the conditions for a stable physical grasp.

By Clinton Enwerem, John S. Baras, Calin Belta

robotics benchmarks

arXiv Machine Learning

Jun 15

Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

arXiv:2606. 14397v1 Announce Type: new Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities.

By Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel, Michal Zakrzewski, Sebastian Montagna, Damian Rynczak, Shreyansh Padarha, Kumail Alhamoud, Zihao Fu, William Lugoloobi, Kai Rawal, Hanna Yershova, Xander Davies, Taras Rumezhak, Guohao Li, Fazl Barez, Baoyuan Wu, Arkadiusz Drohomirecki, Yarin Gal, Chris Russell, Christopher Summerfield, Adam Mahdi, Volodymyr Karpiv, Philip Torr, Adel Bibi

agents benchmarks

arXiv AI

Jun 15

StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance

arXiv:2606. 14571v1 Announce Type: new Abstract: A central role of personal-agent memory is to turn stored information and prior interactions into future-oriented assistance.

By Guanming Liu, Yuqi Ren, Hansu Gu, Peng Zhang, Weihang Wang, Jiahao Liu, Ning Gu, Tun Lu

agents benchmarks

arXiv Machine Learning

Jun 15

What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective

arXiv:2606. 14299v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) such as CLIP have become a standard backbone for open-vocabulary recognition, yet their zero-shot predictions remain vulnerable to distribution shifts encountered at deployment.

By Jiazhen Huang, Xiao Chen, Zhiming Liu, Yaru Sun, Jingyan Jiang, Zhi Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 15

Graph Diffusion Residuals for Control-Function Instrumental Variables

arXiv:2606. 14636v1 Announce Type: new Abstract: Control-function instrumental variable estimators need a first-stage residual, not merely a first-stage prediction.

By Rui Wu, Zongyuan Chen, Hong Xie, Defu Lian, Enhong Chen

diffusion benchmarks

arXiv Machine Learning

Jun 15

EM-NeSy: Expectation Maximization for Neurosymbolic Learning

arXiv:2606. 14463v1 Announce Type: new Abstract: Neurosymbolic (NeSy) models integrate neural networks and symbolic reasoning for robust and interpretable AI.

By Annegret Seibt, Luc De Raedt, Giuseppe Marra

benchmarks

arXiv AI

Jun 15

ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning

arXiv:2606. 14697v1 Announce Type: cross Abstract: Building trustworthy medical multimodal large language models (MLLMs) is critical for reliable clinical decision support.

By Sicheng Yang, Hangjie Yuan, Wenjun Zhang, Jinwang Wang, Yichen Qian, Weihua Chen, Fan Wang, Lei Zhu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 15

Quantile-Free Uncertainty Quantification in Graph Neural Networks

arXiv:2605. 04847v2 Announce Type: replace-cross Abstract: Uncertainty quantification (UQ) in graph neural networks (GNNs) is crucial in high-stakes domains but remains a significant challenge.

By Soyoung park, Hwanjun Song, Sungsu Lim

benchmarks

arXiv Machine Learning

Jun 15

NEST3D: A High-Resolution Multimodal Dataset of Sociable Weaver Tree Nests

arXiv:2606. 14562v1 Announce Type: cross Abstract: Sociable weaver nests function as complex ecological structures offering thermoregulatory microhabitats and sustaining diverse species; however, datasets used in prior studies lack fine-grained 3D structural detail.

By Constanza A. Molina Catricheo, Simon Boeder, Ting-Jia Guo, Giacomo May, Cl\'ement Berthelot, Devis Tuia, Friedrich Fedor Reinhard, Fabio Remondino, Benjamin Risse

llms computer-vision robotics multimodal benchmarks

arXiv AI

Jun 15

ANSR-DT: A Neuro-Symbolic Framework for Adaptive and Explainable Digital Twins

arXiv:2501. 08561v4 Announce Type: replace Abstract: Digital twins are increasingly used to monitor and optimize industrial systems, yet many existing frameworks remain difficult to interpret, slow to adapt, and limited in their ability to incorporate explicit domain knowledge.

By Safayat Bin Hakim, Muhammad Adil, Alvaro Velasquez, Houbing Herbert Song

reinforcement-learning benchmarks safety

Get the daily briefing