Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,941 stories · RSS feed

arXiv AI

Jun 8

SlimSearcher: Training Efficiency-Aware Web Agents via Adaptive Reward Gating

arXiv:2606. 07074v1 Announce Type: cross Abstract: Deep research agents have demonstrated remarkable capabilities in complex information-seeking tasks, yet this power comes at a steep computational cost.

By Zequn Xie, Junjie Wang, Dan Yang, Jie Feng, Yue Shen, Jian Wang, Jinjie Gu

agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 8

MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

arXiv:2605. 05225v3 Announce Type: replace-cross Abstract: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect.

By Bo Li, Chuan Wu, Shaolin Zhu

llms multimodal benchmarks

arXiv Machine Learning

Jun 8

CoMetaPNS: Continually Meta-learning Personalized Neural Surrogates for Cardiac Electrophysiology Simulations

arXiv:2606. 07488v1 Announce Type: new Abstract: Personalized virtual heart simulations face challenges in model personalization and computational cost.

By Ryan Missel, Xiajun Jiang, Linwei Wang

benchmarks

arXiv AI

Jun 8

GP-Adapter: Gaussian Process CLIP-Adapter for Few-Shot Out-of-Distribution Detection

arXiv:2606. 07102v1 Announce Type: cross Abstract: We propose GP-Adapter, a training-free framework that augments CLIP (Contrastive Language-Image Pre-training) with Gaussian Process (GP) uncertainty modeling for few-shot classification and out-of-distribution (OOD) detection.

By Taisei Saito, Koretaka Ogata, Takafumi Hiroi

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jun 8

Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs

arXiv:2601. 12359v1 Announce Type: cross Abstract: Prompt injection attacks have become an increasing vulnerability for LLM applications, where adversarial prompts exploit indirect input channels such as emails or user-generated content to circumvent alignment safeguards and induce harmful or unintended outputs.

By Anirudh Sekar, Mrinal Agarwal, Rachel Sharma, Akitsugu Tanaka, Jasmine Zhang, Arjun Damerla, Kevin Zhu

llms rag benchmarks safety

arXiv AI

Jun 8

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

arXiv:2606. 07462v1 Announce Type: new Abstract: As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution.

By Jiayu Wang, Weijiang Lv, Bowen Fu, Jing Fu, Jiayi Song, Lingyu Zhang, Lanxuan Xue, Luodi Chen, Zepeng Xin, Kaiyu Li, Xiangyong Cao

llms agents benchmarks

arXiv AI

Jun 8

When Does Multi-Agent Collaboration Help? An Entropy Perspective

arXiv:2602. 04234v6 Announce Type: cross Abstract: Multi-agent systems (MAS) have emerged as a prominent paradigm for leveraging large language models (LLMs) to tackle complex tasks.

By Yuxuan Zhao, Sijia Chen, Ningxin Su

llms agents benchmarks

arXiv AI

Jun 8

dots.tts Technical Report

arXiv:2606. 07080v1 Announce Type: cross Abstract: We present dots.

By Shi Lian, Changtao Li, Bohan Li, Hankun Wang, Da Zheng, Junfeng Tian, Yufeng Ma, Colin Zhang, Kai Yu

diffusion efficiency multimodal benchmarks

arXiv AI

Jun 8

It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents

arXiv:2512. 23128v2 Announce Type: replace-cross Abstract: Web-based agents powered by large language models are increasingly used for tasks such as email management or professional networking.

By Karolina Korgul, Yushi Yang, Arkadiusz Drohomirecki, Piotr B{\l}aszczyk, Will Howard, Lukas Aichberger, Chris Russell, Philip H. S. Torr, Adam Mahdi, Adel Bibi

llms agents benchmarks safety

arXiv Machine Learning

Jun 8

Closed-Form Spectral Regularization for Multi-Task Model Merging

arXiv:2606. 07289v1 Announce Type: new Abstract: Model merging combines several independently fine-tuned experts into a single multi-task model without any training data, reducing the storage, serving, and decentralized-development costs of large foundation models.

By Yongxian Wei, Runxi Cheng, Xingxuan Zhang, Li Shen, Chun Yuan, Peng Cui, Dacheng Tao

computer-vision fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 8

Physics Guided Generative Optimization for Trotter Suzuki Decomposition

arXiv:2605. 13268v2 Announce Type: replace-cross Abstract: Trotter Suzuki product formulas are the standard route to Hamiltonian evolution on noisy intermediate-scale quantum (\NISQ{}) hardware, but their accuracy depends on three coupled choices: term grouping, product-formula order, and time-step allocation.

By WenBin Yan

diffusion fine-tuning benchmarks

arXiv AI

Jun 8

Proxy Reconstruction Pre-training for Ramp Flow Prediction at Highway Interchanges

arXiv:2510. 03381v3 Announce Type: replace-cross Abstract: Interchanges are crucial nodes for vehicle transfers between highways, yet the lack of real-time ramp detectors creates blind spots in traffic prediction.

By Yongchao Li, Jun Chen, Zhuoxuan Li, Chao Gao, Yang Li, Chu Zhang, Changyin Dong

multimodal benchmarks

arXiv AI

Jun 8

DuMate-DeepResearch: An Auditable Multi-Agent System with Recursive Search and Rubric-Grounded Reasoning

arXiv:2606. 07299v1 Announce Type: new Abstract: Deep Research (DR) has emerged as a new agentic paradigm to tackle complex, open-ended research tasks, demanding systems that can iteratively frame problems, acquire evidence, verify sources, and synthesize long-form reports.

By Lingyong Yan, Can Xu, Yukun Zhao, Wenxuan Li, Qingyang Chen, Jiulong Wu, Wenli Song, Xiangnan Li, Weixian Shi, Yiqun Chen, Xuchen Ma, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Jianmin Wu, Dawei Yin

agents benchmarks safety

arXiv AI

Jun 8

LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis

arXiv:2510. 24561v3 Announce Type: replace-cross Abstract: LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention.

By Qingyue Zhang, Chang Chu, Tianren Peng, Qi Li, Xiangyang Luo, Zhihao Jiang, Shao-Lun Huang

fine-tuning benchmarks safety

arXiv Machine Learning

Jun 8

Uncertainty-Guided Label Rebalancing for CPS Safety Monitoring

arXiv:2603. 25670v3 Announce Type: replace Abstract: Safety monitoring is essential for Cyber-Physical Systems (CPSs).

By John Ayotunde, Qinghua Xu, Guancheng Wang, Lionel C. Briand

benchmarks safety

arXiv AI

Jun 8

Quantum-Inspired Trace-Augmented Evidence Selection for Reasoning over Structured Hypothesis Spaces

arXiv:2606. 06941v1 Announce Type: new Abstract: Large language models (LLMs) now solve a wide range of expert-level exams at or above human level, yet remain brittle on specialised, evidence-intensive domains such as law.

By Laura Wynter, Nirvik Sahoo, Paul Griffin

llms benchmarks

arXiv AI

Jun 8

Declarative Skills for AI Agents in Knowledge-Grounded Tool-Use Workflows

arXiv:2606. 06923v1 Announce Type: new Abstract: We study orchestration mechanisms for tool-using AI agents in realistic customer-service workflows over an unstructured knowledge base.

By M. Danish Lim, I. Danial Bin Sharudin, Wen Han Chen, Cedric Lim, Laura Wynter

llms rag agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 8

Latent Geometry Beyond Search: Amortizing Planning in World Models

arXiv:2605. 08732v2 Announce Type: replace-cross Abstract: Modern vision-based world models can represent observations as compact yet expressive latent manifolds, but fast goal-oriented planning in these spaces remains challenging.

By Hoang Nguyen, Xiaohao Xu, Xiaonan Huang

robotics benchmarks

arXiv Machine Learning

Jun 8

AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

arXiv:2512. 13278v2 Announce Type: replace-cross Abstract: Agentic reinforcement learning has advanced large language models (LLMs) to reason through long chain-of-thought trajectories while interleaving external tool use.

By Jiaru Zou, Ling Yang, Yunzhe Qi, Sirui Chen, Mengting Ai, Ke Shen, Jingrui He, Mengdi Wang

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jun 8

Mitosis Detection in the Wild: Multi-Tumor and Context-Aware Generalization in the MIDOG 2025 Challenge

arXiv:2606. 07368v1 Announce Type: cross Abstract: Automated mitosis detection is a well-established task in computational pathology.

By Marc Aubreville, Jonas Ammeling, Sweta Banerjee, Viktoria Weiss, Taryn A. Donovan, Robert Klopfleisch, Jiaqi Lv, Shan E Ahmed Raza, Rapha\"el Bourgade, Thomas Walter, Yasemin Topuz, Song\"ul Varl{\i}, Charles-Antoine Collins-Fekete, Zhuoyan Shen, Navya Sri Kelam, Nitin Singhal, Christian Marzahl, Brian Napora, Tengyou Xu, Hongyan Gu, Mario Vento, Gennaro Percannella, Norbert Ropiak, Izabela Wasiak, Jie Xiao, Shaojun Liu, Seungho Choe, April Khademi, Vidushi Walia, Sujatha Kotte, Andrew Broad, Alex Wright, Guillaume Balezo, Esha Sadia Nasir, Mostafa Jahanifar, Yosuke Yamagishi, Shouhei Hanaoka, Mattia Sarno, Francesco Tortorella, Biwen Meng, Jingxin Liu, Sara Krauss, Daniel Hieber, Lavish Ramchandani, Dev Kumar Das, Mieko Ochi, Yuan Bae, Piotr Giedziun, Mateusz Maniewski, Vangala Govindakrishnan Saipradeep, Naveen Sivadasan, Leire Benito-Del-Valle, Adrian Galdran, Kaustubh Atey, Sameer Anand Jha, Adinath Dukre, Imran Razzak, Maxime W. Lafarge, Viktor H. Koelzer, Nils Porsche, Nikolas Stathonikos, Mitko Veta, Dominik Hirling, Zsanett Zs\'ofia Iv\'an, Peter Horvath, Katharina Breininger, Christof A. Bertram

benchmarks

Get the daily briefing