Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,930 stories · RSS feed

arXiv AI

Jun 30

BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis

arXiv:2606. 29102v1 Announce Type: cross Abstract: Jointly learning to segment and classify medical images demands cross-task synergy, yet encoder-sharing architectures limit decoder reconstruction to task-private representations, permanently discarding the boundary cues and semantic priors each branch could supply to the other.

By Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo

computer-vision benchmarks safety

arXiv AI

Jun 30

CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.

By Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, Yi R. Fung

llms agents benchmarks

arXiv AI

Jun 30

When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration

arXiv:2606. 20724v2 Announce Type: replace Abstract: Long-horizon web agents often fail in ways hidden by final-answer evaluation: they may visit useful pages, produce a well-formed answer, and terminate confidently while still missing fields, over-including unsupported items, or relying on stale evidence.

By Aagam Sogani, Botao Rui, Swetha Vaidyanathan, Rishi Agarwal, Minghao Yan, Shivaram Venkataraman

llms agents benchmarks

arXiv Machine Learning

Jun 30

DataComp-VLM: Improved Open Datasets for Vision-Language Models

arXiv:2606. 28551v1 Announce Type: cross Abstract: Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies.

By Matteo Farina, Vishaal Udandarao, Thao Nguyen, Selim Kuzucu, Maximilian B\"other, Andreas Hochlehnert, Adhiraj Ghosh, Marianna Nezhurina, Karsten Roth, Joschka Struber, Yuhui Zhang, Sebastian Dziadzio, Elaine Sui, Soumya Jahagirdar, Dhruba Ghosh, Hasan Hammoud, Thomas De Min, Simone Caldarella, Jehanzeb Mirza, Sedrick Keh, Mehdi Cherti, Hilde Kuehne, Bernt Schiele, Serena Yeung-Levy, Muhammad Ferjad Naeem, Federico Tombari, Ana Klimovic, Elisa Ricci, Matthias Bethge, Sewoong Oh, Ameya Prabhu, Alessio Tonioni, Jenia Jitsev, Massimiliano Mancini, Ludwig Schmidt, Nikhil Parthasarathy

llms multimodal benchmarks

arXiv Machine Learning

Jun 30

DiLaServe: High SLO Attainment Serving for Diffusion Language Models

arXiv:2606. 29094v1 Announce Type: new Abstract: Diffusion language models (DLMs) have recently emerged as a promising alternative to conventional autoregressive language models.

By Tzu-Tao Chang, Benjamin Yuanyang Hong, Kiet Pham, Shivaram Venkataraman

llms diffusion efficiency benchmarks

arXiv AI

Jun 30

From Regulatory Approvals to Patents: Cross-Domain Linking for Cardiovascular Device Traceability

arXiv:2606. 28353v1 Announce Type: cross Abstract: Linking FDA-approved medical devices to their underlying United States Patent and Trademark Office (USPTO) patents enables critical applications such as recall root-cause analysis, M&A-driven IP discovery, and technology trajectory mapping.

By Yang Qingqing, Liu Haijiang, Li Moyan

llms rag benchmarks

arXiv Machine Learning

Jun 30

$\mu$Flow: Leveraging Average Images for Improving Generalisation of Deepfake Faces Detectors

arXiv:2606. 30528v1 Announce Type: cross Abstract: Current generative models, including GANs and diffusion models, have reached an outstanding level of photorealism, posing significant risks to privacy and security.

By Orazio Pontorno, Mattia Litrico, Luca Guarnera, Mario Valerio Giuffrida, Sebastiano Battiato

diffusion benchmarks safety

arXiv AI

Jun 30

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

arXiv:2603. 03915v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs).

By Ji-Lun Peng, Yun-Nung Chen

llms agents benchmarks

arXiv AI

Jun 30

Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

arXiv:2606. 29159v1 Announce Type: new Abstract: Offline root-cause-analysis (RCA) benchmarks commonly rank methods by a single pooled top-1 accuracy across multiple subsystems, and engineers often read the pooled winner as a recommendation for their own subsystem.

By Lining Hu, Ting Liu, Yuzhuo Fu

benchmarks

arXiv AI

Jun 30

PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation

arXiv:2606. 28344v1 Announce Type: cross Abstract: Augmenting large language models (LLMs) with retrieved web text has become a dominant paradigm, yet the web is not natively textual: existing systems depend on complex parsing pipelines that linearize HTML and discard layout, visual structure, and formatting.

By Yichuan Wang, Zhifei Li, Zirui Wang, Paul Teiletche, Lesheng Jin, Matei Zaharia, Joseph E. Gonzalez, Sewon Min

llms rag agents multimodal benchmarks

arXiv AI

Jun 30

ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation

arXiv:2606. 28357v1 Announce Type: cross Abstract: Recent advances in multimodal recommenders excel at feature fusion but remain opaque and inefficient decision-makers, lacking explicit reasoning and self-awareness of uncertainty.

By Yihua Zhang, Mingfu Liang, Jiyan Yang, Rong Jin, Wen-Yen Chen, Yiping Han, Huayu Li, Buyun Zhang, Liang Luo, Frank Shyu, Luke Simon, Sijia Liu, Tianlong Chen, Xi Liu

agents fine-tuning efficiency multimodal benchmarks

arXiv AI

Jun 30

Optimizing Expert-Designed Crystal Graph Networks for Band-Gap Prediction with an Autonomous LLM Research Loop

arXiv:2606. 29717v1 Announce Type: cross Abstract: Predicting a material's properties from its structure is a central, fast-advancing problem in computational materials science.

By Chenmu Zhang, Boris I. Yakobson

llms rag agents benchmarks

arXiv AI

Jun 30

LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis

arXiv:2606. 28362v1 Announce Type: cross Abstract: Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort.

By Yen-Hsun Huang (Department of Education, Taipei Veterans General Hospital, Taipei, Taiwan), Yu-Shiou Lin (Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan)

llms agents benchmarks

arXiv Machine Learning

Jun 30

PCGD: Physics-Guided Conditional Graph Diffusion for TCAD Device Simulation

arXiv:2606. 29272v1 Announce Type: new Abstract: Technology computer-aided design (TCAD) semiconductor device simulation is fundamentally constrained by the high computational cost of iteratively solving coupled drift-diffusion equations.

By Yihan Zhang, Zhiteng Zhang, Kun Chen, Chen Wang

diffusion fine-tuning benchmarks

arXiv AI

Jun 30

Deterministic Decisions for High-Stakes AI. A Zero-Egress Pipeline with the Deployability of RAG and the Accuracy of Machine Learning

arXiv:2606. 29280v1 Announce Type: cross Abstract: We identify intervention bias as a previously unquantified failure mode of zero-shot large-language-model (LLM) educational advisory agents: without task-specific training, they recommend action when a hindsight-optimal oracle policy mandates inaction.

By Craig Atkinson

llms rag agents efficiency benchmarks safety data-engineering

arXiv Machine Learning

Jun 30

BrainRiem: Riemannian Prototype Learning for Source-Free Cross-Site Brain Network Diagnosis

arXiv:2606. 29200v1 Announce Type: new Abstract: Multi-site functional MRI (fMRI) studies are essential for robust neuropsychiatric diagnosis yet suffer severe domain shifts from scanner heterogeneity, demographics, and site-specific acquisition protocols.

By Kunyu Zhang, Tianxiang Xu

fine-tuning benchmarks safety

arXiv Machine Learning

Jun 30

Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks

arXiv:2606. 29082v1 Announce Type: cross Abstract: Would experience designing faster GPU kernels also help close in on a long-standing open mathematical conjecture?

By Young-Jun Lee, Seungone Kim, Minki Kang, Alistair Cheong Liang Chuen, Zerui Chen, Seungho Han, Taehee Jung, Dongyeop Kang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 30

GLACIER: Rethinking Mass Spectrum Prediction as an Object Detection Problem

arXiv:2606. 29161v1 Announce Type: new Abstract: Predicting tandem mass spectra (MS/MS) from molecular structures represents a central task in analytical chemistry with direct relevance to clinical metabolomics, systems biology, and adjacent disciplines.

By Rui-Xi Wang, Runzhong Wang, Connor W. Coley

llms computer-vision fine-tuning benchmarks

arXiv Machine Learning

Jun 30

Few-Step Boltzmann Generators via Scalable Likelihood Flow Maps

arXiv:2606. 29110v1 Announce Type: new Abstract: Recent progress in flow-based generative modeling has led to models that output high-quality samples while using only a small number of function evaluations.

By RuiKang OuYang, Hanlin Yu, Xinyue Ai, Yutong He, Nicholas M. Boffi, Pradeep Ravikumar, Jose Miguel Hernandez-Lobato, Max Simchowitz, Benjamin Kurt Miller, Omar Chehab

efficiency benchmarks

arXiv Machine Learning

Jun 30

MOSAIC: Orchestrating Collaborative Knowledge Tracing with Hierarchical Semantic Alignment

arXiv:2606. 29049v1 Announce Type: new Abstract: Knowledge Tracing (KT) is important for personalized education but traditionally suffers from two key limitations: a reliance on shallow ID-based representations that neglect semantic depth and a restriction to single-granularity mastery estimation that overlooks hierarchical knowledge dependencies.

By Xinjin Li, Mengyue Wang, Yuzhen Lin, Pengbin Feng, Ziqi Sha, Yeyang Zhou, Yu Ma

llms rag benchmarks safety

Get the daily briefing