Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,260 stories · RSS feed

arXiv AI

Aug 5

HERO: Hierarchical Evidential Reasoning Optimization for Radiology Report Generation via Reason-then-Summarize

arXiv:2601. 03321v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have substantially advanced Radiology Report Generation (RRG), yet aligning them through reinforcement learning (RL) remains challenging due to heterogeneous medical supervision.

By Kun Zhao, Guodong Liu, Hui Ji, Siyuan Dai, Pan Wang, Jifeng Song, Chenghua Lin, Liang Zhan, Haoteng Tang

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Aug 5

Beyond Either-Or Reasoning: Transduction and Induction as Cooperative Problem-Solving Paradigms

arXiv:2505. 14744v3 Announce Type: replace-cross Abstract: Traditionally, in Programming-by-example (PBE) the goal is to synthesize a program from a small set of input-output examples.

By Janis Zenkner, Tobias Sesterhenn, Christian Bartelt

benchmarks

arXiv Machine Learning

Aug 5

Investigating reservoir computing for branch prediction in pipelined processors using emerging CMOS memristor devices

arXiv:2607. 27140v2 Announce Type: replace-cross Abstract: This project aimed to develop a novel reservoir compute (RC) implementation framework targeting high-speed operation and integration with CMOS digital logic.

By Harvey Samuel George Johnson, Sendy Phang

benchmarks

arXiv AI

Aug 5

TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions

arXiv:2604. 27975v2 Announce Type: replace-cross Abstract: Traditional Shot Boundary Detection (SBD) inherently struggles with complex transitions by formulating the task around isolated cut points, frequently yielding corrupted video shots.

By Ce Chen, Yi Ren, Yuanming Li, Viktor Goriachko, Zhenhui Ye, Zujin Guo, Zhibin Hong, Mingming Gong

llms computer-vision multimodal benchmarks

arXiv AI

Aug 5

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

arXiv:2608. 03119v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability.

By Yongshi Ye, Liang Zhang, Yidong Chen, Xiaodong Shi, Biao Fu

llms reinforcement-learning benchmarks

arXiv AI

Aug 5

MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc

arXiv:2608. 03397v1 Announce Type: new Abstract: MMLongBench-Doc is a long-document QA benchmark of 1,082 questions over 135 PDFs.

By Mingtian Zhang

llms benchmarks

arXiv AI

Aug 5

KernelBrain: Coarse-to-Fine, Budget-Aware Search for Agentic GPU Kernel Optimization

arXiv:2608. 02611v1 Announce Type: cross Abstract: Automating GPU kernel optimization remains difficult in practice: generated variants can violate correctness constraints, runtime measurements are noisy, and search often stalls early.

By Shuai Che, Gang Peng

llms agents benchmarks

arXiv AI

Aug 5

SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them

arXiv:2607. 27703v2 Announce Type: replace Abstract: Vision-language models (VLMs) are increasingly used in embodied agents to interpret visual inputs, reason about spatial relationships, and make task-level decisions based on that reasoning.

By Yang Zhou, Zixuan Huang, Sunzhu Li, Zhuo Yang, Chen Zhang, Shunian Chen, Caijun Yan, Jianyao Xu, Shunyu Liu, Weijie Fu, Peiliang Li, Xiaozhi Chen, Yuxiang Cai

llms agents computer-vision reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Aug 5

UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks

arXiv:2608. 03018v1 Announce Type: new Abstract: Modern cities rely on an increasing number of digital services to operate, but residents' daily needs are still difficult to meet.

By Jiayu Cao, Xingyuan Zeng, feiyu Li, Zhijing Huang, Xujie Yuan, Rongxiang Chen, Shimin Di, Libin Zheng, Jian Yin

llms agents benchmarks

arXiv AI

Aug 5

LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment

arXiv:2608. 03020v1 Announce Type: new Abstract: Parameter-efficient post-training reduces the number of trainable parameters, but still requires repeated end-to-end backpropagation through the frozen backbone.

By Linhan Xia, Rui Liu, Zhaofeng Zhang, Yihao Wang, Binrui Shen, Shengxin Zhu

llms fine-tuning benchmarks

arXiv AI

Aug 5

Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls

arXiv:2608. 03071v1 Announce Type: new Abstract: Large language model agents derive much of their capability from tool use.

By Guoyao Yu, Xiaoqing Sun, Ziqi Huang, Shaojing Fan, Zhongyi Zhang, Xiaomeng Hu, Xiaobo Xue, Yangyang Shi, Xiong Xiao, Yang Song, Biao Lyu, Rong Wen, Xing Li, Qinming He, Shunming Zhu, Zhenguang Liu

llms rag agents fine-tuning benchmarks

arXiv AI

Aug 5

Verifiable Memory: Learning Unified Memory Management with Local and Global Verifiers for Large Language Model Agents

arXiv:2608. 03137v1 Announce Type: new Abstract: Large language model (LLM) agents must retain reusable information, control a bounded active context, and recover earlier evidence during long-horizon interaction.

By Xiaolong Sun, Qichao Wang, Hangyu Li, Liang Chen

llms agents fine-tuning benchmarks

arXiv AI

Aug 5

Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation

arXiv:2608. 03166v1 Announce Type: new Abstract: Role-Playing Language Agents (RPLAs) are increasingly deployed in high-stakes applications such as healthcare assistance, customer support, and education, where maintaining consistent personas, ethical constraints, and behavioral coherence under adversarial pressure is critical.

By Saqib Shouqi, Abdullah Nazly, Januki Wanniarachchi, Ravisha De Alwis

llms agents robotics benchmarks safety

arXiv AI

Aug 5

Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study

arXiv:2608. 03172v1 Announce Type: new Abstract: Structure-preserving de-identification replaces protected health information (PHI) with realistic same-type surrogates -- "Anna S.

By Qiming Bao, Sherry J. H. Feng, Kim Chester Eugenio, Meng Fon

benchmarks

arXiv AI

Aug 5

When Refusal Looks Safe: The Refusal-Cue Shortcut in Safety Guard Models

arXiv:2608. 03201v1 Announce Type: new Abstract: Safety guards are widely used to filter harmful content and are typically trained via supervised fine-tuning on labeled prompt-response pairs.

By Yu Feng, Chunting Zang, Chen Shen, Rui Miao, Ge Teng, Weidong Cai, Jieping Ye

fine-tuning benchmarks safety

arXiv AI

Aug 5

One Knob to Rule Them All: A Unified Optimal Transport View of Cold-Start Active Learning

arXiv:2608. 03249v1 Announce Type: new Abstract: Cold-Start Active Learning (CSAL) aims to select a valuable subset from an unlabeled pool without any prior knowledge or human assistance.

By Ning Zhu, Xiaochuan Ma, Juntao Xu, Jingze Liang, Mengfei Zhao, An Chen, Liang-Jian Deng

benchmarks safety

arXiv AI

Aug 5

DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning

arXiv:2608. 03292v1 Announce Type: new Abstract: Long Document Visual Question Answering (LongDocVQA) requires Multimodal Large Language Models (MLLMs) to locate, integrate, and reason over heterogeneous document elements distributed across multiple pages.

By Le Xiang, Zhicheng Guan, Hong Chen, Xiaocong Lin, Zhenghua Lei, Teng Hu, Bolei He, Long Zeng

llms rag agents nlp fine-tuning multimodal benchmarks

arXiv AI

Aug 5

Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents

arXiv:2608. 03327v1 Announce Type: new Abstract: Hybrid computer-use agents can act through screenshots or call text tools.

By Siqi Fan, Minghao Li, Xiaoqian Ma, Wenhui Tan, Xiusheng Huang, Juntong Wu, Liujie Zhang, Shuo Shang, Weihang Chen

agents reinforcement-learning multimodal benchmarks

arXiv AI

Aug 5

DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

arXiv:2608. 03451v1 Announce Type: new Abstract: Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia.

By Boyan Li, Zhuowen Liang, Yupeng Xie, Xiaotian Lin, Tianqi Luo, Xinyu Liu, Yizhang Zhu, Zhangyang Peng, Yuan Li, Zhengxuan Zhang, Jiayi Zhang, Nan Tang, Guoliang Li, Yuyu Luo

agents multimodal benchmarks safety

arXiv AI

Aug 5

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

arXiv:2608. 03457v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood.

By Fengqi Zhu, Shaoxuan Xu, Jingyang Ou, Zebin You, Yipeng Xing, Huabin Liu, Xiaolu Zhang, Jun Zhou, Zhenzhong Lan, Yankai Lin, Wayne Xin Zhao, Jianguo Li, Chongxuan Li, Ji-Rong Wen

llms diffusion fine-tuning benchmarks

Get the daily briefing