Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv Machine Learning

Jun 15

Where Black-box Drug-Target Interaction Prediction Models Look: Cross-Method Explainability

arXiv:2606. 14245v1 Announce Type: new Abstract: Drug-target interaction (DTI) and affinity (DTA) predictors increasingly achieve strong benchmark scores, yet their internal use of sequence, fingerprint, and graph features often remains opaque.

By Ali Vefghi, Zahed Rahmati, Mohammad Akbari

benchmarks safety

arXiv AI

Jun 15

Residual Context Diffusion Language Models

arXiv:2601. 22954v2 Announce Type: replace-cross Abstract: Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel.

By Yuezhou Hu, Harman Singh, Monishwaran Maheswaran, Haocheng Xi, Coleman Hooper, Jintao Zhang, Aditya Tomar, Michael W. Mahoney, Sewon Min, Mehrdad Farajtabar, Kurt Keutzer, Amir Gholami, Chenfeng Xu

llms diffusion benchmarks

arXiv AI

Jun 15

Selective Agentic Recovery for UAV Autonomy with a Persistent Mission Runtime

arXiv:2606. 14219v1 Announce Type: cross Abstract: Agentic AI can support unmanned aerial vehicle (UAV) autonomy by providing high-level recovery reasoning when local waypoint- or setpoint-based execution encounters blocked passages, repeated no-progress behavior, or mission-level ambiguity.

By Taewoo Park, Kyeonghyun Yoo, Seunghyun Yoo, Hwangnam Kim

agents benchmarks safety

arXiv Machine Learning

Jun 15

Learning Variable-Length Tokenization for Generative Recommendation

arXiv:2605. 17779v2 Announce Type: replace Abstract: Generative recommendation reformulates recommendation as next-token prediction over discrete semantic identifiers (IDs).

By Minhao Wang, Bowen Wu, Wei Zhang

nlp efficiency benchmarks

arXiv Machine Learning

Jun 15

DRIVE: Distributional and Retrieval-Augmented Bidding with Value Evaluation

arXiv:2606. 14192v1 Announce Type: new Abstract: Auto-bidding is a core component of real-time advertising systems, where decisions must optimize long-term performance under budget and cost constraints, while online exploration is prohibitively risky.

By Miduo Cui, Haochen Wang, Shangqin Mao, Xun Yang, Qianlong Xie, Xingxing Wang, Xuri Ge, Ying Zhou, Zhiwei Xu

llms rag reinforcement-learning benchmarks

arXiv Machine Learning

Jun 15

Scalable Deep Unfolding of Conic Optimizers

arXiv:2606. 13825v1 Announce Type: cross Abstract: Deep unfolding (DU) accelerates iterative optimizers by introducing learnable components and training them through unrolled iterations, but extending DU to the large-scale semidefinite programs (SDPs) common in robotics has remained limited.

By Alex Oshin, Rahul Vodeb Ghosh, Evangelos A. Theodorou

robotics benchmarks

arXiv Machine Learning

Jun 15

Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression

arXiv:2602. 08324v5 Announce Type: replace Abstract: Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs), yet it incurs substantial computational overhead for inference.

By Yuntian Tang, Bohan Jia, Wenxuan Huang, Lianyue Zhang, Jiao Xie, Wenxi Li, Wei Li, Jie Hu, Xinghao Chen Rongrong Ji, Shaohui Lin

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 15

VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation

arXiv:2606. 13735v1 Announce Type: cross Abstract: Large Language Models (LLM) have shown impressive capabilities in Register Transfer Level (RTL) code generation, particularly for Verilog.

By Yijun Shen, Minghao Shao, Yichen Zhao, Zhuoyan Yu, Boyuan Chen, Yik-Cheung Tam, Muhammad Shafique

llms benchmarks data-engineering

arXiv AI

Jun 15

Orchestra-o1: Omnimodal Agent Orchestration

arXiv:2606. 13707v1 Announce Type: new Abstract: The recent success of agent swarms has shifted the paradigm of large language model (LLM)-based agents from single-agent workflows to multi-agent systems, highlighting the importance of agent orchestration for task decomposition and collaboration.

By Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, Pheng-Ann Heng

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 15

Deep Dense Exploration for LLM Reinforcement Learning via Pivot-Driven Resampling

arXiv:2602. 14169v2 Announce Type: replace-cross Abstract: Effective exploration is a key challenge in reinforcement learning for large language models: discovering high-quality trajectories within a limited sampling budget from the vast natural language sequence space.

By Yiran Guo, Zhongjian Qiao, Yingqi Xie, Jie Liu, Dan Ye, Ruiqing Zhang, Shuang Qiu, Lijie Xu

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 15

Same-Origin Policy for Agentic Browsers

arXiv:2606. 14027v1 Announce Type: cross Abstract: Agentic browsers integrate autonomous AI agents into web browsers, enabling users to accomplish web tasks through natural-language instructions.

By Xilong Wang, Xiaoxing Chen, Patrick Li, Dawn Song, Neil Gong

agents benchmarks

arXiv AI

Jun 15

Aligning Quantum Operators with Large Language Models

arXiv:2606. 13811v1 Announce Type: cross Abstract: Can Large Language Models (LLMs) understand and reason about quantum operators?

By Rogerio Feris, Yunchao Liu, Pengyuan Li, Hang Hua, David Kremer

llms diffusion benchmarks

arXiv AI

Jun 15

Quantile-Free Uncertainty Quantification in Graph Neural Networks

arXiv:2605. 04847v2 Announce Type: replace-cross Abstract: Uncertainty quantification (UQ) in graph neural networks (GNNs) is crucial in high-stakes domains but remains a significant challenge.

By Soyoung park, Hwanjun Song, Sungsu Lim

benchmarks

arXiv AI

Jun 15

HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification

arXiv:2606. 13706v1 Announce Type: cross Abstract: We present HierSVA, an integrated suite that combines a pipeline, dataset, and benchmark for LLM-driven hierarchical hardware formal verification.

By Maohua Nie, Jiang Zhu, Jingqun Zhang, Zhichen Zeng, Jiayi Wang, Sibo Zhang, Jialin Wang, C. -J. Richard Shi

llms agents benchmarks

arXiv AI

Jun 15

C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning

arXiv:2603. 05167v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning, yet it remains unclear whether they can reliably assess process faithfulness rather than merely answer plausibility.

By Avni Mittal, Rauno Arike

llms benchmarks

arXiv AI

Jun 15

Token-Level LLM Collaboration via FusionRoute

arXiv:2601. 05106v5 Announce Type: replace Abstract: Large language models (LLMs) exhibit strengths across diverse domains.

By Nuoya Xiong, Yuhang Zhou, Hanqing Zeng, Zhaorun Chen, Furong Huang, Shuchao Bi, Lizhu Zhang, Zhuokai Zhao

llms fine-tuning benchmarks

arXiv AI

Jun 15

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

arXiv:2606. 14249v1 Announce Type: new Abstract: AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts.

By Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, Yuan Xie, Heng Qu, Kun Shao, Jian Luan

agents reinforcement-learning benchmarks

arXiv AI

Jun 15

A Deep Reinforcement Learning (DRL)-Based Transformer Method for Solving the Open Shop Scheduling Problem

arXiv:2606. 13682v1 Announce Type: new Abstract: The open shop scheduling problem (OSSP) arises in many industrial and service settings but remains computationally challenging as the number of jobs and machines increases.

By Faezeh Ardali, Mwembezi A. Nyelele, Gerald M. Knapp

llms reinforcement-learning benchmarks

arXiv AI

Jun 15

MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition

arXiv:2606. 14459v1 Announce Type: cross Abstract: Modern Automatic Speech Recognition (ASR) systems have made remarkable progress on standard benchmarks, yet performance gaps have emerged under real-world distribution shifts, caused by recording conditions, accents, speech impairments, and noise.

By Theresa Pekarek Rosin, Matthias Kerzel, Stefan Wermter

multimodal benchmarks

arXiv AI

Jun 15

SANA: What Matters for QA Agents over Massive Data Lakes?

arXiv:2606. 13904v1 Announce Type: cross Abstract: Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results.

By Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu

llms agents nlp benchmarks

Get the daily briefing