Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,494 stories · RSS feed

arXiv Machine Learning

Jun 2

Accelerating physics-informed neural networks for full waveform inversion using a hybrid quantum-classical finite-basis architecture

arXiv:2606. 01110v1 Announce Type: cross Abstract: Full waveform inversion (FWI) reconstructs heterogeneous material properties from receiver data but remains computationally demanding.

By Hoang Anh Nguyen, Divakar Vashisth, Ali Tura

benchmarks

arXiv AI

Jun 2

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events

arXiv:2606. 02522v1 Announce Type: cross Abstract: Video multimodal large language models (MLLMs) have made rapid progress on general and long-form video understanding, yet their ability to preserve brief answer-critical visual evidence remains underexplored.

By Xiaolin Liu, Yilun Zhu, Xiangyu Zhao, Xuehui Wang, Yan Li, Xin Li, Haoyu Cao, Xing Sun, Shaofeng Zhang, Xu Yang, Zhihang Zhong, Xue Yang

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

arXiv:2511. 04791v2 Announce Type: replace Abstract: Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases.

By Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Mark Hill, Murali Annavaram

llms efficiency benchmarks

arXiv AI

Jun 2

PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Vision-Language Models

arXiv:2602. 00415v2 Announce Type: replace Abstract: Memory is not merely a storage mechanism for intelligent systems, but a structure for organizing evidence and constraining belief.

By Zhisheng Chen, Tingyu Wu, Zijie Zhou, Zhengwei Xie, Jinhan Li, Ziyan Weng, Liang Lin, Jingwei Song, Zikai Xiao, Yingwei Zhang

llms multimodal benchmarks

arXiv AI

Jun 2

OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

arXiv:2603. 14771v3 Announce Type: replace Abstract: Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting the capabilities of LLM agents.

By Peigen Liu, Rui Ding, Yuren Mao, Ziyan Jiang, Yuxiang Ye, Yunjun Gao, Ying Zhang, Renjie Sun, Longbin Lai, Zhengping Qian

llms agents benchmarks

arXiv Machine Learning

Jun 2

Reconsidering Positional Supervision in Masked Diffusion Language Model Training

arXiv:2601. 22947v2 Announce Type: replace-cross Abstract: Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models.

By Mengyu Ye, Keito Kudo, Ryosuke Takahashi, Jun Suzuki

llms diffusion fine-tuning benchmarks safety

arXiv AI

Jun 2

Process Reward Agents for Steering Knowledge-Intensive Reasoning

arXiv:2604. 09482v2 Announce Type: replace Abstract: Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources.

By Jiwoong Sohn, Tomasz Sternal, Kenneth Styppa, Torsten Hoefler, Michael Moor

rag agents benchmarks

arXiv AI

Jun 2

MASCOT: Towards Multi-Agent Socio-Collaborative Companion Systems

arXiv:2601. 14230v2 Announce Type: replace-cross Abstract: Multi-agent systems (MAS) are emerging as promising socio-collaborative companions for emotional and cognitive support.

By Yiyang Wang, Yiqiao Jin, Alex Cabral, Josiah Hester

llms agents benchmarks safety

arXiv AI

Jun 2

ANDRE: An Attention-based Neuro-symbolic Differentiable Rule Extractor for Inductive Logic Programming

arXiv:2605. 04193v2 Announce Type: replace Abstract: Inductive Logic Programming (ILP) aims to learn interpretable first-order rules from data, but existing symbolic and neuro-symbolic approaches struggle to scale to noisy and probabilistic settings.

By Iman Sharifi, Peng Wei, Saber Fallah

benchmarks

arXiv AI

Jun 2

AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

arXiv:2507. 08038v3 Announce Type: replace-cross Abstract: Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge.

By Talor Abramovich, Gal Chechik

llms agents benchmarks

arXiv AI

Jun 2

Neural Decision-Propagation for Answer Set Programming

arXiv:2605. 01797v2 Announce Type: replace Abstract: Integration of Answer Set Programming (ASP) with neural networks has emerged as a promising tool in Neuro-symbolic AI.

By Thomas Eiter, Katsumi Inoue, Sota Moriyama

benchmarks

arXiv AI

Jun 2

Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Apps

arXiv:2510. 00481v2 Announce Type: replace-cross Abstract: In 2025, Large Language Model (LLM) services have launched a new feature -- AI video chat -- allowing users to interact with AI agents via real-time video communication (RTC), just like chatting with real people.

By Jiayang Xu, Xiangjie Huang, Zijie Li, Antariksh Verma, Zili Meng

llms agents benchmarks

arXiv AI

Jun 2

Causal state binding predicts action control in language agents

arXiv:2605. 09692v3 Announce Type: replace Abstract: Autonomous language agents increasingly expose traces, memories, plans and constraints, but existing evaluations rarely test whether these state variables are bound to final actions.

By Xiao Jia

llms agents benchmarks

arXiv AI

Jun 2

Beyond the Frontier: Stochastic Backtracking for Efficient Test-Time Scaling

arXiv:2605. 25143v2 Announce Type: replace Abstract: Test-time scaling improves language model reasoning by spending additional compute to explore multiple solution trajectories.

By Dao Tran, Duc Anh Le, Ngoc Luu, Quan Pham, Tung Pham, Hung Bui

llms benchmarks

arXiv AI

Jun 2

Truth, Trust, and Trouble: Medical AI on the Edge

arXiv:2507. 02983v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering.

By Mohammad Anas Azeez, Rafiq Ali, Ebad Shabbir, Zohaib Hasan Siddiqui, Gautam Siddharth Kashyap, Jiechao Gao, Usman Naseem

llms nlp benchmarks safety

arXiv AI

Jun 2

Toward accurate RUL and SoH estimation using reinforced graph-based physics-informed neural networks enhanced with dynamic weights

arXiv:2507. 09766v2 Announce Type: replace-cross Abstract: Accurate estimation of Remaining Useful Life (RUL) and State of Health (SoH) is essential for reliable Prognostics and Health Management (PHM), supporting timely maintenance and dependable industrial operation.

By Mohamadreza Akbari Pour, Ali Ghasemzadeh, Mohamad Ali Bijarchi, Mohammad Behshad Shafii

reinforcement-learning benchmarks

arXiv AI

Jun 2

Community-Aware Assessment of Social Textual Engagement and Resonance: A Human-Centric Perspective on User-Generated Content Evaluation

arXiv:2606. 01897v1 Announce Type: new Abstract: Traditional Video Quality Assessment (VQA) focuses narrowly on aesthetic fidelity, overlooking the complex social dynamics that define quality in User-Generated Content (UGC).

By Tianjiao Li, Kai Zhao, Xiang Li, Yang Liu, Huyang Sun

reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

BatteryMFormer: Multi-level Learning for Battery Degradation Trajectory Forecasting

arXiv:2605. 27044v2 Announce Type: replace Abstract: Early battery degradation trajectory forecasting (BDTF), which predicts the full-life state-of-health trajectory from early operational data, is critical for battery optimization, manufacturing, and deployment.

By Ruifeng Tan, Jintao Dong, Weixiang Hong, Jia Li, Jiaqiang Huang, Tong-Yi Zhang

llms benchmarks

arXiv AI

Jun 2

Benchmarking AI for low-resource contexts: Thinking beyond leaderboards

arXiv:2605. 28508v2 Announce Type: replace Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality.

By Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam, Prasaanth Balraj, Nakul Jain

rag benchmarks

arXiv AI

Jun 2

Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation

arXiv:2605. 30000v2 Announce Type: replace Abstract: Front-end web code has become a core product surface for every frontier LLM release, yet evaluating these interactive applications at development speed remains costly because human-judged leaderboards like Arena do not scale.

By Haoyue Yang, Zhangxiao Shen, Fan Ding, Hangting Lou, Yifeng Kou, Haoqing Yu, Jingyao Li, Zhengfan Wu, Siqi Bao, Jing Liu, Hua Wu

llms agents benchmarks

Get the daily briefing