Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,813 stories · RSS feed

arXiv AI

Jul 28

Beyond Block Boundaries: Multi-Block Editing for Diffusion Large Language Models

arXiv:2607. 22663v1 Announce Type: new Abstract: Block diffusion has emerged as the dominant paradigm for scaling discrete diffusion language models (dLLMs), because decoding text in fixed-size blocks preserves parallel generation within each block while keeping the quadratic attention cost tractable.

By Xingyu Mou, Zijin Huang, Tianze Zhang, Yuxin Ma, Lanning Wei, Zengfeng Huang, Da Zheng, Lun Du

llms diffusion fine-tuning efficiency benchmarks

arXiv AI

Jul 28

Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents

arXiv:2607. 22689v1 Announce Type: new Abstract: Graphical user interface (GUI) agents are systems powered by large multimodal models (LMMs).

By Zedong Yu, Qianxing Li, Zhi Gao, Liuyu Xiang, Chenrui Shi, Yang Liu, Huiming Wu, Yujie Wei, Yuhao Fei, Yubo Fu, Zhaofeng He

llms agents multimodal benchmarks

arXiv AI

Jul 28

SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows

arXiv:2607. 23123v1 Announce Type: new Abstract: Existing evaluations of large language models cover knowledge, reasoning, coding, and tool use, but they rarely treat a verifiable deliverable produced within a constrained workflow as the unit of evaluation.

By Summer Sun (Shaqiu Community)

llms agents benchmarks

arXiv AI

Jul 28

An Ontology for Machine Learning Interatomic Potentials

arXiv:2607. 23219v1 Announce Type: new Abstract: Machine learning interatomic potentials (MLIPs) approximate quantum-mechanical energies and forces---conventionally computed by density functional theory (DFT) or wave-function methods---at a fraction of the cost.

By Daniel Hern\'andez, Jong Hyun Jung, Yuji Ikeda, Yongliang Ou, Pranav Kumar, Tom Sch\"achtel, Wenchuan Liu, Xin Li, Xi Zhang, Xiang Xu, Lifang Zhu, Fritz K\"ormann, Steffen Staab, Blazej Grabowski

benchmarks

arXiv AI

Jul 28

Do LLMs Know Their Vulnerable Scenarios?

arXiv:2607. 23496v1 Announce Type: new Abstract: Safety-aligned large language models are trained to refuse harmful requests, yet embedding the same requests in particular scenarios can bypass their safeguards.

By Ziheng Peng, Huiqi Deng, Haoran Jing, Xuankun Rong, Jiahui Han, Xiting Wang, Na Zou, Xia Hu

llms rag benchmarks safety

arXiv AI

Jul 28

Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory

arXiv:2607. 23927v1 Announce Type: new Abstract: A conversational AI that cannot tell its own output from what a user said will treat its own mistakes as user-provided facts.

By Saurabh Ranjan, Konstantina Sokratous, Brian Odegaard

llms agents benchmarks safety

arXiv AI

Jul 28

GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models

arXiv:2607. 23913v1 Announce Type: new Abstract: Modern vision-language models (VLMs) increasingly rely on dynamic or high-resolution visual encoding, producing thousands of visual tokens that substantially increase downstream language-model inference cost.

By Jun Ling, Tao Huang, Junzhuo Liu, Bowen Tang, Peng Wang

llms multimodal benchmarks

arXiv AI

Jul 28

DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models

arXiv:2607. 23944v1 Announce Type: new Abstract: Human visual reasoning typically follows a coarse-to-fine attention process, starting from global scene understanding and gradually focusing on question-relevant regions.

By Hao Yang, Jin Wang, Xuejie Zhang

llms multimodal benchmarks safety

arXiv AI

Jul 28

The Cost of Knowing: A Resource-Aware Protocol for Benchmarking Hallucination Beyond Static Leaderboards

arXiv:2607. 24063v1 Announce Type: new Abstract: On standard factuality tasks, frontier models now cluster near the top of the scale.

By Keyu Li, Jin Gao, Dequan Wang

llms agents nlp benchmarks safety

arXiv AI

Jul 28

From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search

arXiv:2607. 24280v1 Announce Type: new Abstract: Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision.

By Junlin Liu, Jiangwang Chen, Zixin Song, Shuaiyu Zhou, Chunji Lv, Hank Wu, Kailin Jiang, Jinyang Wu, Bohan Yu, Chenxi Zhou

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 28

Comparing Optimization Models for Radiotherapy Scheduling

arXiv:2607. 22539v1 Announce Type: cross Abstract: The Radiotherapy Scheduling Problem (RTSP) involves determining an optimal schedule for patients undergoing radiation treatments, a task that has a massive impact on clinical outcomes given the central role of radiotherapy in cancer care.

By C. C. Rambaldi Migliore, D. Stanicel, N. Musliu, G. Iacca, M. Roveri

benchmarks

arXiv AI

Jul 28

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.

By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

llms reinforcement-learning benchmarks safety data-engineering

arXiv AI

Jul 28

Real-time Reconstruction of Human Visual Perception from fMRI

arXiv:2607. 22753v1 Announce Type: cross Abstract: Real-time closed-loop neurofeedback based on functional magnetic resonance imaging (fMRI) has led to important scientific and clinical advances.

By Rishab S. Iyer, Jiaxin Cindy Tu, Cesar Kadir Torrico Villanueva, Anish Mahishi, Ross P. Kempner, Jacob S. Prince, Ernest W. Lo, Akash Bhowmick, Hritik Arasu, Amaar Chughtai, Elizabeth A. McDevitt, Paul S. Scotti, Kenneth A. Norman

benchmarks

arXiv AI

Jul 28

Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training

arXiv:2607. 22794v1 Announce Type: cross Abstract: Automatic depression detection with deep learning has shown promise but often suffers from limited generalization due to domain shift arising from inter-speaker variability.

By Ali Tabaraei, Federico Simonetta, Stavros Ntalampiras

multimodal benchmarks safety

arXiv AI

Jul 28

Agentic Autoresearch for CT Reconstruction

arXiv:2607. 22824v1 Announce Type: cross Abstract: Comparing CT reconstruction methods fairly is labor-intensive and largely manual, and many benchmarks use idealized data.

By Andreas Maier, Lucas Kachelriess, Siming Bayer, Yixing Huang, Yan Xia, Amber Simpson, Moritz Zaiss

llms agents benchmarks

arXiv AI

Jul 28

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

arXiv:2607. 22864v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably.

By Patrick Rim, Tom Long, Ekta Prashnani, Ruth Rosenholtz, Ben Boudaoud, Peter Xenopoulos, Alex Wong, Joohwan Kim, Jae-Hyun Jung

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jul 28

Robustifying pathology foundation models via fine-tuning

arXiv:2607. 22861v1 Announce Type: cross Abstract: Pathology foundation models (FMs) produce powerful tile-level representations which remain sensitive to scanner and staining variability, undermining deployment across laboratories.

By Alexandre Filiot, Oskar Thaeter, Benoit Schmauch, Lionel Guillou

fine-tuning benchmarks

arXiv AI

Jul 28

Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval

arXiv:2607. 22841v1 Announce Type: cross Abstract: We present DS@GT's submission to FinMMEval 2026 Task 1, a multilingual financial exam question answering benchmark spanning English, Spanish, Greek, Chinese, and Hindi.

By Justice Ayela, Kabir Sahni

llms rag nlp benchmarks

arXiv AI

Jul 28

VecTree-RAG: An Agentic Retrieval-Augmented Generation Framework Combining Vector and Tree Retrieval for Efficiency and Accuracy

arXiv:2607. 23006v1 Announce Type: cross Abstract: Scientific question answering requires a retrieval system to solve two distinct problems: identifying which papers are relevant and locating the supporting evidence within those papers.

By Xinyan Zhong, Yuwei Shi, Yuqi Wei, Chen Shen, Tianhang Zhou, Zhenghao Wu

llms rag agents nlp benchmarks

arXiv AI

Jul 28

KAYROS: An Anytime and Exact Open-Source Solver for Duration-Minimization Time-Dependent Vehicle Routing. A Technical Report and a Case Study in Human-AI Engineering

arXiv:2607. 23116v1 Announce Type: cross Abstract: KAYROS is an open-source solver for duration-minimization time-dependent vehicle routing problems, with or without time windows (TDVRPTW, TDVRP).

By Florian Rascoussier

benchmarks

Get the daily briefing