Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,285 stories · RSS feed

arXiv AI

Jun 30

TraceLab: Characterizing Coding Agent Workloads for LLM Serving

arXiv:2606. 30560v1 Announce Type: cross Abstract: Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging.

By Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci

llms agents benchmarks

arXiv AI

Jun 30

SVC-Probe: A Framework for Evaluating Perturbation Generalization in Spatial Foundation-Model Embeddings

arXiv:2606. 28465v1 Announce Type: cross Abstract: This work examines perturbation generalization in spatial foundation-model embeddings derived from fluorescence microscopy images.

By Jake Y. Chen, Huu Phong Nguyen, Fuad Al Abir, Ehsan Saghapour

rag benchmarks

arXiv AI

Jun 30

An Agentic AI Pipeline for Appliance-Level Energy Anomaly Detection and LLM-Driven Recommendations

arXiv:2606. 28467v1 Announce Type: cross Abstract: Appliance-level energy monitoring in office buildings produces noisy alerts that non-expert facility managers struggle to use.

By Dihia Falouz, Aida Douaibia, Amine Bechar, Youssef Elmir, Abbes Amira, Adel Oulefki

llms rag agents benchmarks

arXiv AI

Jun 30

RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation

arXiv:2606. 28410v1 Announce Type: cross Abstract: Open-vocabulary semantic segmentation (OVSS) enables text-guided segmentation of unseen objects, breaking fixed-class limitations to achieve open-world understanding.

By Shanwen Wang, Xin Sun, Sirui Wang, Xiao Xiang Zhu

llms computer-vision benchmarks safety

arXiv Machine Learning

Jun 30

Multi-Block Diffusion Language Models

arXiv:2606. 29215v1 Announce Type: new Abstract: Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation.

By Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu, Yi Tu, Jiajun Li, Dandan Tu, Xiaohui Yan, Kai Yu, Pengfei Liu, Zhijie Deng

llms diffusion benchmarks

arXiv Machine Learning

Jun 30

Randomized neural operator for parametric PDEs with fast training and conformal uncertainty quantification

arXiv:2606. 29440v1 Announce Type: new Abstract: Repeatedly solving parametric PDEs is essential for uncertainty quantification, design optimization and inverse problems, but conventional neural operators require expensive non-convex training.

By Zirui Deng, Jingbo Sun, Deyu Meng, Fei Wang

benchmarks

arXiv AI

Jun 30

Aligning Language Model Benchmarks with Pairwise Preferences

arXiv:2602. 02898v3 Announce Type: replace Abstract: Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance.

By Marco Gutierrez, Xinyi Leng, Hannah Cyberey, Jonathan Richard Schwarz, Ahmed Alaa, Thomas Hartvigsen

llms benchmarks safety

arXiv AI

Jun 30

Steerable Visual Representations

arXiv:2604. 02327v2 Announce Type: replace-cross Abstract: Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation.

By Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, Yuki M. Asano

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jun 30

DRIFT: A Benchmark for Task-Free Continual Graph Learning with Continuous Distribution Shifts

arXiv:2605. 12998v3 Announce Type: replace Abstract: Continual graph learning (CGL) aims to learn from dynamically evolving graphs while mitigating catastrophic forgetting.

By Guiquan Sun, Xikun Zhang, Jingchao Ni, Dongjin Song

benchmarks

arXiv AI

Jun 30

An AI agent for treatment reasoning over a biomedical tool universe

arXiv:2606. 28692v1 Announce Type: new Abstract: Treatment reasoning underpins every therapeutic decision, integrating disease context, comorbidities, medications, contraindications, and evolving biomedical knowledge to select an appropriate therapy.

By Shanghua Gao, Ayush Noori, Richard Zhu, Curtis Ginder, Zhenglun Kong, Xiaorui Su, Justin Kauffman, Benjamin S. Glicksberg, Joshua Lampert, Ankit Sakhuja, Ashwin Sawant, ATHENA-R1 Evaluation Consortium, David A. Clifton, Noa Dagan, Ran Balicer, Marinka Zitnik

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF

arXiv:2606. 29758v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training.

By Doo Hwan Hwang, Kee-Eung Kim

llms reinforcement-learning benchmarks

arXiv AI

Jun 30

LAMP: Lean-based Agentic framework with MCP and Proof Repair

arXiv:2606. 28841v1 Announce Type: cross Abstract: Large language models are increasingly capable of mathematical reasoning, but the proofs they generate are often unreliable and hard to verify.

By Santhana Srinivasan R, Maithilee Patawar

llms agents fine-tuning benchmarks

arXiv AI

Jun 30

LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis

arXiv:2606. 28362v1 Announce Type: cross Abstract: Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort.

By Yen-Hsun Huang (Department of Education, Taipei Veterans General Hospital, Taipei, Taiwan), Yu-Shiou Lin (Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan)

llms agents benchmarks

arXiv AI

Jun 30

Optimizing Expert-Designed Crystal Graph Networks for Band-Gap Prediction with an Autonomous LLM Research Loop

arXiv:2606. 29717v1 Announce Type: cross Abstract: Predicting a material's properties from its structure is a central, fast-advancing problem in computational materials science.

By Chenmu Zhang, Boris I. Yakobson

llms rag agents benchmarks

arXiv AI

Jun 30

ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation

arXiv:2606. 28357v1 Announce Type: cross Abstract: Recent advances in multimodal recommenders excel at feature fusion but remain opaque and inefficient decision-makers, lacking explicit reasoning and self-awareness of uncertainty.

By Yihua Zhang, Mingfu Liang, Jiyan Yang, Rong Jin, Wen-Yen Chen, Yiping Han, Huayu Li, Buyun Zhang, Liang Luo, Frank Shyu, Luke Simon, Sijia Liu, Tianlong Chen, Xi Liu

agents fine-tuning efficiency multimodal benchmarks

arXiv AI

Jun 30

PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation

arXiv:2606. 28344v1 Announce Type: cross Abstract: Augmenting large language models (LLMs) with retrieved web text has become a dominant paradigm, yet the web is not natively textual: existing systems depend on complex parsing pipelines that linearize HTML and discard layout, visual structure, and formatting.

By Yichuan Wang, Zhifei Li, Zirui Wang, Paul Teiletche, Lesheng Jin, Matei Zaharia, Joseph E. Gonzalez, Sewon Min

llms rag agents multimodal benchmarks

arXiv AI

Jun 30

From Regulatory Approvals to Patents: Cross-Domain Linking for Cardiovascular Device Traceability

arXiv:2606. 28353v1 Announce Type: cross Abstract: Linking FDA-approved medical devices to their underlying United States Patent and Trademark Office (USPTO) patents enables critical applications such as recall root-cause analysis, M&A-driven IP discovery, and technology trajectory mapping.

By Yang Qingqing, Liu Haijiang, Li Moyan

llms rag benchmarks

arXiv Machine Learning

Jun 30

HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction

arXiv:2606. 26744v2 Announce Type: replace Abstract: We present HyperDFlash, a block-parallel speculative decoding framework tailored to DeepSeek-V4's Hyper-Connections (HC).

By Luxi Lin, Shuang Peng, Rui Ma, Junhao Hua, Shuwei Fan, Zhengda Qin, Qiang Wang, Hongjian Sun, Fangmin Chen, Songwei Liu

efficiency benchmarks safety

arXiv AI

Jun 30

Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents

arXiv:2606. 30383v1 Announce Type: new Abstract: A rapidly growing class of LLM agents is multi-party: the agent acts for a principal (who briefs it, sends follow-ups, and receives results) while also conversing in a separate channel with a counterparty whose interests may diverge (negotiating with a vendor, screening inbound requests, or mediating between employees).

By Bojie Li, Noah Shi

llms agents efficiency benchmarks safety

arXiv AI

Jun 30

HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning

arXiv:2606. 28349v1 Announce Type: cross Abstract: Long-context reasoning requires models to access, retrieve, and integrate evidence scattered across documents, dialogues, and accumulated interaction histories.

By Zeju Li, Ziyang Zheng, Yizhou Zhou, Qiang Xu

rag agents benchmarks

Get the daily briefing