Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,633 stories · RSS feed

arXiv AI

Jul 21

AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Language

arXiv:2607. 16372v1 Announce Type: cross Abstract: Interactive theorem proving (ITP) underpins program verification and formalized mathematics, but its manual effort limits scalability.

By Qiyuan Xu, Joshua Ong Jun Leang, Renxi Wang, Wenda Li, Haonan Li, Luke Ong, Conrad Watt

llms agents benchmarks

arXiv AI

Jul 21

Fantastic Adaptive Taxonomies and How to Use Them

arXiv:2607. 16387v1 Announce Type: cross Abstract: An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback.

By Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica

llms agents benchmarks

arXiv AI

Jul 21

A Quantum-Classical Hybrid Framework for Multivariate Time-Series Forecasting Complexity-Fidelity Trade-offs and Limitations

arXiv:2607. 16358v1 Announce Type: cross Abstract: This paper presents a unified quantum-classical hybrid framework for multi-horizon time-series forecasting, introducing two model variants Quantum Reservoir Forecaster (QRC-F) and Variational Quantum Forecaster (VQF-F).

By Sanjay Chakraborty, Fredrik Heintz

efficiency benchmarks

arXiv AI

Jul 21

Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration

arXiv:2607. 16352v1 Announce Type: cross Abstract: A fundamental intent asymmetry plagues modern 3D asset creation: while state-of-the-art 3D toolchains demand precise, executable parameters, ordinary users typically provide vague, underspecified instructions.

By Xiaoye Zhu, Weixin Li, Junan Huo, Bozhong Wang, Jia Zeng, Yi Yang, Cen Chen, Qi Liu

agents benchmarks

arXiv AI

Jul 21

Privacy-Aware Synthetic Video Benchmarking and Relational Evaluation for Worker-Under-Suspended-Load Detection

arXiv:2607. 16351v1 Announce Type: cross Abstract: Publicly shareable construction-video benchmarks remain scarce, especially for safety-critical hazards that are rare, dangerous to stage, and difficult to release.

By Anshu Singh, Alejandro Seif

diffusion computer-vision benchmarks safety

arXiv AI

Jul 21

Joint-Embedding Predictive Architecture for Sensor-based Activity Recognition

arXiv:2607. 16350v1 Announce Type: cross Abstract: Sensor-based human activity recognition (HAR) has achieved significant progressed in fully supervised learning settings.

By Mohd Halim Mohd Noor, Abdulrahman M. A. Baraka

rag benchmarks

arXiv AI

Jul 21

Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.

By Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin

llms agents multimodal benchmarks

arXiv AI

Jul 21

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

arXiv:2607. 16243v1 Announce Type: cross Abstract: Multimodal industrial anomaly inspection assistants are a critical component of next-generation smart factories, enabling interactive vision-language-based querying.

By Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

llms multimodal benchmarks safety

arXiv AI

Jul 21

Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

arXiv:2607. 18086v1 Announce Type: new Abstract: Background: LLM judges increasingly score whether clinical language models give overconfident answers under incomplete evidence, yet whether a measured "safety gain" reflects real behavior change or the judge's calibration is unresolved.

By Koyar Afrasyab

llms benchmarks safety

arXiv AI

Jul 21

Enhancing Rubric-based RL via Self-Distillation

arXiv:2607. 18082v1 Announce Type: cross Abstract: Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks.

By Mingxuan Xia, Yuhang Yang, Chao Ye, Shuai Zhu, Shenzhi Yang, Guangcheng Zhu, Yuhang Zhang, Cheng Peng, Haobo Wang, Siqing Wang

llms reinforcement-learning efficiency benchmarks

arXiv AI

Jul 21

Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning

arXiv:2607. 16209v1 Announce Type: new Abstract: Context reranking and pruning have become essential for improving the efficiency of modern Retrieval-Augmented Generation (RAG) systems, yet an interpretable and unified framework remains underexplored.

By Yanqiao Chen, Dongsheng Hou, Yuhan Rui, Zhen Cao, Yepang Liu

llms rag fine-tuning efficiency benchmarks

arXiv AI

Jul 21

Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

arXiv:2607. 16212v1 Announce Type: new Abstract: Large language models hallucinate numbers and units when summarizing scientific text, a failure mode that can silently invert a scientific claim.

By Genpei Zhang

llms fine-tuning benchmarks

arXiv AI

Jul 21

It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

arXiv:2607. 16205v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self generated rollouts.

By Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Liwei Qian, Xin Pei, Jizhou Huang

llms reinforcement-learning benchmarks

arXiv AI

Jul 21

LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.

By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang

llms rag multimodal benchmarks

arXiv AI

Jul 21

"Not in My Backyard": LLMs Uncover Online and Offline Social Biases Against Homelessness

arXiv:2508. 13187v4 Announce Type: replace-cross Abstract: Homelessness is a persistent social challenge, impacting millions worldwide.

By Jonathan A. Karr Jr., Benjamin F. Herbst, Matthew L. Sisk, Xueyun Li, Ting Hua, Matthew Hauenstein, Georgina Curto, Nitesh V. Chawla

llms benchmarks safety

arXiv AI

Jul 21

Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory

arXiv:2607. 17545v1 Announce Type: new Abstract: Language agents depend on memory across interactions.

By Qingcan Kang, Mingyang Liu, Shixiong Kai, Kaichao Liang, Zhentao Tang, Yuqi Cui, Tao Zhong, Mingxuan Yuan

llms agents benchmarks safety

arXiv Machine Learning

Jul 21

Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?

arXiv:2607. 17425v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) compress model activations into sparse codes, but equal reconstruction error and sparsity can preserve different linearly decodable signals.

By Aniket Deshpande

llms efficiency benchmarks

arXiv AI

Jul 21

Environment-free Synthetic Data Generation for API-Calling Agents

arXiv:2607. 16900v1 Announce Type: new Abstract: Training API-calling large language model (LLM) agents demands massive amounts of high-quality trajectories.

By Seanie Lee, Sanjoy Chowdhury, Chao Jiang, Cheng-Yu Hsieh, Ting-Yao Hu, Alexander T Toshev, Oncel Tuzel, Raviteja Vemulapalli

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jul 21

Depth-Regularized JEPA World Models Learn More Transferable Representations from Real Outdoor Robot Data

arXiv:2607. 16314v1 Announce Type: cross Abstract: World models, especially based on JEPA architectures, have been shown to learn robust dynamics of various environments.

By Usman M. Khan

robotics benchmarks

arXiv Machine Learning

Jul 21

When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

arXiv:2602. 14161v2 Announce Type: replace Abstract: Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization.

By Max Fomin

llms agents benchmarks safety

Get the daily briefing