Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,829 stories · RSS feed

arXiv AI

Jul 23

Fara-1.5: Scalable Learning Environments for Computer Use Agents

arXiv:2606. 20785v2 Announce Type: replace Abstract: Collecting computer use data from human demonstrations is expensive and slow, motivating the need for scalable generation strategies.

By Ahmed Awadallah, Sahil Gupta, Yash Lara, Yadong Lu, Hussein Mozannar, Akshay Nambi, Zach Nussbaum, Yash Pandya, Aravind Rajeswaran, Corby Rosset, Alexey Taymanov, Luiz do Valle, Vibhav Vineet, Spencer Whitehead, Andrew Zhao

llms agents fine-tuning benchmarks data-engineering

arXiv AI

Jul 23

Generative Augmented Inference of LLM-generated Data for Market Research: Theory and Empirical Evidence

arXiv:2604. 14575v3 Announce Type: replace-cross Abstract: Marketing research often relies on parameters estimated from costly human-generated data, such as conjoint survey responses, purchase decisions, and field experiment outcomes.

By Cheng Lu, Mengxin Wang, Dennis J. Zhang, Heng Zhang

llms benchmarks

arXiv AI

Jul 23

Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

arXiv:2607. 19678v1 Announce Type: cross Abstract: AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer.

By Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull, Gregory E Dean, Maria Liakata

llms nlp benchmarks

arXiv Machine Learning

Jul 23

Q-PhotoNAS: Hybrid Quantum Neural Architecture Search Framework on Photonic Devices

arXiv:2605. 22097v2 Announce Type: replace-cross Abstract: Photonic quantum computing is a promising platform for scalable quantum machine learning, but designing effective hybrid architectures remains challenging under hardware and optimization constraints.

By Farah Elnakhal, Alberto Marchisio, Nouhaila Innan, Gabriel Falcao, Muhammad Shafique

computer-vision benchmarks

arXiv AI

Jul 23

TRUST-ESD: A Risk-Calibrated and Governance-Aware AI Framework for Enterprise Strategic Decision Support Under Uncertainty

arXiv:2607. 20065v1 Announce Type: new Abstract: Enterprise strategic decision support requires AI systems that are not only accurate, but also uncertainty-aware, risk-calibrated, explainable, and governance-compliant.

By Tian Qiu, Li Yan, Mahabubur Rahman Miraj, Shanqin Yi, Md Intekhab Rahman Galib, Jahid Hasan

benchmarks

arXiv Machine Learning

Jul 23

JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models

arXiv:2607. 17572v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) is a powerful reinforcement learning algorithm for aligning generative models with human preferences.

By Ruiyi Ding, Jie Li, He Kang, Ziyan Liu, Chengru Song, Yuan chen

llms diffusion reinforcement-learning benchmarks

arXiv AI

Jul 23

Stateful Guardrails for Multi-Turn LLM Systems: A Conversational Risk Accumulation Framework

arXiv:2607. 19361v1 Announce Type: cross Abstract: Most safety guardrails for large language models (LLMs) evaluate each prompt-response pair in isolation, which misses failures that arise only over a dialogue as benign turns compose into harm.

By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik

llms benchmarks safety

arXiv AI

Jul 23

AgentCgroup: Understanding and Controlling OS Resources of AI Agents

arXiv:2602. 09345v3 Announce Type: replace-cross Abstract: AI agents are increasingly deployed in multi-tenant cloud environments, where they execute diverse tool calls within sandboxed containers, each call with distinct resource demands and rapid fluctuations.

By Yusheng Zheng, Jiakun Fan, Quanzhi Fu, Yiwei Yang, Wei Zhang, Andi Quinn

llms agents benchmarks

arXiv Machine Learning

Jul 23

QLIF-CAST: Quantum Leaky-Integrate-and-Fire for Time-Series Weather Forecasting

arXiv:2605. 18333v2 Announce Type: replace-cross Abstract: Accurate and efficient time-series forecasting remains a challenging problem for both classical and quantum neural architectures, particularly in multivariate environmental settings.

By Alberto Marchisio, Aayan Ebrahim, Nouhaila Innan, Muhammad Kashif, Muhammad Shafique

benchmarks

arXiv AI

Jul 23

Knowledge-Centric Self-Improvement

arXiv:2607. 19592v1 Announce Type: new Abstract: Self-improving AI systems typically treat the agent as the object that improves, by optimizing prompts, workflows, harnesses, or even the agent's own code.

By Xuefei Julie Wang, Lauren Hyoseo Yoon, Chengrui Qu, Amanda Zichang Wang, Atharva Sehgal, Eric Mazumdar, Yisong Yue

llms rag agents efficiency benchmarks

arXiv AI

Jul 23

Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering

arXiv:2607. 19867v1 Announce Type: cross Abstract: FinMMEval 2026 Task 2 evaluates short-answer financial question answering over multilingual evidence.

By Zhuohan Xie, Xueqing Peng, Georgi Georgiev, Dimitar Dimitrov, Yuyang Dai, Rania Elbadry, Vanshikaa Jani, Lingfei Qian, Fan Zhang, Jimin Huang, Jiahui Geng, Yankai Chen, Ye Yuan, Haolun Wu, Yuxia Wang, Ivan Koychev, Veselin Stoyanov, Mingzi Song, Yu Chen, Xue Liu, Preslav Nakov

llms rag nlp benchmarks

arXiv AI

Jul 23

A Systematic Benchmark of Intensity Normalisation Methods for 3D Knee MRI Segmentation and Cross-Domain Generalisability

arXiv:2607. 20028v1 Announce Type: cross Abstract: Robust out-of-the-box performance is essential for the clinical deployment of deep learning models in medical imaging.

By Oliver Mills, Philip Conaghan, Samuel Relton

computer-vision benchmarks

arXiv Machine Learning

Jul 23

TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs

arXiv:2602. 18109v3 Announce Type: replace Abstract: Real-time schedulers must reason about tight deadlines under strict compute budgets.

By Rong Fu, Yibo Meng, Zeyu Zhang, Ziming Guo, Jia Yee Tan, Xiaojing Du, Simon James Fong

llms rag nlp reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 23

Material-agnostic temperature field prediction for metal additive manufacturing via a parametric PINN framework

arXiv:2604. 14562v2 Announce Type: replace Abstract: Accurate temperature field prediction in metal additive manufacturing (AM) is essential for understanding the process-structure-performance relationship.

By Hyeonsu Lee, Jihoon Jeong

benchmarks

arXiv Machine Learning

Jul 23

Multi-stage Dynamic Selection for Cross-Project Defect Prediction

arXiv:2607. 20151v1 Announce Type: cross Abstract: Cross-Project Defect Prediction (CPDP) involves building models using data from external projects, called training projects, to predict modules from the target project.

By Juscimara G. Avelino, Juscelino S. A. Junior, George D. C. Cavalcanti, Rafael M. O. Cruz

benchmarks

arXiv AI

Jul 23

Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certification

arXiv:2607. 19442v1 Announce Type: cross Abstract: Machine unlearning is commonly evaluated by matching a retrained oracle on trained probes.

By Sen Yang, Yuen-Hei Yeung

benchmarks

arXiv Machine Learning

Jul 23

STN-TGAT: Top-K Portfolio Construction via Prior-Guided Graph Attention with Learnable Soft-Threshold Sparsification

arXiv:2607. 19385v1 Announce Type: new Abstract: This paper tackles the problem of stock ranking and portfolio construction under realistic investment settings by jointly modeling temporal dynamics and cross-sectional dependencies.

By Haoran Guo, Yutong Lu, Li Zhang

llms benchmarks

arXiv Machine Learning

Jul 23

Efficient Clustering with Provable Guardrails for LLM Inference at Scale

arXiv:2607. 19704v1 Announce Type: new Abstract: Scaling LLM-based applications to millions of users is bottlenecked by the inference cost and latency of modern foundation models.

By Longshaokan Wang, Wai Tsang Keung, Punit Ghodasara, Roman Wang, Ali Dashti, Francesc Moreno-Noguer

llms rag benchmarks safety

arXiv Machine Learning

Jul 23

PG-KINN: A Physics-Informed Petrov-Galerkin Kolmogorov-Arnold Network for Solving Forward and Inverse PDEs

arXiv:2607. 20378v1 Announce Type: new Abstract: Physics-informed learning of partial differential equations (PDEs) has been dominated by multilayer perceptrons (MLPs), whose spectral bias and dense parameterization limit both accuracy and interpretability.

By Amirhossein Sadr, Nima Soltani, Vahideh Moghtadaiee, Aida Pakniyat, Dara Rahmati, Saeid Gorgin

benchmarks safety

arXiv Machine Learning

Jul 23

CRB-Driven Beamforming and Trajectory Optimization for UAV-assisted ISAC System

arXiv:2607. 19609v1 Announce Type: cross Abstract: In this paper, we study an unmanned aerial vehicle (UAV)-assisted integrated sensing and communication (ISAC) system, where a UAV enhances the sensing capability of a base station (BS) towards a target while ensuring reliable communication towards a downlink user.

By Yi Yang, Qianqian Zhang, Huaxia Wang

reinforcement-learning benchmarks

Get the daily briefing