arXiv AI

AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance

Hugging Face Blog

Jan 21

AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality

arXiv AI

Jun 19

DynAMO:Dynamic Asset Management Orchestration via Topological Multi-Agent Scheduling

arXiv:2606. 19382v1 Announce Type: cross Abstract: While LLM-powered agents offer end-to-end automation for industrial asset lifecycles, real-world Industry 4.

By Kanishk Kushwaha, Vikrant Vinod Bansode, Harsh Vardhan, Dhaval C. Patel

llms agents efficiency benchmarks safety

arXiv AI

Aug 19

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

StartupBench is a new benchmark that evaluates general-purpose agents on end-to-end workflows derived from AI startup products that have proven market adoption. It translates real-world product workflows into deliverable-oriented tasks and assesses them with detailed rubrics. The study finds that even the best models complete only about 30% of these tasks, highlighting challenges such as complex instruction following and domain expertise.

By Liya Zhu, Xin Ma, Tao Liu, Haodong Wang, Ge Zhang, Jingzhe Ding, Qingshui Gu, Yongjie Zhong, Jinxiang Meng, Yuan Gao, Yunqiu Zhou, Hao Zhu, Jifeng He, Yongzhi Liao, Xinyi Zhang, Chaoxin Li, Yi Zhu, Xi Lin, Duju Zeng, Xiang Gao, Wen Zhang, Yunyang Wang, Duo Wang, Huan Zhou, Zuo Wang, Jin Chen, Kaiyuan Zhang, Chuqian Yu, Tianhao Yu, Longxiang Liu, Jianbo Xue, Huimin Che, Jiahao Wang, Yujia Qin, Jiaheng Liu, Shen Yan, Xiaolong Chang, Wenhao Huang

llms agents benchmarks

arXiv AI

Jul 3

AgenticDataBench: A Comprehensive Benchmark for Data Agents

arXiv:2607. 01647v1 Announce Type: cross Abstract: Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society.

By Zhaoyan Sun, Shan Zhong, Daizhou Wen, Jiaxing Han, Guoliang Li, Ying Yan, Peng Zhang, Yu Su, Xiang Qi, Baolin Sun, Chengyuan Yang, Tao Fang, Huaiyu Ruan

llms agents benchmarks

Hugging Face Trending Papers

Aug 18

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

StartupBench is a benchmark that evaluates general‑purpose agents on end‑to‑end workflows derived from real AI startup products that have proven market adoption. It translates these product workflows into deliverable‑oriented tasks and assesses them with detailed rubrics that capture complex requirements. Even the best current models complete only about 30% of the tasks, highlighting failures in instruction following and domain expertise.

llms agents benchmarks

arXiv AI

Jun 10

Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields

arXiv:2606. 11042v1 Announce Type: new Abstract: Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks.

By Liya Zhu, Jingzhe Ding, Jian Zhang, Jianbo Xue, Shihao Liang, Ge Zhang, Xiang Gao, Qingshui Gu, Mailun Gao, Huimin Che, Yan Zhao, Peiheng Zhou, Haojun Wang, Chaobo Xian, Lili Le, Chi Wu, Yiwei Liu, Shengda Long, Jiale Yang, Fangzhi Xu, Sijin Wu, Haodong Duan, Yi Zhu, Chao He, Zhaojian Li, Minchao Wang, Huan Zhou, Jiani Hou, Chuqian Yu, Weiran Shi, Hongwan Gao, Jiamin Chen, Guanhong Chen, Tingqin Luo, Kaiyuan Zhang, Zhixin Yao, Qing Hua, Yuhao Jiang, Jin Chen, Pu Chen, Zhenyu Hu, Xingyu Li, Zhengxuan Jiang, Meng Cao, Tianfeng Long, Haozhe Wang, Mingzhang Wang, Yichen Zhang, Yiming Dai, Chenchen Zhang, Jiaying Wang, Zhiyong Wu, Shen Yan, Yujia Qin, Wenhao Huang, Zaiyuan Wang, Xiaolong Chang

agents benchmarks

arXiv AI

Oct 15

AssetForge: Asset-Augmented Task Synthesis for Business Automation Agents

arXiv:2610.18330v1 Announce Type: new Abstract: Training business automation agents requires executable tasks that connect a natural-language request to an application state and a verifiable end stat...

By Xiangrui Liu, Shuo Tang, Siheng Chen

agents fine-tuning

arXiv AI

Sep 1

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...

By Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, Chetan Bansal

llms agents benchmarks

arXiv AI

Sep 30

AnyAct: Universal Action for Self-Evolving Agents

AnyAct introduces a universal action layer that consolidates diverse tool capabilities into a self‑evolving action space for AI agents operating in open‑world environments. It tackles the scale dilemma, tool non‑stationarity, and heterogeneous feedback by using hierarchical progressive retrieval and test‑time reliability evolution, while a heterogeneous observation grounding module unifies multi‑modal feedback. Evaluations on LiveMCPBench and the newly created OSMCP benchmark show state‑of‑the‑art performance, with significant gains in task success rate and reduced execution steps, especially for models with limited native capabilities.

By Lingrui Xu, Yangqin Jiang, Jiachang Zhang, Xubin Ren, Chao Huang

llms agents efficiency multimodal benchmarks

arXiv AI

Jul 7

Gypscie: A Cross-Platform AI Artifact Management System

arXiv:2604. 10311v2 Announce Type: replace Abstract: Artificial Intelligence (AI) models, encompassing both traditional machine learning (ML) and more advanced approaches such as deep learning and large language models (LLMs), play a central role in modern applications.

By Fabio Porto, Eduardo Ogasawara, Gabriela Moraes Botaro, Julia Neumann Bastos, Augusto Fonseca, Esther Pacitti, Patrick Valduriez

llms

arXiv AI

Jun 30

Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystem

arXiv:2606. 29116v1 Announce Type: new Abstract: Large Language Models (LLMs) are rapidly being adopted in low-code and no-code automation platforms, where non-expert users design workflows that combine natural language understanding with external services and APIs.

By Yutian Tang, Yuming Zhou, Huaming Chen

llms agents safety

arXiv AI

Jul 28

Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents

arXiv:2607. 22563v1 Announce Type: new Abstract: Industrial agent benchmarks require realistic evaluation scenarios that integrate telemetry, failure modes, maintenance records, and domain standards.

By Sagar Chethan Kumar, Rohith Kanathur, Dhaval Patel, Kaoutar El Maghraoui

llms agents benchmarks safety

Related stories