Hugging Face Blog

IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST

arXiv AI
Aug 6

Terminal Agents Suffice for Enterprise Automation

arXiv:2604. 00073v3 Announce Type: replace-cross Abstract: There has been growing interest in building agents that can interact with digital platforms to execute meaningful enterprise tasks autonomously.

By Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, Sai Rajeswar
arXiv AI
Sep 17

ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software

ERPBench introduces a new evaluation paradigm for computer-use agents that operate via screenshots and simulated actions, focusing on enterprise software such as ERP systems. The benchmark tests agents on a live, reproducible ERP platform and scores tasks against ground-truth database values, highlighting challenges like dense interfaces, multi-step interactions, and persistent record errors. Experiments with six agents show that strong general GUI performance does not translate to reliable enterprise outcomes, with many agents frequently saving incorrect data.

By Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan, Yuying Zhao, Eugene Siow
Towards Data Science
Aug 18

Building Enterprise Agent Systems that People can Trust, Verify and Improve

The article outlines five principles that guide the successful deployment of enterprise agent systems, illustrated with a real-world example from a $100M+ company. It explains how these principles help ensure that such systems can be trusted, verified, and improved over time. The post serves as a practical guide for building reliable agent-based solutions in production environments.

By Sheila Teo
arXiv AI
Jul 3

Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

arXiv:2607. 01418v1 Announce Type: cross Abstract: Organizations rolling out agentic command line tools like Anthropic's Claude Code and GitHub's Copilot CLI need to know who will try them, who will keep using them, and whether the tools produce enough output to justify their cost.

By Emerson Murphy-Hill, Jenna Butler, Alexandra Savelieva
Towards Data Science
Aug 31

AgentOps Is Not MLOps: What Breaks in Your Monitoring Stack When Agents Go to Production

The article explains how the five core assumptions of MLOps monitoring are violated when agents are deployed to production, leading to inherited signals that incorrectly mark failed runs as healthy. It highlights the specific ways in which agent-based systems disrupt traditional monitoring stacks and the implications for reliability and performance. The piece serves as a warning for practitioners transitioning from MLOps to AgentOps, outlining the critical monitoring gaps that arise.

By Mostafa Ibrahim
Towards Data Science
Jul 10

The Big Con of Agentic AI

What our over-dependence on external consulting teaches us about delegating our minds to machines The post The Big Con of Agentic AI appeared first on Towards Data Science .

By Chinmay Kakatkar
arXiv AI
Jun 8

Measuring Agents in Production

arXiv:2512. 04123v4 Announce Type: replace-cross Abstract: LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful.

By Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Quincy Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, Koushik Sen, Dawn Song, Joseph E. Gonzalez, Ion Stoica, Matei Zaharia, Marquita Ellis
arXiv AI
Sep 7

ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies

ERPBench is a benchmark that evaluates large language model agents in enterprise decision-making through a six‑round ERP simulation covering pricing, production, procurement, inventory, finance, and market competition. It tests the same 100 problems in two market ecologies—Solo, where agents compete against rule‑based opponents, and Arena, where six agents compete together—producing 1,200 model trajectories across 7,200 decision rounds. Results show that model performance varies by ecology, with DeepSeek best in Solo and Gemini best in Arena, and only 21 of 100 problems yield the same top performer across both settings.

By Xinran Zhang, Pengrui Lu, Lyumanshan Ye, Pengfei Liu