arXiv AI By Xinran Zhang, Pengrui Lu, Lyumanshan Ye, Pengfei Liu

ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies

Read the original on arXiv AI →

ERPBench is a benchmark that evaluates large language model agents in enterprise decision-making through a six‑round ERP simulation covering pricing, production, procurement, inventory, finance, and market competition. It tests the same 100 problems in two market ecologies—Solo, where agents compete against rule‑based opponents, and Arena, where six agents compete together—producing 1,200 model trajectories across 7,200 decision rounds. Results show that model performance varies by ecology, with DeepSeek best in Solo and Gemini best in Arena, and only 21 of 100 problems yield the same top performer across both settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Sep 1

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

E-Commerce Bench is an open‑source benchmark that simulates a year‑long e‑commerce operation, requiring LLM agents to manage multiple online stores, negotiate with suppliers, optimize sales, fulfill orders, handle returns, and manage cash flow. The environment uses real product and supplier data, a calendar of promotions and shocks, and deterministic customer and negotiation models to enable reproducible evaluation. The study evaluates 18 state‑of‑the‑art models across seven metrics, finding no single model dominates, with GPT‑5.6 Sol achieving the highest year‑end assets but lagging in fraud avoidance and operational efficiency.

By Wei Fan, Xinjie Shen, Xudong Guo, Jianhong Tu, Yang Su, Yinger Zhang, Lianghao Deng, Fengyu Wang, Baohua Dong, Yangqiu Song, Dayiheng Liu
arXiv AI
Oct 21

EGI-BENCH: An Enterprise General Intelligence Benchmark Suite

EGI‑BENCH is a unified benchmark suite for evaluating large language model agents in enterprise settings, consolidating eight existing benchmarks (CRMARENA, HERB, PERSONABENCH, USERBENCH, MCP‑UNIVERSE, MCPEVAL, BENCHFORCE, LOCOBENCH) under a single command‑line interface with two runtime‑budgeted modes. It introduces the Enterprise General Intelligence (EGI) Score, a six‑dimensional framework (M1‑M6) that aggregates fine‑grained sub‑metrics from each benchmark rather than simple averages, providing a more nuanced view of model capabilities. The paper reports results for fourteen LLMs across GPT, Claude, and Gemini families, showing varied strengths across task outcome, planning, retrieval, tool actioning, safety, and engineering quality.

By Zhiwei Liu, Zhujun Lan, Jielin Qiu, Zixiang Chen, Liangwei Yang, Ming Zhu, Akshara Prabhakar, Wenting Zhao, John Emmons, Jason Wu, Shafiq Rayhan Joty, Junnan Li, Shelby Heinecke, Silvio Savarese, Huan Wang
arXiv AI
Oct 2

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Argo-Bench is an evaluation framework that tests data agents on 210 enterprise-scale data science and analytics tasks, simulating a 2024 New York City food delivery platform with 81 million orders and 7.5 billion rows across 235 tables. Unlike traditional text‑to‑SQL benchmarks, the tasks require agents to navigate a realistic ERP warehouse, reconstruct facts, and perform actions such as banning accounts or allocating budgets, with outcomes scored by a simulator. Each task includes an executable reference solution that proves solvability using only the warehouse data.

By Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, Joseph J Ma