arXiv Machine Learning September 1, 2026 By Wei Fan, Xinjie Shen, Xudong Guo, Jianhong Tu, Yang Su, Yinger Zhang, Lianghao Deng, Fengyu Wang, Baohua Dong, Yangqiu Song, Dayiheng Liu

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

Read the original on arXiv Machine Learning →

E-Commerce Bench is an open‑source benchmark that simulates a year‑long e‑commerce operation, requiring LLM agents to manage multiple online stores, negotiate with suppliers, optimize sales, fulfill orders, handle returns, and manage cash flow. The environment uses real product and supplier data, a calendar of promotions and shocks, and deterministic customer and negotiation models to enable reproducible evaluation. The study evaluates 18 state‑of‑the‑art models across seven metrics, finding no single model dominates, with GPT‑5.6 Sol achieving the highest year‑end assets but lagging in fraud avoidance and operational efficiency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 3

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

arXiv:2607. 28956v1 Announce Type: new Abstract: Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria.

By Qiming Shi, Yulong Tao, Linbo Jin, Zhaolu Kang, Yibo Dou, Jiawen Zhu, Tianjun Pan, Shaokang Fu, Chengyu Wang, Siyue Li, Yaping Cheng, Di Weng, Chengfu Huo

llms agents benchmarks

arXiv AI

Jun 16

RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments

arXiv:2606. 15862v1 Announce Type: new Abstract: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.

By Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang

llms agents benchmarks

arXiv AI

Aug 11

When LLM Agents Negotiate: Private Information and Dynamic Bargaining in Supply Chains

arXiv:2608. 07538v1 Announce Type: new Abstract: As LLM agents move from decision support to autonomous procurement, firms need to know whether delegated negotiators create value, divide it predictably, and avoid money-losing contracts.

By Chen Liang, Fasheng Xu

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jul 10

RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments

arXiv:2603. 16453v3 Announce Type: replace Abstract: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.

By Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang

llms agents benchmarks

arXiv AI

Aug 11

Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

arXiv:2608. 08621v1 Announce Type: new Abstract: Running a business is a challenging form of intelligent work.

By Yijun Pan, Yukun Lian, Kunyu Shi, Junbo Li, Hongwei Xue, Sicong Xie, Guannan Zhang, Xiaoying Xing

llms agents benchmarks

arXiv AI

Oct 9

StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents

StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.

By Daksh Raghuvanshi, Ved Vedere, Yifan Wang

llms agents reinforcement-learning benchmarks

One email a morning, machine-written

Related stories

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments

When LLM Agents Negotiate: Private Information and Dynamic Bargaining in Supply Chains

RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments

Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents