The paper introduces a deterministic, reproducible e‑commerce environment that pre‑commits customer and trajectory parameters, enabling a simulated consumer to attempt purchasing a target cart with the help of an evaluated model. The environment records every assistant action and state, allowing post‑trial evaluation of specific conversation components and applying penalties based on tool‑call accuracy. Using this setup, the authors benchmark eight open‑weight agents (20B–35B parameters) across 160 trials and 44 metrics, revealing nuanced performance issues such as under‑action, over‑purchase, unsupported product attributes, and poor search that are hidden by overall success rates.
By Nimit Shah, Haitz S\'aez de Oc\'ariz Borde
CARTBench is a new benchmark for evaluating large language model shopping assistants on complex, ambiguous requests that involve timelines. It compiles 1,603 timestamped dialogues from 17 retail domains and pairs them with product records, then uses case‑specific rubrics based on Ideal Response Principles to score responses across Core, Excellence, and Safety criteria. In a full‑suite evaluation of ten commercial APIs and several open‑weight models, only three models scored above 90% on Core and none surpassed 45% on Excellence, revealing a gap between required behavior and high‑quality purchase advice.
By Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen
arXiv:2605.16116v2 Announce Type: replace
Abstract: Developing and evaluating e-commerce web agents requires environments that preserve meaningful task structure while enabling controllable, reproduc...
By Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang
arXiv:2607. 16239v1 Announce Type: new Abstract: AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges, tasks, and domains.
By Lei Shi, Anlan Zhang, Rita Lyu, Zhengmian Hu, Tong Yu, David Arbour, Avi Feller, Saayan Mitra, Ritwik Sinha
MOSS (Model Sanity Service) is a production framework that embeds LLM-based relevance scoring into a continuous monitoring loop for recommendation, search, and offer surfaces, using targeted sampling, strategy-aware prompting, and structured alerting to diagnose semantic failures. It has been deployed for over six months at a large U.S. retailer, identifying two confirmed serving defects and quantifying a 76.4% reduction in alert rate for a backfill blending fix. Benchmark results show that LLM judges outperform non-LLM baselines on both recommendation and search tasks, demonstrating that a single LLM monitoring framework can deliver actionable quality signals across multiple product discovery surfaces.
By Alina Nesen, Prathyusha Kanmanth Reddy, David Relyea, Rohit Avinash Mujumdar, Samaneh Karimi, Bo Shen, Jordan Hashemi
The paper introduces ASER, a pipeline that fine‑tunes a large language model to extract sensory attribute‑value pairs (e.g., color: matte black, scent: vanilla) from product reviews and distills them into a compact sensory bank for each catalog item. During recommendation, a frozen backbone uses a lightweight relational metric over this bank, achieving consistent improvements in HR@10 and NDCG@10 across five Amazon domains and four backbones, with evidence‑grounded extractions validated in 94.8% of cases. The study demonstrates that structured, evidence‑based sensory features enhance ranking quality beyond generic non‑sensory controls.
By Yeo Chan Yoon, Chanjun Park, Kyuhan Koh