arXiv AI

PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants

PACEShop introduces a new evaluation framework, PACE, for shopping assistants that emphasizes personalized, actionable, compositional, and evidence‑grounded responses. The benchmark dataset contains 22,625 records with structured personas, auditable evidence pools, and detailed defect annotations, while PACEJudge offers a training‑free protocol for assessing these dimensions. Experiments demonstrate that generic judges miss key diagnostic fields, whereas PACEJudge improves evaluation across persona alignment, cross‑component consistency, grounding, and defect localization without retraining.

arXiv Machine Learning
Sep 16

Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification

The paper introduces a deterministic, reproducible e‑commerce environment that pre‑commits customer and trajectory parameters, enabling a simulated consumer to attempt purchasing a target cart with the help of an evaluated model. The environment records every assistant action and state, allowing post‑trial evaluation of specific conversation components and applying penalties based on tool‑call accuracy. Using this setup, the authors benchmark eight open‑weight agents (20B–35B parameters) across 160 trials and 44 metrics, revealing nuanced performance issues such as under‑action, over‑purchase, unsupported product attributes, and poor search that are hidden by overall success rates.

By Nimit Shah, Haitz S\'aez de Oc\'ariz Borde
arXiv Computation and Language
Nov 5

CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

CARTBench is a new benchmark for evaluating large language model shopping assistants on complex, ambiguous requests that involve timelines. It compiles 1,603 timestamped dialogues from 17 retail domains and pairs them with product records, then uses case‑specific rubrics based on Ideal Response Principles to score responses across Core, Excellence, and Safety criteria. In a full‑suite evaluation of ten commercial APIs and several open‑weight models, only three models scored above 90% on Core and none surpassed 45% on Excellence, revealing a gap between required behavior and high‑quality purchase advice.

By Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen
arXiv Machine Learning
Jul 21

BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges

arXiv:2607. 16239v1 Announce Type: new Abstract: AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges, tasks, and domains.

By Lei Shi, Anlan Zhang, Rita Lyu, Zhengmian Hu, Tong Yu, David Arbour, Avi Feller, Saayan Mitra, Ritwik Sinha
arXiv AI
Oct 21

MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems

MOSS (Model Sanity Service) is a production framework that embeds LLM-based relevance scoring into a continuous monitoring loop for recommendation, search, and offer surfaces, using targeted sampling, strategy-aware prompting, and structured alerting to diagnose semantic failures. It has been deployed for over six months at a large U.S. retailer, identifying two confirmed serving defects and quantifying a 76.4% reduction in alert rate for a backfill blending fix. Benchmark results show that LLM judges outperform non-LLM baselines on both recommendation and search tasks, demonstrating that a single LLM monitoring framework can deliver actionable quality signals across multiple product discovery surfaces.

By Alina Nesen, Prathyusha Kanmanth Reddy, David Relyea, Rohit Avinash Mujumdar, Samaneh Karimi, Bo Shen, Jordan Hashemi
arXiv AI
Oct 5

Sensory-Aware Sequential Recommendation via Review-Distilled Representations

The paper introduces ASER, a pipeline that fine‑tunes a large language model to extract sensory attribute‑value pairs (e.g., color: matte black, scent: vanilla) from product reviews and distills them into a compact sensory bank for each catalog item. During recommendation, a frozen backbone uses a lightweight relational metric over this bank, achieving consistent improvements in HR@10 and NDCG@10 across five Amazon domains and four backbones, with evidence‑grounded extractions validated in 94.8% of cases. The study demonstrates that structured, evidence‑based sensory features enhance ranking quality beyond generic non‑sensory controls.

By Yeo Chan Yoon, Chanjun Park, Kyuhan Koh
arXiv AI
Sep 28

CARGO: Context-Aware Retrieval-Gated Evaluation of Agentic AI in Production

CARGO is a framework for evaluating agentic AI systems in production that addresses the problem of reference-instance divergence (RID), where reference-based judges penalize correct answers that involve different entity identifiers. It treats retrieved references as procedural exemplars, grounds judgments in the live instance’s context, assigns a three-way status to claims, and gates evaluation by retrieval confidence. Using the CARGO-Bench diagnostic suite, CARGO eliminates false penalties and improves discrimination while revealing a limitation in detecting procedural corruptions.

By Mukul Chhabra, Shail Patel, Luigi Medrano
arXiv AI
Oct 31

SkillCT: A Multidimensional Benchmark and Execution-Free Baseline for Agent Skill Assessment

SkillCT is a new benchmark that expands agent skill assessment beyond downstream task execution to a multidimensional evaluation across nine dimensions grouped into Static, Dynamic, and Human-Assisted assessments. It measures specification quality, behavioral robustness, semantic alignment, and responsibility boundaries without requiring repeated executions. The benchmark also introduces an execution-free baseline that predicts skill quality profiles by comparing to similar reference skills, reducing token usage and evaluation time while enabling skill refinement and repository maintenance.

By Mengyu Sun, Zunlong Zhou, Ziyuan Yang, Yi Zhang
arXiv AI
Aug 20

The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

The paper introduces a lifecycle framework for LLM-as-a-Judge systems used to evaluate recommendation explanations at Netflix. It outlines four phases—Birth, Training, Deployment, and Monitoring—detailing how each stage addresses specific technical and operational challenges. The authors report that after five weeks of A/B testing, judge-aligned explanations increased novel content viewing and successful browse-to-play sessions without quality takedowns.

By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Veli Balin, Rohan Gosain, Louis Garcia, Minsu Jang
arXiv AI
Sep 30

From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks

The paper investigates how the design of LLM-as-a-Judge protocols influences both the intrinsic quality of judgments and their downstream utility in open-ended tasks. By varying verdict granularity, critique usage, and evaluation batching, and by applying Judge guidance to test-time inference methods such as Best-of-N selection, revision, and beam search, the authors find that judgment quality and downstream performance do not always align and that protocol choices significantly affect outcomes. The study highlights the need for comprehensive evaluation of LLM Judges that considers both judgment quality and practical utility.

By Zheng Zhang, Lufei Li, Xinyue Tan, Yuanhao Zeng, Ziwei Shan, Yexin Li, Kan Ren
arXiv AI
Jun 3

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

arXiv:2606. 03650v1 Announce Type: cross Abstract: Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness.

By Alexander Apartsin, Yehudit Aperstein
arXiv AI
2d ago

Toward Agentic Retrieval-Augmented Generation for Enterprise Retail Workflows over Heterogeneous Data Sources

The paper introduces RetailBridge, a benchmark for evaluating agentic retrieval‑augmented generation (RAG) in enterprise retail workflows that involve chaining evidence across heterogeneous data sources. RetailBridge uses persona‑driven simulations to create questions that require agents to identify relevant sources, transfer evidence between them, and produce grounded answers. The authors also present RetailRAG, a retail‑aware framework that outperforms existing agentic RAG baselines and highlights multi‑source chaining as the main bottleneck for large language models in retail contexts.

By Jeonghwan Choi, Jibin Hwang, Hayoon Park, Hyeonjae Cheon, Hwanjun Song