MOSS (Model Sanity Service) is a production framework that embeds LLM-based relevance scoring into a continuous monitoring loop for recommendation, search, and offer surfaces, using targeted sampling, strategy-aware prompting, and structured alerting to diagnose semantic failures. It has been deployed for over six months at a large U.S. retailer, identifying two confirmed serving defects and quantifying a 76.4% reduction in alert rate for a backfill blending fix. Benchmark results show that LLM judges outperform non-LLM baselines on both recommendation and search tasks, demonstrating that a single LLM monitoring framework can deliver actionable quality signals across multiple product discovery surfaces.
arXiv:2606. 17698v1 Announce Type: new Abstract: As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked.
The paper proposes a single hierarchical Semantic ID (SID) system to unify product identification across multiple merchants in e-commerce. By learning SID representations from product content, the authors demonstrate that ranking algorithms can aggregate consumer affinity and product performance over SID prefixes, improving offline relevance and online engagement. For query reformulation, SID concepts guide navigation and refinement, yielding better intent preservation and higher-quality suggestions compared to taxonomy or raw query transitions.
arXiv:2607. 23121v1 Announce Type: cross Abstract: Dynamic Product Ads (DPA) require retrieving relevant items from multi-million product catalogs, balancing two competing objectives: retargeting (re-surfacing known interests) and prospecting (discovering new categories).
arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.
RPTune is an end‑to‑end framework that improves in‑context catalog search for small merchant businesses by learning to curate product catalogs and fine‑tuning large language models (LLMs) with catalog‑grounded supervision. It uses an encoder‑reorganizer curator to order and prune products based on LLM feedback, and then applies context‑relative rewards during LLM post‑training. Across seven real merchants and 100 complex conversational queries per merchant, RPTune boosts search accuracy by up to 31.4 percentage points from curation alone and an additional 10.3 points on average from post‑training.
The paper introduces a lifecycle framework for LLM-as-a-Judge systems used to evaluate recommendation explanations at Netflix. It outlines four phases—Birth, Training, Deployment, and Monitoring—detailing how each stage addresses specific technical and operational challenges. The authors report that after five weeks of A/B testing, judge-aligned explanations increased novel content viewing and successful browse-to-play sessions without quality takedowns.
arXiv:2606. 29947v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes.
The paper introduces Mine and Refine, a two‑stage contrastive training framework designed to improve embedding‑based retrieval for large‑scale e‑commerce search. It tackles graded relevance, hard‑sample mining, and unstable similarity separability by using a lightweight LLM as a scalable labeler and a multi‑level circle loss to enforce margin‑controlled separation across relevance levels. The method has been deployed in production across multiple product verticals, yielding statistically significant increases in user engagement, gross order value, and retrieval relevance metrics.
The paper introduces Continual Search, an iterative framework that guides large language models to persistently search for diagnostic evidence in long AI agent execution logs, addressing the limitations of one-shot judgments. Evaluated on four existing RCA benchmarks and a new large-scale dataset called MegaRCA-Mix, Continual Search consistently boosts attribution performance, achieving a 40% F1 improvement for GPT‑5.5 on MegaRCA‑Mix. The results show that effective search can outweigh raw model scale, enabling lower-tier models to outperform higher-tier ones in root‑cause attribution tasks.
PACEShop introduces a new evaluation framework, PACE, for shopping assistants that emphasizes personalized, actionable, compositional, and evidence‑grounded responses. The benchmark dataset contains 22,625 records with structured personas, auditable evidence pools, and detailed defect annotations, while PACEJudge offers a training‑free protocol for assessing these dimensions. Experiments demonstrate that generic judges miss key diagnostic fields, whereas PACEJudge improves evaluation across persona alignment, cross‑component consistency, grounding, and defect localization without retraining.
LLMAR is a tuning‑free recommendation framework designed for sparse, text‑rich industrial B2B domains. It transforms user behavioral history into structured semantic motives using LLM inference, employs a reflection loop to self‑correct hallucinations, and operates cost‑effectively with asynchronous batch processing. Experiments on MovieLens‑1M, Amazon Prime Pantry, and a construction risk dataset show LLMAR surpasses state‑of‑the‑art learning models, achieving up to a 54.6% nDCG@10 improvement while keeping inference costs around $1 per 1,000 users.
FORTE is a model‑agnostic production framework that discovers, validates, and removes features from industrial recommender systems. It uses efficient perturbation and grouped ablation to evaluate candidate feature sets daily, and lineage tracing to identify all downstream models affected. In two large‑scale systems, FORTE removed up to 43.3% of features without online regressions, cutting trainable parameters by 29.9% and improving serving throughput by 19.8‑25.5%.