The paper introduces a lifecycle framework for LLM-as-a-Judge systems used to evaluate recommendation explanations at Netflix. It outlines four phases—Birth, Training, Deployment, and Monitoring—detailing how each stage addresses specific technical and operational challenges. The authors report that after five weeks of A/B testing, judge-aligned explanations increased novel content viewing and successful browse-to-play sessions without quality takedowns.
By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Veli Balin, Rohan Gosain, Louis Garcia, Minsu Jang
The paper investigates how large language models (LLMs) can evaluate explanations in recommender systems. It generates 18 explanation prototypes and has 14 LLMs rate them, comparing the results to human ratings from a user study. Findings show that while LLMs mimic human rating patterns and correlate moderately with human judgments, their absolute agreement is low and varies with model size and evaluation design, leading to four practical recommendations for using LLMs in this context.
By Kathrin Wardatzky, Oana Inel, Luca Rossetto, Abraham Bernstein
The paper proposes treating an LLM agent’s long‑term memory selection as a budgeted slate recommendation problem, framing each request as a user, stored memories as items, and the context window as a slate. By applying recommender‑system techniques—candidate generation, learning‑to‑rank, and slate construction—AgentMemRec selects relevant memories without any LLM calls. Experiments on LongMemEval and LoCoMo show that AgentMemRec outperforms existing memory and context‑selection methods by 10–31 accuracy points and achieves comparable or better performance than an LLM reranker while reducing serving cost and eliminating LLM calls at write time.
By Xinyu Zhang, Weiwei Sun
CORAL is an LLM‑native harness that automates continual optimization of production recommender systems. It operates in a closed loop: an agent observes system signals, reasons over past decisions, and uses tools—including a numerical optimizer—to reconfigure the recommender while staying within a fixed operating budget. In A/B experiments on two large social platforms, CORAL improved engagement without extra serving cost on one platform and reduced serving cost without harming engagement on the other, demonstrating that a single agentic loop can replace manual engineering for ongoing system tuning.
By Muhammad Rafay Azhar, Yuhang Zhou, Gilbert Jiang, Yuchen Wang, Rahul Sharma, Matthew DeSousa, Jiayi Liu, Xin Guo, Lizhu Zhang, Xiangjun Fan
LIGE‑GR is a framework that transitions traditional ranking‑based recommender systems to a generative, listwise approach inspired by large language models. It extends existing pointwise recommendation models into a listwise generation system, enabling sequence‑level optimization without overhauling the entire infrastructure. Experiments on Instagram Reels and Facebook Video show modest gains in user time spent—1.14 % and 0.72 % respectively—while adding only slight inference overhead.
By Venkat Srinivas, Chenzhang He, Sam Woodmansee, Shawn Lian, Wenjie Hu, Renjie Jiang, Ziheng Huang, Xinyuan Zhang, Zhihao Zheng, Zhuoran Yu, Rui Li, Lei Yuan, Ziwei Li, Jimmy Jia, Mert Terzihan, Ekrem Kocaguneli, Yiming Liao, Zhichen Zhao, Yue Yin, Yue Weng, Wanlin Ma, Xufeng Cai, Weimiao Wu, Yezhou Huang, Du Zhang, Yukun Ding, Aaron Johnston, Yueming Wang, Zhaojie Gong, Yuting Zhang, Serena Li, Adithya Ganesh, Boying Liu, Haichuan Yang, Xialu Li, Matt Ma, Qunshu Zhang, John Joshua Miller, Praveen Rathinavelu, Cheng Huang, Aadhar Sachdeva, Josh Karns, Andres Aaron Gutierrez, Neil Agarwal, Gustas Pladis, Vladimir Batygin, Gopal Ray, Aditya Priyadarshi, Shantanu Patil, Zhe Wang, Penny Pan, Yiping Han, Arun Singh, Guangdeng Liao, Bi Xue, Xinyao Hu, Yang Song, Yisong Song, Meihong Wang, Haotian Wu, Deepak Agarwal, Ji Liu
The paper introduces PROVE-REC, a two‑pass framework that generates verifiable preference proofs for large language model (LLM) recommendation systems. In Pass A, the model condenses user interaction histories into a compact proof of positive and avoidance claims linked to specific evidence entries. Pass B then uses only this proof and its evidence to predict the next item, ensuring the recommendation follows the reasoning path. Verification steps compare masked evidence and removed claims to confirm grounding and influence, while a ranking‑preservation objective retains useful historical information. Experiments on diverse real‑world datasets show PROVE‑REC outperforms strong baselines by up to 7.45%, producing claims that are both better grounded and more influential to recommendation quality.
By Yu Hou, Nathaniel Kang, Pengkai Wang, Hua Li