arXiv Machine Learning
Aug 6

The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing

arXiv:2608. 04432v1 Announce Type: cross Abstract: On two-sided content platforms, symmetric two-sided isolation (assigning matched fractions of creators and viewers to isolated treatment and control submarkets) is widely used for creator-side and cold-start experiments because it removes cross-arm marketplace interference.

By Yuanyuan Shen, Yiren Yan, Wenjie Li, Chunhui Zhu
arXiv Machine Learning
Oct 21

Baseline and Incremental: User, Item, and Request Debiasing in Instagram Feed Ranking

The paper proposes a multi‑signal value model for Instagram Feed ranking that separates baseline predictions from incremental gains, arguing that retaining raw predictions and adding incremental terms improves ranking quality. It describes how three independent debiasing families—viewer habit, item popularity, and request context—are combined into a single parameterized operator, with a design procedure selecting optimal points. Controlled ablation studies show that the two‑term model outperforms single‑term variants in most metrics, and the approach has been adopted across Meta’s products, boosting key engagement metrics.

By Hsin-Chiang Chen, Hao Wang, Gedi Zhou, Mingze Gao, Abhishek Kumar, Yiben Yang, Haoyu Wang, Hongjing Xia, Bo Chen, Zhengyu Su, Xian Sun, Chang Xu, Kevin Han, Weiwei Li, Zhongchang Sun, Yao Zhang, Qifei Wang, Peggy Yang, Yihan Wu, Haoran Liu, Siyang Sun, Xingyu Pan, Shannon Phu, Haoming Chen, Kevin Zhao, Chen Liang
arXiv AI
Sep 25

Delay-of-Gratification as a Multi-Agent Survival Micro-benchmark for Long-Horizon LLMs: Social Exposure, Personas, and Tool Use Budgets

The paper introduces a new multi‑agent micro‑benchmark called Delay‑of‑Gratification, modeled after the Stanford marshmallow experiment, to evaluate large language models (LLMs) in long‑horizon, multi‑turn interactions. In the benchmark, ReAct agents use a per‑step “raise a question” tool under various constraints—social context (broadcast vs. isolated), persona traits (age, hedonic drive), and tool‑use policy (mandatory vs. optional). Across 19,200 trajectories, the study finds that most agents exhibit an early impulse to “eat,” only 75.9% persist to the end, and factors such as isolation and hedonic drive significantly influence survival and questioning behavior, with ablations showing that removing hedonic drive and age can improve completion rates.

By Olga Manakina, Igor Bogdanov, Chung-Horng Lung
arXiv AI
Aug 20

The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

The paper introduces a lifecycle framework for LLM-as-a-Judge systems used to evaluate recommendation explanations at Netflix. It outlines four phases—Birth, Training, Deployment, and Monitoring—detailing how each stage addresses specific technical and operational challenges. The authors report that after five weeks of A/B testing, judge-aligned explanations increased novel content viewing and successful browse-to-play sessions without quality takedowns.

By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Veli Balin, Rohan Gosain, Louis Garcia, Minsu Jang