Offline A/B Testing of Slate Recommendation Systems with LLMs: Reducing the Dependency on Pre-Collected User Interaction Data

The paper explores using large language models (LLMs) to generate pairwise preferences between slates for synthetic A/B testing of slate recommendation systems. It introduces a validation protocol that checks how well these synthetic preferences align with traditional RecSys metrics and satisfy preference axioms, and examines how LLM pre‑training and configuration influence preference articulation. By combining the synthetic preferences with a generalized Rao‑Kupper model, the authors show that LLM‑based A/B testing can recover stable ranking orderings across different utility weightings, offering a cost‑effective screening step before conducting live experiments.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• safety