The paper examines how different preprocessing steps in fine‑tuning data influence the performance and fairness of a generative large language model recommender system. Using a controlled pipeline that includes supervised fine‑tuning, catalog‑constrained decoding, and prompt augmentation, the authors evaluate three preprocessing choices—item‑title representation, demographic balancing, and auxiliary item‑knowledge enrichment—on the MovieLens‑1M dataset. Results show that catalog constraints eliminate out‑of‑catalog items, prompt augmentation improves consistency and utility, and that demographic balancing and knowledge enrichment can reduce utility gaps for certain groups while preserving overall performance, though gender balancing may lower utility without improving gender indicators.
By Furkan Yasir Goksu, Yaren Yilmaz, Sule Gunduz Oguducu
arXiv:2512. 07019v3 Announce Type: replace-cross Abstract: The proliferation of Large Language Models (LLMs) necessitates valid evaluation methods to provide guidance for both downstream applications and actionable future improvements.
By Zhiyu Xu, Jia Liu, Yixin Wang, Yuqi Gu
arXiv:2409.00696v4 Announce Type: replace-cross
Abstract: Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs)....
By Jasper Dekoninck, Maximilian Baader, Martin Vechev
The paper argues that language models’ intransitive preferences arise from multiple internally consistent latent orderings rather than noise around a single ordering. By demonstrating that a single ordering cannot explain observed inconsistencies and introducing a noise‑augmented mixture Bradley‑Terry model, the authors show that mixtures of orderings better capture preference structure across several models and tasks. A case study on Moral Machine dilemmas further illustrates that models can share latent components even when aggregate preferences differ.
By Aviral Chawla, William H. W. Thompson, Jean-Gabriel Young
arXiv:2606. 17165v1 Announce Type: cross Abstract: Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost.
By Joel Persson, M{\aa}rten Schultzberg, Sebastian Ankargren
arXiv:2608.25200v2 Announce Type: replace-cross
Abstract: We study learning a mixture of $k$ Plackett-Luce models from multi-way ranking responses from annotators that may represent heterogeneous und...
By Dongyue Li, Ziniu Zhang, Lu Wang, Hongyang R. Zhang