The paper investigates how large language models (LLMs) express subjective preferences in short-form content such as numbers, baby names, and fictional cities. It identifies 29 aspects (e.g., length, alphabetic order, popularity) that influence model choices and shows that these preferences are strong, reproducible, and sensitive to how questions are phrased. The study also reveals that the interaction between generating alternatives and ranking them can produce unexpected preference patterns, raising concerns about potential manipulation.
By Katy Blumer, Jon Kleinberg, Ravi Kumar, Andrew Tomkins
arXiv:2606. 02883v1 Announce Type: cross Abstract: Recommender systems have grown from content-organization tools into sophisticated systems that shape daily behavior.
By Amir Ghasemian, Homa Hosseinmardi, Upasana Dutta, Duncan J. Watts
Typed decision models provide structured choices, scores, and probabilities for low‑cost automated evaluation. In a controlled audit across Jev, Clef Flash, and Perplexity Decider, the study examined how presentation factors—such as item position, identifiers, authorship claims, endorsements, decision framing, and rating rubrics—affect judgments on humor, helpfulness, and summarization tasks. Results show significant shifts in probability advantages, selected humor texts, and agreement with human preferences, indicating that presentation robustness, repeatability, and human alignment are distinct evaluator properties that must be validated.
By Jatin Agrawal, Radhika Mamidi
arXiv:2608.21430v1 Announce Type: new
Abstract: Multimodal language models increasingly show promise for enabling the large-scale computational analysis of film, opening up new avenues for learning a...
By David Bamman, Kent K. Chang, Allison Cooper, Juishan Hsu, Reina Kushihashi, Madison Mar, Arnav Podichetty, Rachael Samberg, Ipek Nil Sancak, Yuhan Shao
arXiv:2610.35254v1 Announce Type: new
Abstract: The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload o...
By Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz
The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload of researchers. At the same time, large language m...
arXiv:2609.36218v1 Announce Type: cross
Abstract: Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film rema...
By Mir Tafseer Nayeem, Susmoy Chakraborty, Davood Rafiei
arXiv:2608. 03659v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting.
By Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez
arXiv:2610.35254v2 Announce Type: replace
Abstract: The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review worklo...
By Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz
arXiv:2609.23264v1 Announce Type: new
Abstract: Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high sc...
By Shakiba Amirshahi, Sajad Ebrahimi, Hai Son Le, Negar Arabzadeh, Ebrahim Bagheri
As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions.
arXiv:2606. 16240v1 Announce Type: cross Abstract: Activation steering has emerged as a powerful tool for shaping the behaviour of large language models at inference time, yet most prior work injects a \emph{single} semantic direction into the residual stream.
By Subramanyam Sahoo, Justin Shenk