The paper "Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation" examines how arena-style rankings of language models, which rely on pairwise comparisons and the Bradley‑Terry model, can be manipulated. It identifies three attack vectors—strategic voting, strategic prompting, and strategic nomination—that can each boost a model’s rank by 15‑20 places, and combined attacks can improve a rank by up to 45 places. The authors analyze the conditions that enable these vulnerabilities and propose safeguards for any preference‑based evaluation that permits open prompting, voting, or model submission.
By Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell, Dirk Hovy
arXiv:2608. 14629v1 Announce Type: cross Abstract: As Large Language Models (LLMs) become the mainstay for information retrieval and summarization tasks, ensuring that they are always non-partisan and invulnerable to political bias is a critical step towards safer and more trustworthy Artificial Intelligence (AI).
By Tejaswi V. Panchagnula, Bruce Coburn, Bryce J. Dietrich, Robert X. Browning, Edward J. Delp, Fengqing Zhu
arXiv:2601. 21816v2 Announce Type: replace Abstract: Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards.
By Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar, Stefan Feuerriegel
The paper introduces a style-aware paraphrasing method for text anonymization that leverages pretrained large language models to build compact stylistic profiles from minimal samples and rewrite text to suppress identifiable style markers while preserving meaning. It demonstrates that this approach reduces authorship attribution F1 scores by 60‑70% on blog and review datasets, outperforming both differential privacy‑based and non‑DP baselines, and maintains content quality and readability.
By Ahmed Sohair Khan, Estrid He, Monica Wachowicz, Elham Naghizade
The paper introduces DIAL, a framework that uses large language models (LLMs) as judges while mitigating position bias and aligning their judgments with human preferences. DIAL separates judge‑specific position effects, learns shared structure in debiased LLM preferences, and adaptively calibrates this structure toward human targets using limited human comparisons. Experiments on simulations and three human‑preference benchmarks show that DIAL remains robust to unbalanced response order, achieves strong human‑aligned rankings with few labels, and adapts when LLM information is imperfect, supported by a real‑data study of over 410K judgments from 21 LLM judges.
By Zesheng Cai, Yingqi Fan, Sichang Chen, Jin-Hong Du
arXiv:2605. 15416v2 Announce Type: replace-cross Abstract: Jung et al.
By Gaojie Jin, Yong Tao, Lijia Yu, Tianjin Huang
arXiv:2609.22112v1 Announce Type: new
Abstract: Large language models (LLMs) have demonstrated the ability to generate user-specific text with high stylistic fidelity. However, the personal data that...
By Muhammed Nazmul Arefin, Omar Jamal Hammad
arXiv:2508. 11847v4 Announce Type: replace-cross Abstract: We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data.
By Jenny Y. Huang, Yunyi Shen, Dennis Wei, Tamara Broderick
arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.
By Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou
arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.
By Bora Kargi, David Salinas
arXiv:2606. 00334v1 Announce Type: cross Abstract: Various language domains have undergone remarkable changes in recent years; these shifts are largely attributed to the advent of Large Language Models and their misalignment with natural language usage.
By Xiaoyang Ming, Jose Hernandez, Thomas Stephan Juzek
arXiv:2606. 04928v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance.
By Fr\'ed\'eric Berdoz, Luca A. Lanzend\"orfer, Kaan Bayraktar, Roger Wattenhofer