XTC (Exclude Top Choices) is a lightweight, head‑aware decoding operator that improves diversity in autoregressive language models by removing overly probable tokens that dominate the next‑token distribution. It works by identifying tokens above a plausibility threshold, probabilistically excluding the dominant choices, and renormalizing the remaining distribution. Across 60 experiments on models such as Gemma 3 and DeepSeek R1, XTC boosts Distinct‑2 scores by 11–15 % and cuts repeat trigrams by 27–47 %, while a Mechanical Turk study shows a 62.3 % preference for XTC‑generated text without loss of fluency.
By Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder, Sanjay Basu, Ravid Shwartz-Ziv
arXiv:2609.07901v1 Announce Type: new
Abstract: Weight quantization largely determines the economics of serving open-weight LLMs. Its costs are usually assessed with capability benchmarks, on which 4...
By Dachi Kurtskhalia
The paper introduces Variance‑Calibrated Modulation (VCM), a training‑free pre‑decoding technique that reshapes language model probability distributions before truncation. VCM uses two dynamic mechanisms: a Contextual Searchlight via PMI to suppress stopwords and highlight context‑relevant tokens, and an Adaptive Self‑Debiasing that applies scale‑invariant penalization based on real‑time logit standard deviation. Experiments on open‑ended generation, factual QA, and mathematical reasoning show that VCM consistently reduces the likelihood trap, improving diversity, coherence, and reasoning accuracy with minimal computational cost.
By Yuanhao Ding, Meimingwei Li, Esteban Garces Arias, Matthias A{\ss}enmacher, Christian Heumann, Chongsheng Zhang
arXiv:2410.02343v2 Announce Type: replace
Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...
By Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
arXiv:2606. 01561v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO).
By Xiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu, Zhipeng Wang, Huayu Li, ZhengXiao He, Xuanzhao Dong, Prayag Tiwari, Mingkun Xu, Yujian Xiong, Feng Luo, Abolfazl Razi, Brendan Hogan Rappazzo, Anderson Schneider, Yuriy Nevmyvaka
The paper introduces Neighborhood Elites (NE), a quality‑diversity search method that constructs input rewrites for large language models in question answering by controlling both the type of edit and the explicitness of support. NE outperforms three other neighborhood constructors across five extractive‑QA datasets and five instruction‑tuned LLMs, achieving higher AUROC and AUPRC scores and surpassing semantic entropy from decoder samples. The method demonstrates that carefully designed perturbations improve failure detection in QA systems.
arXiv:2610.19996v1 Announce Type: new
Abstract: Large language models often produce fluent but factually incorrect output. Gray-box detectors identify such hallucinations from the token-probability s...
By Shaked Goldshtein, Yitzchak Grunbaum, Fabrizio Frasca, Guy Bar-Shalom, Haggai Maron
The paper investigates how preference optimization can inadvertently create a shared directional shift—termed a common‑mode offset—in a model’s hidden states. This offset, which affects both chosen and rejected responses, accumulates without being penalized by the ranking loss, eventually projecting onto the output layer to form a vocabulary well that dominates decoding and leads to generation collapse. Experiments across multiple base‑model lineages and fine‑tuning regimes confirm that masking this offset or using per‑token ranking losses can prevent the collapse, while standard ranking metrics fail to detect the risk.
By Yun Li, Ehsan Javanmardi, Simon Thompson, Jiachen Gong, Qunli Zhang, Zifan Zeng, Shiming Liu, Peng Wang, Zixuan Guo, Manabu Tsukada
Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked.
The paper introduces Neighborhood Elites (NE), a quality‑diversity search method for generating answer‑preserving rewrites in question answering. NE explicitly controls two dimensions—type of edit and explicitness of support—to create diverse perturbations that improve uncertainty quantification. Across five extractive‑QA datasets and five instruction‑tuned LLMs, NE outperforms other constructors, achieving higher AUROC and AUPRC scores and surpassing semantic entropy baselines.
By Mesut Ceylan, Arda Baris Basaran, Elgar Fleisch, Filipe Barata
arXiv:2609.16454v1 Announce Type: new
Abstract: Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend...
By Kirill Skobelev, Eric Fithian, X. Y. Han
arXiv:2510. 01171v4 Announce Type: replace-cross Abstract: Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse.
By Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael R. Tomz, Christopher D. Manning, Weiyan Shi