The paper investigates how pruning large language models can lead to text degeneration, particularly repetition loops, even when perplexity and task accuracy stay stable. By treating decoding as a dynamical process, the authors separate degeneration into loop entry risk and loop persistence, showing that persistence depends on the escape mass given to plausible alternatives. They introduce two token‑level guidance objectives—FOCUS, which reweights distillation toward high‑confidence teacher regions, and RePAIR, which uses onset‑centered positive/negative continuation pairs with a margin loss—to reduce repetition and improve generation quality in pruned models.
By Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il Hong Suh, Yeseong Kim
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
By Tianyi Li, Mingda Chen, Bowei Guo, Zhiqiang Shen
The paper introduces Variance‑Calibrated Modulation (VCM), a training‑free pre‑decoding technique that reshapes language model probability distributions before truncation. VCM uses two dynamic mechanisms: a Contextual Searchlight via PMI to suppress stopwords and highlight context‑relevant tokens, and an Adaptive Self‑Debiasing that applies scale‑invariant penalization based on real‑time logit standard deviation. Experiments on open‑ended generation, factual QA, and mathematical reasoning show that VCM consistently reduces the likelihood trap, improving diversity, coherence, and reasoning accuracy with minimal computational cost.
By Yuanhao Ding, Meimingwei Li, Esteban Garces Arias, Matthias A{\ss}enmacher, Christian Heumann, Chongsheng Zhang
The paper introduces PILL, a new infilling technique for diffusion language models that eliminates the need for a preset initial length and reduces inference overhead. PILL uses probing-based length-free decoding, cutting down on extra forward passes and speeding up generation. Experiments across five diffusion models and eight benchmarks show PILL outperforms the strongest baseline with higher pass rates and BLEU-2 scores while running 1.82× faster.
By Haobo Xu, Sirui Chen, Yuanchen Bei, Lingjie Chen, Yuchen Yan, Dongqi Fu, Jingrui He, Hanghang Tong
Large language models can reveal memorized training sequences when prompted with a prefix, but real-world systems often use additional context such as instructions or retrieved documents. This study measures how such contextual conditioning affects memorization by comparing suffix probabilities with empty prompts versus retrieved contexts across three instruction-tuned models. The results show that context does not simply erase memorization; instead, it preserves a core set of robustly extractable samples while also shifting the boundary of marginal cases, sometimes suppressing and sometimes enabling new exposures.
By Ali Satvaty, Narjes Sharafi, Jirui Qi, Suzan Verberne, Fatih Turkmen
arXiv:2609.13154v1 Announce Type: new
Abstract: Recent advances in large language models (LLMs) have made prompts increasingly large and complex. Techniques such as chain-of-thought reasoning (Wei et...
By Shamin Chokshi