arXiv AI By Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder, Sanjay Basu, Ravid Shwartz-Ziv

Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time

Read the original on arXiv AI →

The paper introduces DRY, a sampling-time logit adjustment that penalizes token generation only when it would extend the current suffix into an exact repetition of an earlier span, thereby preventing verbatim loops in large language model outputs. Experiments across models ranging from 1.5B to 120B parameters and various prompt families show that DRY cuts suffix-extension rates by 47% and improves lexical diversity, while preserving benchmark performance. The method has been adopted by popular open-source LLM inference frameworks, indicating its practical relevance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 28

FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models

The paper investigates how pruning large language models can lead to text degeneration, particularly repetition loops, even when perplexity and task accuracy stay stable. By treating decoding as a dynamical process, the authors separate degeneration into loop entry risk and loop persistence, showing that persistence depends on the escape mass given to plausible alternatives. They introduce two token‑level guidance objectives—FOCUS, which reweights distillation toward high‑confidence teacher regions, and RePAIR, which uses onset‑centered positive/negative continuation pairs with a margin loss—to reduce repetition and improve generation quality in pruned models.

By Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il Hong Suh, Yeseong Kim
arXiv Computation and Language
4d ago

Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding

The paper introduces Variance‑Calibrated Modulation (VCM), a training‑free pre‑decoding technique that reshapes language model probability distributions before truncation. VCM uses two dynamic mechanisms: a Contextual Searchlight via PMI to suppress stopwords and highlight context‑relevant tokens, and an Adaptive Self‑Debiasing that applies scale‑invariant penalization based on real‑time logit standard deviation. Experiments on open‑ended generation, factual QA, and mathematical reasoning show that VCM consistently reduces the likelihood trap, improving diversity, coherence, and reasoning accuracy with minimal computational cost.

By Yuanhao Ding, Meimingwei Li, Esteban Garces Arias, Matthias A{\ss}enmacher, Christian Heumann, Chongsheng Zhang
arXiv AI
5d ago

Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models

The paper introduces PILL, a new infilling technique for diffusion language models that eliminates the need for a preset initial length and reduces inference overhead. PILL uses probing-based length-free decoding, cutting down on extra forward passes and speeding up generation. Experiments across five diffusion models and eight benchmarks show PILL outperforms the strongest baseline with higher pass rates and BLEU-2 scores while running 1.82× faster.

By Haobo Xu, Sirui Chen, Yuanchen Bei, Lingjie Chen, Yuchen Yan, Dongqi Fu, Jingrui He, Hanghang Tong