Transduced language models (TLMs) combine a pretrained source language model with a finite‑state transducer to produce a language model over target strings. The paper introduces an unbiased stochastic estimator that resamples source prefixes without replacement and reweights them, allowing accurate estimation of target prefix probabilities while reducing computation compared to threshold‑pruned beam summing. Experiments on encyclopedic text, DNA, and DNA‑to‑amino‑acid transduction show improved compute–variance trade‑offs and significant runtime reductions, and the method also lowers estimated corpus surprisal in a reading‑time analysis without altering its conclusions.
By V\'esteinn Sn{\ae}bjarnarson, Samuel Kiegeland, Manuel de Prada Corral, Ryan Cotterell, Tim Vieira
arXiv:2602.00612v3 Announce Type: replace
Abstract: Diffusion Large Language Models (dLLMs) have demonstrated promising generative capabilities and are increasingly used to produce formal languages d...
By Yitong Zhang, Yongmin Li, Yuetong Liu, Jia Li, Xiaoran Jia, Zherui Li, Ge Li
arXiv:2603. 03305v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable.
By Avinash Reddy, Thayne T. Walker, James S. Ide, Amrit Singh Bedi
arXiv:2606. 00722v1 Announce Type: cross Abstract: Controlling language model outputs is essential for ensuring structural validity, reliability, and downstream usability, and diffusion language models are no exception.
By Hyundong Jin, Yo-Sub Han
The paper introduces Grammar‑Aligned Decoding (GAD), addressing the issue that conventional grammar‑constrained decoding (GCD) can distort a large language model’s probability distribution, yielding grammatical but low‑likelihood outputs. GAD proposes an adaptive sampling method, Approximate Expected Futures (ASAp), which uses prior samples to over‑approximate future grammaticality, ensuring outputs remain both grammatical and faithful to the model’s conditional probabilities. Experiments on code generation and structured NLP tasks demonstrate that ASAp often produces higher‑likelihood outputs than existing GCD techniques while still enforcing the required grammatical constraints.
By Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, Loris D'Antoni
arXiv:2605. 18476v2 Announce Type: replace-cross Abstract: Coding and computation remain major bottlenecks in Markov chain Monte Carlo (MCMC) workflows, especially as modern sampling algorithms have become increasingly complex and existing probabilistic programming systems remain limited in model support, extensibility, and composability.
By Jungang Zou, Alex Ziyu Jiang, Qixuan Chen
arXiv:2601. 12247v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) present a promising non-sequential paradigm for text generation, distinct from standard autoregressive (AR) approaches.
By Miao Li, Hanyang Jiang, Sikai Cheng, Hengyu Fu, Yuhang Cai, Baihe Huang, Tinghan Ye, Xuanzhou Chen, Pascal Van Hentenryck
arXiv:2607. 07026v1 Announce Type: new Abstract: Constrained decoding is essential for serving LLMs, ensuring that generated outputs follow specific structures such as JSON schema-formatted function calls.
By Meihua Dang, Stefano Ermon
Selective Regenerative Decoding (SRD) is a new inference-time decoding method that improves large language model reasoning by allowing segment-level intervention on candidate trajectories. Instead of discarding or keeping entire trajectories, SRD selectively refines only the degraded suffix while preserving useful prefixes, leading to higher expected trajectory quality and better sample efficiency. Experiments on MATH500, GPQA Diamond, HotpotQA, and AlpacaEval show that SRD matches Best-of-N accuracy with fewer generated tokens and outperforms speculative rejection in low‑compute settings.
By Sophia Xiao Pu, Yumo Xu, Sailik Sengupta, Millennium Bismay, Ruixue Lian, James Gung, Yi-an Lai, Arshit Gupta
arXiv:2606. 19264v1 Announce Type: new Abstract: The knowledge encoded in large language models (LLMs) can serve as a substrate for structured reasoning over variables describing a complex world, but accessing this knowledge in a probabilistically coherent manner poses a difficult inference problem.
By Sanghyeok Choi, Henry Gouk, Esmeralda S. Whitammer
Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints.
arXiv:2606. 14943v1 Announce Type: cross Abstract: Causal Transformers model sequences through an autoregressive factorization of the joint distribution, which enables efficient left-to-right decoding and conditional likelihood computation.
By Yinhan Lu, Eric Elmoznino, L\'eo Gagnon, Sarthak Mittal, Tejas Kasetty, Guillaume Lajoie