The paper investigates whether constrained decoding harms or helps large language model (LLM) performance. It introduces a taxonomy of three constraint types—format, validity, and reasoning—and shows that suitable constraints can preserve or even improve performance by eliminating invalid paths. Experiments across multiple model families reveal that perceived performance drops are mainly due to non-equivalent prompting and output formats, while validity constraints boost reliability and reasoning constraints can enhance accuracy when they guide beneficial reasoning trajectories.
arXiv:2608. 12426v1 Announce Type: new Abstract: Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints - reasoning structure, safety boundaries, output schemas.
arXiv:2603. 03305v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable.
arXiv:2607. 20520v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated on mathematical problem solving, yet prior work often treats representationally equivalent formulations as interchangeable and conflates reasoning errors with interface failures.
arXiv:2607. 16727v1 Announce Type: new Abstract: Autoregressive multimodal large language models (MLLMs) suffer from error snowballing: a single incorrect inference early in a chainof-thought (CoT) trace corrupts all downstream reasoning.
The paper introduces a constraint‑aware training objective that externalizes program analysis constraints used during inference, avoiding redundant learning of these constraints during model training. It presents three theorems demonstrating that this approach can reduce model size and improve data efficiency. A controlled synthetic experiment confirms that constraint‑aware training achieves lower prediction loss than standard cross‑entropy training at the same parameter count and data volume.
arXiv:2602.00612v3 Announce Type: replace Abstract: Diffusion Large Language Models (dLLMs) have demonstrated promising generative capabilities and are increasingly used to produce formal languages d...
RECAST is a new framework that generates datasets with far more constraints per example than existing benchmarks, aiming to push large language models (LLMs) to better follow complex instructions. The authors built RECAST-30K, a 30,000‑instance dataset covering 19 constraint types extracted from real prompt‑response pairs, and showed that fine‑tuning on it improves LLMs’ ability to handle complex tasks without harming general performance. RECAST also provides rule‑based and LLM‑based validators for automatic constraint verification, enabling reward‑based reinforcement learning to further enhance model performance on challenging tasks.
arXiv:2602. 15983v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk: code that executes and returns solver-feasible solutions may encode semantically incorrect formulations---a feasibility--correctness gap reaching 90 percentage points on compositional problems.
The paper introduces ramework, a blackbox prompt‑minimization framework that identifies the minimal subset of few‑shot prompts necessary for large language models (LLMs). In a case study, the framework reduces few‑shot exemplars by an average of 65.3% in character count while maintaining full propositional output fidelity, revealing that models tend to keep logical identifiers and constraint declarations while discarding natural language prose. The analysis further distinguishes between universal encoder and decoder models, offering insights into prompt compression and structural analysis.
arXiv:2509.14093v3 Announce Type: replace-cross Abstract: Chain-of-Thought (CoT) prompting can substantially improve the reasoning ability of large language models (LLMs), but it often comes with hig...
The paper introduces MeMoDecode, a decoding framework that enforces both concrete and abstract syntax as well as well‑formedness constraints for domain‑specific languages (DSLs) during large language model generation. By mapping generated text to partial metamodel instances and retrying from the last consistent snapshot, MeMoDecode guarantees consistency without sacrificing alignment metrics. Experiments on four DSLs and six open‑weight LLMs show a 23% average improvement over unconstrained decoding and an 8% improvement over grammar‑constrained decoding, with modest runtime overhead.
arXiv:2603. 05290v2 Announce Type: replace Abstract: Large language models (LLMs) achieve promising performance, yet their ability to reason remains poorly understood.