CM-DPO introduces a Constraint-Margin Direct Preference Optimization method that replaces the binary preference signal of traditional DPO with a continuous margin based on a deterministic symbolic verifier, scaled by violation severity. Hard and soft constraints are separated via a lexicographic objective to prevent hard constraints from being traded off against preferences. Using procedurally generated constraint profiles and minimal-edit distillation within the SynPlan-R framework, an 8B model fine‑tuned with CM‑DPO achieves high pass and solve rates on several planning benchmarks, matching multi‑agent systems with lower latency and outperforming GPT‑4o on unseen tasks.
arXiv:2606. 04816v1 Announce Type: new Abstract: Large language models (LLMs) increasingly translate natural-language optimization problems into executable solver code.
arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.
arXiv:2607. 04854v1 Announce Type: new Abstract: Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications.
arXiv:2608. 16068v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as agents that rely on system prompts to use tools and complete tasks.
arXiv:2607. 15562v1 Announce Type: cross Abstract: Packing for air travel is recurring and error-prone: the checklist must be personal and context-aware, yet feasible under safety rules, item dependencies, and luggage limits.
Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process.
Continuous-Utility Direct Preference Optimization (CU‑DPO) replaces binary preference labels with continuous scores to better capture fine‑grained reasoning quality. The framework aligns language models to multiple prompt‑based cognitive strategies, proving a θ(K log K) sample‑complexity improvement and convergence to an entropy‑regularized utility‑maximizing policy. A two‑stage pipeline—strategy selection via best‑vs‑all comparisons and execution refinement using margin‑stratified pairs—yields significant gains on mathematical reasoning, code generation, and causal reasoning benchmarks.
The paper introduces Graph Direct Preference Optimization (GraphDPO), a generalization of Direct Preference Optimization that uses directed acyclic graphs to capture richer preference structures from multiple rollouts per prompt. GraphDPO encodes dominance relations as edges and optimizes a Plackett–Luce-inspired objective that aggregates supervision over dominated neighborhoods, preserving transitivity and handling ties or incomparable responses via an equivalence-class construction. Experiments on reasoning and program synthesis tasks show that this graph-structured approach outperforms traditional pairwise and listwise alignment objectives while maintaining linear per‑prompt complexity.
The paper introduces In-Context Preference-Conditioned Pareto Set Learning (ICPSL), a framework that uses a pretrained hypernetwork to generate a problem-specific, preference-conditioned Pareto solution model from limited offline data. ICPSL eliminates the need for repeated preference-wise searches or problem-specific training by producing a reusable model in a single forward pass. Experiments on synthetic and real-world problems demonstrate that ICPSL yields high-quality Pareto solutions while significantly speeding up model construction.
arXiv:2604. 20140v2 Announce Type: replace Abstract: Direct Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks.
arXiv:2608. 08127v1 Announce Type: new Abstract: The runtime of Constraint Programming (CP) solvers is highly sensitive to modeling choices, such as symmetry breaking, implied constraints, global constraints, constraint reformulation, and variable representation.
arXiv:2607. 10601v1 Announce Type: new Abstract: Large Language Model (LLM) agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation.