The paper introduces a reason-guided preference-based reinforcement learning framework that learns a human reward model from trajectory-level feedback. It augments traditional preference labels with reason tags that explain why a behavior is preferred, thereby capturing both human choices and their underlying rationale. By combining the learned human reward with the environment reward during policy optimization, the agent can improve task performance while producing more interpretable, human-aligned, and preference-sensitive behavior.
arXiv:2606. 11982v1 Announce Type: new Abstract: Preference-based reinforcement learning (PbRL) learns policies from human trajectory-level comparisons, avoiding explicit reward design and expert demonstrations.
arXiv:2606. 01123v1 Announce Type: new Abstract: Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback.
arXiv:2506. 13741v2 Announce Type: replace-cross Abstract: Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions.
arXiv:2506. 13702v4 Announce Type: replace-cross Abstract: Single-trajectory preference optimization methods learn from datasets of ((prompt, response, reward)) tuples, offering a practical alternative to pairwise preference learning by directly leveraging scalar feedback.
arXiv:2509. 22851v4 Announce Type: replace-cross Abstract: Margin-based optimization is fundamental to improving generalization and robustness in classification tasks.
arXiv:2607. 13172v1 Announce Type: new Abstract: We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available.
arXiv:2506. 12529v2 Announce Type: replace-cross Abstract: Preference-based Reinforcement Learning (PbRL) entails a variety of approaches for aligning models with human intent to alleviate the burden of reward engineering.
The paper introduces PrefMind, an online dual‑agent reinforcement learning framework that jointly trains a response Generator and a preference Critic to explicitly learn and apply human preference judgments. By having the agents discuss and correct each other’s evaluations of candidate responses, PrefMind embeds preference understanding directly into the generation process rather than relying solely on a scalar reward model. Experiments on models of varying sizes demonstrate gains in both preference judgment accuracy and downstream response quality.
Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process.
arXiv:2607. 04854v1 Announce Type: new Abstract: Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications.
The paper proposes a reinforcement learning framework that emphasizes designing feedback environments rather than solely relying on rewards. It integrates human domain knowledge via rubrics and model reasoning with hierarchical rewards to evaluate complex, partially successful outputs in open‑ended tasks. Applied to interactive web application generation, the model achieves 41.25% pass rate on MiniAppBench and 76.19% on ArtifactsBench, with ablation studies confirming the value of each component.
The paper critiques the standard fixed-rule approach for deriving labels from human feedback in human-robot collaboration, showing that human-provided implication labels often differ and improve reward learning. It introduces IMPLIED, a method that starts with fixed-rule implications but learns to infer and revise accepted/rejected action labels over time, outperforming both the fixed rule and LLM baselines on recorded trajectories and a physical pizza‑making study. As a result, IMPLIED reduces preference‑estimation error and yields robot actions that better align with combined reward objectives.