Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2512.24407v2 Announce Type: replace Abstract: In many sequential decision-making problems, researchers observe actions but not the rewards that drive behavior, yet still wish to evaluate and co...
BIPPO (Budget-aware Independent Proximal Policy Optimization) is a multi‑agent reinforcement learning framework designed for energy‑efficient client selection in federated learning (FL) over IoT systems. It addresses infrastructure constraints such as limited resources and device churn, which traditional FL and RL approaches overlook. Evaluated on two image‑classification tasks with non‑IID data, BIPPO improves mean accuracy over non‑RL methods, standard PPO, and IPPO while consuming only a negligible portion of the budget, even as client numbers grow.
AgentMercury is a scalable framework that synthesizes executable environments from high‑level business scenarios instead of task‑specific benchmarks. It creates a persistent world with entities, services, tools, and invariants, allowing diverse tasks and interaction trajectories to emerge naturally. The authors generated 4,783 environments across 14 industries and 50 countries, and training reinforcement‑learning agents on them improved performance on enterprise workflows and out‑of‑domain benchmarks, while the construction process itself can be learned to increase authoring success.
The paper introduces OraRL, a reinforcement learning framework that leverages annotations as oracle rollouts to improve sample efficiency and scalability for video multimodal large language models (MLLMs). By decoupling advantage estimation and employing sign‑balanced pruning, OraRL achieves faster training and better performance across multiple video‑perception benchmarks compared to existing methods. The approach scales from 0.8B to 9B parameters and handles up to 100k prompts, delivering significant gains in temporal mIoU, tracking accuracy, segmentation, and spatial‑intelligence metrics.
The study presents a reinforcement learning approach, Advantage Aggregation (AdvA), to control the X‑point target (XPT) divertor in the EXL‑50U tokamak experiment. By preserving objective‑wise temporal credit and applying a residual correction, AdvA‑PPO outperforms standard Reward‑PPO and a feedforward‑plus‑PID baseline, achieving a 0.81 worst‑channel score and reducing X‑point flux RMSE by roughly 20× over a 500 ms rollout. The method remains robust under measurement uncertainties and across varied initial equilibria, offering a simulation‑based foundation for future real‑time XPT validation on EXL‑50U.
The paper proposes a two-stage personalized thermal comfort system that combines multimodal physiological and environmental sensing with reinforcement learning for decision-making. It aims to move beyond static HVAC setpoints and generic comfort models by tailoring thermal interventions to individual physiological variability. The approach seeks to improve occupant wellbeing and enable more responsive building-control strategies.
The paper introduces reinforcement learning for Continuous-Time Jump Markov Decision Processes (CTJMDPs) with general discrete state spaces and continuous/discrete actions. It develops entropy‑regularized continuous‑time control and establishes theoretical foundations for q‑learning in this setting, providing model‑free algorithms that outperform naive discretization. Numerical tests on network dynamic pricing demonstrate the method’s ability to learn near‑optimal policies and scale to large networks.
The paper investigates how agentic systems decide between acting and abstaining, focusing on the fidelity of their reasoning explanations. Using Qwen3‑8B in a multi‑party conversation setting, the authors compare direct decision policies, reasoning policies, supervised fine‑tuning, and reinforcement learning, finding a trade‑off: strong direct policies yield higher performance but no traceable reasoning, while reasoning policies provide an audit trail at the cost of lower recall. The study also uncovers that exposing reasoning can alter the agent’s policy and that common faithfulness metrics may overstate the alignment between reasoning and decisions.
arXiv:2608.20814v1 Announce Type: new Abstract: Though Multimodal Large Language Models (MLLMs) have shown impressive potential in video understanding, long video understanding (LVU) remains challeng...
The paper introduces VA‑DPO, a method that trains language models to generate text with a specified continuous affect point in the Valence‑Arousal plane. By using a frozen VA regressor to score candidate generations and selecting pairs with a distance margin, the approach modifies Direct Preference Optimization to better hit target emotions. Experiments on Llama‑3.1‑8B‑Instruct show a 33% reduction in mean VA distance compared to system‑prompting and 25% over few‑shot prompting, while maintaining performance on benchmarks like MMLU, HellaSwag, and TruthfulQA.
arXiv:2608. 20686v1 Announce Type: new Abstract: Many scientific discovery problems require searching combinatorial hypothesis spaces under complex domain constraints.
The paper introduces IR‑LQR, an optimistic online linear quadratic regulator that incorporates intrinsic rewards and variance regularization to encourage exploration while maintaining the standard LQR structure. By only adjusting the cost function, IR‑LQR remains computationally simple yet achieves the optimal worst‑case regret rate of √T. The authors validate the method with numerical experiments on aircraft pitch angle control and a UAV example, comparing it to state‑of‑the‑art online LQR algorithms.
The paper introduces the concept of behavior-consistent deep reinforcement learning, aiming to produce high-performing policies that remain distributionally similar across different training runs. It shows that maximum-entropy RL can control behavioral divergence by anchoring runs to a common prior, and proves that for Boltzmann policies, a temperature proportional to Q‑function disagreement limits pairwise KL divergence. Building on this, the authors propose Q‑value Expectile Disagreement (QED), a state‑dependent temperature schedule that uses double‑critic disagreement to approximate cross‑run disagreement, and demonstrate that QED reduces across‑run divergence by two orders of magnitude on 18 continuous‑control tasks without sacrificing performance.
arXiv:2106.06060v4 Announce Type: replace-cross Abstract: Traditional competitive markets do not account for negative externalities; indirect costs that some participants impose on others, such as th...
The paper explores how dense, turn-level reward structures can improve reinforcement learning for large language model agents in multi-turn tasks. It introduces three reward granularity types—terminal, delayed, and per-turn—and adapts Group Relative Policy Optimization and Proximal Policy Optimization to each. Experiments on search and game agents show that per-turn rewards consistently yield better training dynamics, faster convergence, and higher answer correctness compared to sparse terminal or delayed rewards.
The paper introduces Conformalized Agentic Search (CAS), a framework that applies Conformal Prediction to improve the reliability of search agents during reinforcement learning fine-tuning. CAS uses an Adaptive Prediction Set (APS) to dynamically truncate retrieved documents based on statistical coverage, and Adaptive Conformal Inference (ACI) to construct confidence-aware prediction sets that penalize low‑confidence trajectories in the Group Relative Policy Optimization objective. Experiments on single‑hop and multi‑hop QA datasets show that CAS enhances reasoning accuracy and reduces redundant tool invocations, offering a more reliable and efficient agent paradigm.
arXiv:2603.20004v4 Announce Type: replace-cross Abstract: Translating natural language questions to SQL queries (Text-to-SQL) is a long-standing problem in database research. Recent efforts have focu...
arXiv:2608.21175v1 Announce Type: cross Abstract: Safe and efficient shape-aware navigation in heterogeneous crowds and robot fleets remains challenging. Traditional approaches often assume homogeneo...
arXiv:2601.04786v3 Announce Type: replace-cross Abstract: Recent advances in large language models (LLMs) enable agentic systems trained with reinforcement learning (RL) over multi-turn interaction,...
The paper introduces Preference‑Based Self‑Distillation (PBSD), a new on‑policy self‑distillation method that replaces traditional KL matching with a reward‑regularized objective. PBSD derives a reward‑reweighted teacher distribution, optimizing preference gaps between teacher and student samples while keeping on‑policy sampling. Experiments on mathematical reasoning and tool‑use tasks show PBSD achieves stronger average performance, improved training stability, and maintains token efficiency compared to prior self‑distillation baselines.