Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2610.28386v1 Announce Type: cross Abstract: Visual tool use enables vision-language models (VLMs) to acquire task-relevant evidence, but its effectiveness can be limited by predefined visual op...
arXiv:2610.28556v1 Announce Type: cross Abstract: Large language models (LLMs) can leverage semantic feature names alongside summary statistics for tabular feature selection, yet it remains unclear w...
arXiv:2610.28586v1 Announce Type: cross Abstract: Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of wher...
arXiv:2410.03246v3 Announce Type: replace-cross Abstract: Deep Reinforcement Learning (DRL) enables robots to learn complex behaviors through interaction with the environment. However, due to the unr...
The paper surveys multimodal generative models (MGMs), outlining six main text‑conditioned modalities and four key techniques—self‑supervised learning, mixture of experts, reinforcement learning from human feedback, and chain‑of‑thought prompting—used across them. It evaluates models on faithfulness, compositionality, and robustness, and reports four empirical findings about preference gains, reward optimisation, intermediate planning, and expert granularity. The study also discusses trustworthiness risks such as bias, privacy leakage, and media misuse, and reviews mitigation evidence.
arXiv:2609.35767v2 Announce Type: replace-cross Abstract: Unified multimodal models can both look at and render images, so in principle they can repair their own generations: diagnose what an image g...
arXiv:2610.25916v2 Announce Type: replace-cross Abstract: As coding models and agent harnesses become more capable, reliable security verification is becoming a central bottleneck. Improving secure c...
LLM agents are commonly trained with RLVR, but the sparse nature of typical RLVR rewards makes credit assignment challenging, especially for long-horizon tasks. We propose Self-Credit, an end-to-end a...
Asked to name a running shoe brand, language models say Nike. Asked to recommend one, they say Brooks. This paper asks where models learn that taste. Across 102 models and 42 brand categories, the bra...
The paper proposes a centralized training with decentralized execution framework for multi‑agent UAV pathfinding in stochastic environments, leveraging multi‑agent reinforcement learning enhanced by model predictive control concepts. Agents communicate only with a centralized planner and use a distance‑weighted mean‑field observation to plan feasible paths, while multi‑step value convergence improves training efficiency. Experiments, including ablation and real‑robot studies, demonstrate the method’s effectiveness and generalization.
The paper investigates emergent misalignment in reward models, showing that fine‑tuning reward models to prefer harmful content in a single domain (medical or financial) causes the preference to generalize to seven other domains. The authors fine‑tune two state‑of‑the‑art reward models—Skywork‑Reward‑V2‑Llama‑3.1‑8B and Skywork‑Reward‑V2‑Qwen3‑8B—and find that the models still perform well on the standard RewardBench 2 evaluation while exhibiting a preference for harm. Mechanistically, the study identifies two key changes in the reward models’ representations: a geometric compression that reduces the angle between sound and harmful responses, and a content‑specific rotation that aligns harmful representations more closely with the scalar head, thereby increasing their reward score.
The paper introduces RecReg, a safe reinforcement learning framework that enforces non‑Markovian regular safety properties through recovery‑based shielding. It provides theoretical foundations and rigorous safety and convergence guarantees in the tabular setting, and demonstrates empirical effectiveness using the MASA‑Safe‑RL library. The authors also outline an extension of RecReg to deep RL with state‑of‑the‑art model‑based methods.
The paper introduces Self-Contrastive Guidance (SCG), a method that contrasts a model’s predictions on real versus self-generated samples to directly guide generation toward the true data manifold. SCG is incorporated into the likelihood objective through reparameterization, resulting in the Self-Contrastive Alignment for Likelihood Estimation (SCALE) framework, which produces SCG-enhanced predictions in a single forward pass. Experiments on class-conditional and text-to-image generation with diffusion and autoregressive models show that SCALE consistently outperforms existing guidance-free baselines and further benefits from combining with Direct Preference Optimization (DPO).
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning proposes TreeOPSD, a critic‑free framework that treats each interaction turn as a tree node, branches alternative continuations, and recursively backs up terminal rewards to generate implicit turn‑level credit. The method maps this credit to a bounded teacher distribution based on the likelihood ratio to a frozen behavior policy, and minimizes the divergence between teacher and student to recover policy‑gradient directions while limiting updates across mini‑batches. Empirical results show TreeOPSD enhances agentic performance on long‑horizon benchmarks, notably improving Qwen3‑30B‑A3B‑Thinking‑2507’s accuracy by 4.97 points over GRPO in the Deep Research setting.
The paper introduces MoEvolve, a bidirectional distillation framework that leverages increased activation budgets in Mixture-of-Experts (MoE) models to create a stronger internal teacher. By first restoring a high‑k model and then transferring its enhanced capabilities back to the original low‑k configuration, MoEvolve improves performance across various MoE models and benchmarks, achieving up to 22.2 domain‑average points and outperforming same‑budget RLVR baselines by up to 6.5 points. The resulting low‑k models often surpass their activation‑expanded teachers.
The paper introduces LM2LM, a method that trains large language models (LLMs) to propose improvements for other LLMs using reinforcement learning (RL). It employs a frozen research world model (RWM) to predict the gains of proposals, providing a cheaper surrogate reward, and periodically validates high‑scoring proposals to keep feedback reliable. Experiments across eight LM development environments show that LM2LM enables a Qwen3.8‑27B model to outperform GPT‑6‑sol on held‑out tasks while reducing data‑generation cost by 33%, and it generalizes to unseen model families.
R$^2$ Flow is a recursive self‑improvement framework for large‑language‑model agents that alternates policy learning, independent verification, and versioned skill‑library updates on a shared‑state orchestration graph. By merging histories that differ only in the order of independent steps, it pools evidence across equivalent executions and uses flow‑share readouts, signed utility ranks, verifier evidence, and residual‑variance plateaus to decide when and which skills to edit. Experiments on question answering, mathematical reasoning, interactive decision making, and code generation show that R$^2$ Flow improves task accuracy and library‑edit precision compared to heuristic orchestration, reinforcement learning, and other skill‑evolution baselines, and it transfers across different executors.
The paper introduces EDIC, a framework for per‑instruction compliance in large language models that interact with external data. It decomposes the task into identifying, classifying, and acting on each embedded directive, using a scene‑grounded pipeline and a generative judge trained with GRPO to attribute compliance at the instruction level. Experiments on two backbones and five benchmarks show that EDIC improves safety and utility, achieving up to a 7.7% increase in harmful‑rejection and a 12.5% boost in Arena‑Hard‑Auto utility on Qwen3‑4B.
SpeechCritic learns a diagnostic speech judge from only about 300 human-labeled comparisons, calibrating a frontier audio-language model with non-binding hints that select acoustic measurements agreeing with human judgments. The approach improves dimension-level agreement by 6.3 points and reduces mismatch in Tie rates by 10.4 points, and training a 7B judge with SFT, OPD, and RL yields distinct behaviors, with RL producing more specific rationales. The method works across language pairs, demonstrating a path from limited human preferences to a diagnostic speech judge.
The paper introduces a framework for reinforcement learning in Markov decision processes where policies are ordered by the information they provide, allowing data from one policy to estimate the performance of others. An epoch‑based algorithm is developed for infinite‑horizon average‑cost MDPs with continuous state and action spaces, achieving regret that scales with the width of the policy class’s information order. The authors apply the framework to lost‑sales inventory, dual‑sourcing, and state‑dependent queueing systems, demonstrating improved regret bounds and empirical performance without requiring global convexity or specific arrival‑rate structures.