Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2610.23193v1 Announce Type: cross Abstract: Motivated by applications in reinforcement learning from human feedback, we consider a setting where a given collection of rankings needs to be compr...
arXiv:2505.24776v2 Announce Type: replace Abstract: Diffusion has emerged as a powerful framework for generative modeling, achieving remarkable success in applications such as image and audio synthes...
arXiv:2610.23234v1 Announce Type: new Abstract: In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult b...
arXiv:2610.23587v1 Announce Type: new Abstract: Post-training is widely assumed to reduce the social biases language models inherit from pretraining, yet direct evidence for this assumption remains m...
arXiv:2606.27210v2 Announce Type: replace Abstract: We argue that safety classifiers should model user intent as an explicit signal between the prompt and the final label. To study this, we introduce...
arXiv:2610.17560v2 Announce Type: replace Abstract: Complex and long-horizon agent tasks often rely on frontier models deployed in the cloud to achieve strong performance. However, using these models...
The paper investigates the initial step of automated research ideation: generating actionable limitations from literature. By creating the ActLim-Bench dataset of 18,279 evidence‑linked supervision triples, the authors develop a three‑stage post‑training pipeline that improves recall over existing methods. Human evaluations show that the generated limitations are preferred 64.9% of the time and enhance downstream ideation systems.
JAXtari is an open‑source, high‑performance reimplementation of Atari 2600 environments that enables massive GPU parallelization. It allows PPO agents to complete 100 M environment steps on a single A100‑GPU in under an hour, far faster than the CPU‑based ALE. Additionally, JAXtari provides object‑centric observations and a flexible modification system, facilitating research into causal modeling, neuro‑symbolic methods, and rapid agent adaptation.
The paper introduces a reason-guided preference-based reinforcement learning framework that learns a human reward model from trajectory-level feedback. It augments traditional preference labels with reason tags that explain why a behavior is preferred, thereby capturing both human choices and their underlying rationale. By combining the learned human reward with the environment reward during policy optimization, the agent can improve task performance while producing more interpretable, human-aligned, and preference-sensitive behavior.
arXiv:2610.22797v1 Announce Type: cross Abstract: Reasoning rerankers write a rationale before ranking a candidate slate, and existing systems read item metadata to do so, even though generative reco...
arXiv:2610.22608v1 Announce Type: new Abstract: Recent advances in video generation let users create high-quality videos from natural language descriptions, yet obtaining results that match user inte...
arXiv:2307.10631v2 Announce Type: replace-cross Abstract: The practice of code reuse is crucial in software development for a faster and more efficient development lifecycle. In reality, however, cod...
WEAVE is a framework that converts verified pull‑request histories into executable, dependency‑structured tasks for code agents. It organizes related PRs into bounded directed acyclic graphs (DAGs) using repository‑calibrated typed change loci, creating unified tasks that reflect real‑world code evolution. The framework also introduces Graph Credit, a deterministic, test‑grounded metric that measures partial progress consistent with dependencies, providing dense, verifiable rewards for reinforcement learning.
EvolvingBrain introduces an adaptive skill system that enables multi-modal agents to self‑evolve in open worlds. The system comprises Stratified Skill Self‑Evolution, which extracts procedural and problematic patterns from trajectories to build a multi‑granularity skill architecture, and State‑Aware Adaptive Execution, which uses structured memory to guide skill fusion based on the agent’s current state. Experiments on VAB‑Minecraft and DiscoveryWorld show improved success rates and generalization, with code available for public release.
arXiv:2610. 21055v1 Announce Type: new Abstract: Recursive self-improvement, in which AI systems build stronger AI systems, is one promising route to superintelligence.
CleanRM is a reward‑modeling approach that mitigates the bias caused by noisy preference data in RLHF. It calibrates sample‑wise noise ratios using a small set of clean feedback via optimal transport, aligning the joint distribution of noisy and clean preferences. The method then trains a reward model with a noise‑corrected surrogate loss, proven to be unbiased with respect to the clean risk, and outperforms existing denoising baselines in experiments.
The paper studies how the distribution of information across chain‑of‑thought steps—measured by LP variance and Caveman compression ratio—relates to answer correctness in large language models. Across six model sizes, two training paradigms, and 60 GSM8K problems, it finds that lower LP variance predicts accuracy in supervised fine‑tuned models but not in reinforcement‑learning distilled ones, that models prefer more complex distributions, and that steering density at inference time harms accuracy while pre‑generation nudging and a new Entropy‑Gated Density Routing method improve it. The work demonstrates that natural density patterns are functionally important for reasoning and offers practical techniques to enhance performance.
The paper examines the challenge of safely allowing autonomous AI agents to improve themselves. It formalizes the problem by modeling agents as programs and defining improvement via a potentially uncomputable semantic score. The authors prove that in general, no terminating procedure can reliably accept only genuine improvements while never missing any, but they identify a practical bounded regime where exhaustive simulation can achieve these guarantees, noting its limitations.
The paper introduces PRM4RL, a framework that combines task planning and dense reward generation using large language models for reinforcement learning. It decomposes complex tasks into manageable subtasks and employs a fine‑grained progress function to monitor execution, which is then used to construct a Progress Reward Model for potential‑based reward shaping. Experiments on robotics control tasks show that PRM4RL outperforms existing LLM‑based planning and reward methods, achieving state‑of‑the‑art performance.
EliteReplay is a post‑training method that combines GRPO with a quality‑diversity archive to preserve diverse solutions in reinforcement learning. By collecting distinct strategies from a frozen base model before training and replaying them during training, the approach improves accuracy while maintaining a broader set of correct solutions. On text‑to‑SQL tasks, EliteReplay outperforms baselines, achieving higher pass@128 scores and up to 77% more distinct correct strategies, and it better handles ambiguous questions by covering multiple interpretations.