Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2609.00504v1 Announce Type: cross Abstract: In this work, we study radically uncoupled learning in discounted general-sum Markov games. Assuming ``$\mathsf{ETH}$ for $\mathsf{PPAD}$", we show t...
The paper introduces a stage‑wise closed‑loop evaluation method to track how compression techniques affect learned driving policies. Using a POMDP formulation and PPO‑trained belief‑state policy in Gym‑Duckietown, the authors compress the actor step by step and test it across five driving curricula. They find that structured pruning first degrades driving capability, distillation can partially recover it but is limited by rehearsal data, and integer quantization further loses curricula requiring stopping and resuming, whereas the unpruned actor retains all curricula.
The paper introduces SAGE, a framework that selectively queries a Vision‑Language Model (VLM) teacher only when the learner is uncertain, using the teacher’s suggestions to guide training and distill them into a lightweight reinforcement learning policy. SAGE weights teacher actions by environment‑derived advantages, allowing the policy to improve beyond the imperfect VLM. Experiments on sparse‑reward visual reasoning and navigation tasks show that the learned policies can act without VLM guidance at evaluation, reduce VLM usage during training, and sometimes outperform the teacher itself.
arXiv:2609.01597v1 Announce Type: cross Abstract: Natural language is emerging as a primary feedback channel for improving language agents, capable of conveying intent, preferences, and causal struct...
arXiv:2506.13502v4 Announce Type: replace Abstract: Next-word prediction (NWP) trains language models against a single observed continuation, even though many contexts admit multiple plausible next w...
The study investigates whether post‑training methods—GRPO, SFT, and DPO—improve language models’ ability to follow prompt evidence that conflicts with memorized knowledge. By comparing nine training variants across different scales and families, the authors find that grounding gains are modest for GRPO, moderate for Conflict‑SFT, and near‑ceiling for DPO, but all largely rely on the same causal attention‑head set present in the starting checkpoint. Removing the starting‑model grounding direction suppresses these gains, while adding it back recovers a significant portion of DPO’s improvement, indicating that existing model machinery drives most of the observed gains.
The paper introduces Online Hyperparameter Optimization (OHPO), framing it as an infinitely many‑armed bandit problem over mixed and conditional search spaces. It proposes the IMABO framework, which couples any bandit policy with any oracle for proposing new configurations, and presents IMOSS—a restart‑free anytime policy with provable regret bounds. Experiments show that IMABO, combined with practical oracles such as TPE, an incumbent‑mutation oracle, and a pretrained tabular foundation model, outperforms random search across a range of settings from classical ML models to LLM‑based agents.
arXiv:2609.00892v1 Announce Type: new Abstract: Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforceme...
arXiv:2609.01418v1 Announce Type: cross Abstract: To mitigate the sample complexity of real-world reinforcement learning (RL), a common practice is to first train a policy in a simulator, where sampl...
arXiv:2603.10359v2 Announce Type: replace Abstract: Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitations of rejection s...
arXiv:2609.00213v1 Announce Type: new Abstract: Reinforcement learning fine-tuning of large language models increasingly adopts multiple reward dimensions, including verifiable rules, task-specific e...
AdaSearch introduces a two‑stage reinforcement learning framework that separates problem solving from the decision to search in large language models. By using an F1‑based decision metric, it explicitly evaluates when external search is needed, reducing unnecessary search calls while maintaining high question‑answering performance. Experiments show that AdaSearch improves search‑decision quality with only a minor impact on accuracy compared to always‑search strategies.
The paper introduces a two‑stage training framework that combines Supervised Fine‑Tuning (SFT) and Direct Preference Optimization (DPO) to improve multimodal disaster severity assessment. It creates two datasets—ReasoningSet for validated rationales and PreferenceSet for paired rationales—using a single Human‑in‑the‑Loop workflow. Experiments on InternVL‑3‑8B and LLaVA‑1.5‑7B show that SFT boosts classification accuracy and Macro‑F1, while DPO further enhances interpretability and alignment with human judgment.
The paper introduces CRAFT, a data‑centric fine‑tuning approach that aligns small language models (SLMs) for pre‑hoc reasoning in AI‑native 6G radio access networks (RAN). By automatically generating verified (input, trace, label) triplets and fine‑tuning with low‑rank adaptation, CRAFT achieves high accuracy and F1 scores on TRACTOR and IC xApp datasets while avoiding parse failures that plague RL methods like GRPO. It also reduces energy consumption by 59% compared to GRPO baselines, offering a more sustainable path to auditable AI in 6G RAN.
CoBRA is a counterfactual boundary‑learning framework designed to improve when a tool‑augmented language model should call an external tool. It builds internal and external experts from the same base model, collects paired trajectories, and estimates the reward margin between answering with and without tools. Using these margins, CoBRA partitions data into internal‑favored, external‑favored, and ambiguous cases, then applies Boundary‑Aware Cold‑Start SFT and MARS‑RL to optimize boundary decisions, leading to more efficient tool use and better accuracy on tool‑dependent out‑of‑distribution questions.
arXiv:2510.14980v3 Announce Type: replace Abstract: Large language models (LLMs) have shown strong abilities in writing and revising programs, yet many program-synthesis benchmarks still evaluate pro...
The paper introduces a discrete‑time Markov decision process (DTMDP) model for a finite‑horizon, multi‑item capacitated lot‑sizing problem where demand quantities are deterministic but demand‑arrival times are stochastic. It compares stochastic instances to deterministic counterparts, showing that stochastic timing significantly enlarges the state space, transitions, solution time, and memory usage. A genetic algorithm (GA) is proposed to search feasible state‑feedback policies, achieving an average optimality gap of about 3.44 % and a speedup of roughly 6.89× on challenging benchmark instances.
DiagEvo is a self‑evolution framework that guides language‑model training by extracting recurring error causes from a solver’s own failure history and storing them in a hierarchical error‑cause memory. The system classifies causes as Active or Mastered, uses this information to balance targeted question generation with exploration, and applies double‑confidence filtering to keep only intermediate‑difficulty questions. Experiments show that DiagEvo outperforms baselines on nine benchmarks for three solvers, achieving up to 72.3% mean accuracy on five mathematical reasoning tasks.
The paper introduces AnySearch, a reinforcement‑learning framework that trains a single policy to perform budget‑aware search for large language models under any budget constraint. The training proceeds in two phases: first, the agent learns with explicit budget state injection and structured reasoning prompts under linearly decaying budgets; second, the scaffold is removed and the agent adapts to randomly sampled budgets that match deployment conditions. The reward combines answer accuracy and budget efficiency, with adaptive weighting to emphasize efficiency for high‑accuracy queries and reduce it for low‑accuracy ones. Experiments on seven QA benchmarks demonstrate that AnySearch outperforms baselines across all budget scales, generalizes to unseen constraints, and improves tool productivity without excessive token overhead.
arXiv:2609.00638v1 Announce Type: cross Abstract: Retrieval is the first stage of modern search and advertising systems, selecting a candidate set from a large item universe for downstream ranking an...