Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2609.16563v1 Announce Type: cross Abstract: We have developed the MAL Simulator, a cyber operation simulator based on the Meta Attack Language (MAL). The MAL Simulator is intended for decision-...
arXiv:2607.09709v2 Announce Type: replace Abstract: Post-training a code generator against a learned judge can optimize proxy features that raise the score without improving the artifact. We study th...
Meta-LinEXP3 is an online-within-online algorithm designed for adversarial linear contextual bandits with random action sets. It builds a task-level prior from completed tasks to guide an inner LinEXP3 learner, achieving an σO(√n) per‑task regret when context distributions are known and an σO(n^{2/3}) regret with a past‑only regularized moment estimator when they are unknown. The paper also links prior accuracy to transfer regret, showing that better priors yield sublinear, transfer‑dependent regret across tasks, and demonstrates the method on structured hyperspectral tensor sampling.
arXiv:2609.16340v1 Announce Type: cross Abstract: Machine translation systems are periodically upgraded to stronger models, but the available preference signal is human post-edits of an older system'...
GrowMTP is a method that trains a draft head entirely within the reinforcement learning (RL) loop, using supervision from the RL verification step and a rollout distribution that is narrower than pretraining. By detaching draft‑head updates from the policy backbone, it enables online training of the draft head from scratch. Experiments on Qwen3‑4B, MiMo‑7B‑SFT, and Qwen3.5‑4B‑Base show rollout speedups ranging from 1.36× to 2.13× and overall end‑to‑end speedups from 1.20× to 1.60×, making it a modular acceleration component for RL frameworks lacking pretrained draft heads.
FluxVLA Engine is an open, configuration‑driven platform that unifies the fragmented components of embodied policy development—datasets, visual‑language and world models, action heads, learning methods, distributed training, simulation evaluation, inference, and robot interfaces—into a reproducible data‑to‑deployment workflow. It adds features such as compositional dual‑arm simulation, scalable automatic data generation, human‑in‑the‑loop rollout and correction, Real‑Time Chunking for fast inference, and lightweight remote GPU serving, thereby linking offline learning, simulation validation, online correction, and real‑robot execution under shared, auditable contracts. The engine aims to eliminate engineering bottlenecks that currently separate promising embodied‑learning algorithms from reliable, reproducible deployment.
The paper investigates whether frozen language models can detect a corrupted reward signal by using a single verified record in a two‑option game. In the game, a payout swap and a lying reporter produce identical histories, but a single line confirming the true outcome allows the models to almost perfectly identify the liar. However, the models frequently misclassify honest reporters as liars, with error rates ranging from 26% to 58% depending on model size and wording, indicating a significant limitation in their ability to interpret verified data.
MiCRo is a two‑stage framework that improves personalized preference learning for large language models. It first uses a context‑aware mixture model to capture diverse human preferences from large binary preference datasets, then applies an online routing strategy to dynamically adjust mixture weights based on context, reducing ambiguity. Experiments on multiple datasets show that MiCRo captures diverse preferences and enhances downstream personalization.
The paper introduces ANCHOR, an external supervisory framework driven by large language models (LLMs) that provides evaluative feedback at multiple stages of self‑evolving agents. By integrating ANCHOR into two open‑source self‑evolving agent frameworks, the authors demonstrate that it significantly improves safety performance while preserving core capabilities across coding, mathematical reasoning, and safety tasks. The study also finds that supervision based on execution results is especially effective and that increasing supervision frequency yields diminishing returns, offering practical guidance for future research.
ViCo is a training framework that improves the visual quality of AI-generated academic charts by using iterative self-reflection to align chart images with reference designs. It introduces a self-supervised warm-up stage that augments Monte Carlo Tree Search with consistency-based pruning, and a multi-step reinforcement learning algorithm that employs counterfactual baselines to address sparse rewards. An automatic, multifaceted evaluation framework assesses style, layout, and semantic consistency through a hierarchical heterogeneous layout graph, and experiments on three public benchmarks show ViCo’s performance approaching that of proprietary large language models.
The paper introduces a meta‑reinforcement learning framework for wireless edge caching that uses a shared initialization to adapt quickly to each base station (BS). Each BS runs a local PPO agent modeled as a Semi‑Markov Decision Process, while a shared meta‑policy is trained via a MAML‑style loop. To reduce the high variance of the meta‑gradient estimator caused by uniform random sampling of BSs, the authors propose gradient‑based clustering, grouping BSs by local gradient similarity and sampling proportionally from each cluster, proving that this yields a lower‑variance estimator under BS heterogeneity.
The paper studies recurrent Twin Delayed Deep Deterministic Policy Gradient (TD3) agents in environments with evolving hidden disturbances, focusing on how observation history, action history, history length, and network structure influence performance. Three recurrent architectures are compared under controlled disturbances, revealing that action history is crucial when responses depend on prior actions and that a unified temporal sequence of action-observation pairs outperforms separate branches. The authors introduce H‑TD3, which reuses actor-generated recurrent states to initialize the critic, and demonstrate that these architectures excel in a rover wheel‑slip simulation, with policies trained on temporally structured disturbances transferring better to unseen slip dynamics.
The paper introduces CoER, a framework that defends language‑model agents against adaptive indirect prompt injection (IPI) by employing attacker‑defender co‑evolution and refinement. CoER models IPI as a general‑sum Markov game, uses Co‑PPO to maintain historical opponent populations, and fine‑tunes defenders only on verified safe demonstrations. In experiments across seven domains, CoER cuts attack success from 38.5% to 0.2% while boosting task utility from 63.2% to 76.3%.
The paper introduces DiaWhisper-DPO, an end‑to‑end model that fine‑tunes Whisper-large-v3 with LoRA and a frame‑level role head to transcribe and attribute utterances in clinical interviews. It further refines the system using failure‑mined preference optimization (DPO) that leverages genuine decoding failures as rejected completions, eliminating the need for human preference data. On the DAIC‑WOZ dataset, DiaWhisper‑DPO attains 0.973 role accuracy and 0.119 DER, outperforming cascaded baselines by 72% and dramatically reducing seed variation, while also improving performance on the cross‑lingual PDCH‑HAMD dataset.
SAVOR is a training framework for multimodal large language models that adds token and answer confidence to the output schema, optimises a Group Relative Policy Optimisation objective to penalise calibration error and poor abstention, and uses the learned confidence at inference to revisit visual evidence only when uncertain. Experiments on POPE, HallusionBench, AMBER, and MMHal-Bench with InternVL3-8B and Qwen3-VL-8B backbones show that SAVOR reduces hallucination while maintaining general capability on MME and MMBench, achieving lower Expected Calibration Error than DPO and decoding baselines.
arXiv:2605.27293v2 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards has become a standard recipe for improving the reasoning abilities of large language models. Existin...
arXiv:2609.16372v1 Announce Type: new Abstract: Masked diffusion language models (dLLMs) generate text by iteratively denoising masked tokens with bidirectional attention. Extending reasoning across...
arXiv:2609.16056v1 Announce Type: cross Abstract: Humans carry behaviour knowledge of how to act in familiar situations into every new task rather than relearning it from scratch. There is no reason...
arXiv:2507.21931v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks....
arXiv:2609.16222v1 Announce Type: new Abstract: Stopgrads are widely used in training machine learning models, but stopgrads can alter the gradient, stationary points and convergence guarantees of th...