arXiv Machine Learning
Jul 31
Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search
arXiv:2607. 27422v1 Announce Type: new Abstract: Good action rankings do not make a contrastive critic safe to maximize.
Goal Retrieval versus Action-Value Ordering: Contrastive Critics under Best-of-$K$ Selection
Related stories
arXiv Machine Learning
Aug 28
Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning
The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.
arXiv Machine Learning
Jul 10
Beyond Success Rates: Trainability and Extractability for Offline GCRL
arXiv:2602. 05459v2 Announce Type: replace Abstract: Offline goal-conditioned reinforcement learning (GCRL) is typically benchmarked by the best tuned success rate of each method.
arXiv AI
Sep 2
CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training
arXiv:2609.00892v1 Announce Type: new Abstract: Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforceme...
arXiv AI
Oct 12
Unsupervised Multimodal Reward Modeling
The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.
arXiv AI
Sep 10
ARC-Bench: Closed-Loop Replanning Masks Broken Action Ranking in Frozen JEPA World Models
ARC‑Bench is a new benchmark that tests whether frozen JEPA‑style latent world models can correctly rank candidate actions by latent distance. The study finds that the assumption of latent rankability fails dramatically in both navigation and manipulation tasks, with the top‑scored actions often being suboptimal. Closed‑loop replanning masks this defect, but reducing replanning frequency reveals the underlying ranking failures.
arXiv Machine Learning
Aug 11
CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning
arXiv:2608. 07719v1 Announce Type: new Abstract: Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment.
arXiv AI
Oct 12
Matched-Context Group Policy Optimization for LLM Agent Training
Matched-Context Group Policy Optimization (MCGPO) is a method for training long-horizon language model agents by constructing weighted action preferences from pre-collected trajectories. It defines comparison contexts using pre-action state summaries and rubric progress, and employs two-action support with exact and coarse matching rules to weight evidence for learning. Experiments across six environment-backbone settings show that MCGPO improves mean success rates over matched GraphGPO, achieving up to 5.16 percentage point gains on WebShop with Qwen3-4B and 8B models.
arXiv AI
Sep 24
Reinforcement Learning with Decomposed Subtasks
The paper introduces Reinforcement Learning with Decomposed Subtasks (RLDS), a method that splits trajectory rewards into per‑subtask shares before policy updates, replacing the scalar advantage used in Group Relative Policy Optimization (GRPO). RLDS employs Subtask‑Decomposed Advantage Estimation (SDAE) to compute group‑relative advantages and distribute credit to tokens based on subtask importance, focusing on steps where a reflection marks a subtask as consequential. Experiments on four benchmarks—FrozenLake, HotpotQA, ScienceWorld, and DeepResearch—show that RLDS improves performance on high‑heterogeneity tasks (ScienceWorld and FrozenLake) and is more compute‑efficient than scalar GRPO for long rollouts.
arXiv Machine Learning
Oct 7
VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
arXiv:2610.08402v1 Announce Type: new Abstract: Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two r...
arXiv AI
Jul 29
Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
arXiv:2607. 25136v1 Announce Type: new Abstract: Research on preference optimization often varies the training objective while holding the data fixed.
arXiv AI
6d ago
Verdict: Encode Once, Decide Many
Verdict-MQ is an open encoder that processes a single piece of state once and then answers multiple typed questions about that state, avoiding separate passes for each question. By using a structured attention mask, it ensures the state encoding is independent of question order and prevents questions from influencing one another. On the typed‑decisions benchmark, Verdict‑MQ outperforms a larger specialist model, achieving higher accuracy and better calibration, while on other benchmarks it shows mixed results compared to fine‑tuned baselines.