arXiv:2510. 05592v2 Announce Type: replace Abstract: Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios.
The paper introduces Multi-Level Reasoning (MLR), a two-level stochastic framework that separates long chain-of-thought generation into a high-level planner producing structured step descriptors and a low-level executor generating detailed reasoning. MLR replaces single-policy reinforcement learning with an iterative Step‑DPO preference optimization that uses Twisted Sequential Monte Carlo for scalable, stepwise supervision, improving training stability and accuracy. Experiments on math, science, and logic benchmarks show that MLR outperforms both supervised fine‑tuning and strong RL baselines while maintaining robustness over extended reasoning horizons.
arXiv:2611.05847v1 Announce Type: cross Abstract: Joint-Embedding Predictive Architectures (JEPAs) support planning by comparing predicted latent states with an encoded goal. For this comparison to g...
APEx is a hierarchical framework that organizes a deep research agent’s interaction history into instance-level trajectory memories and category-level procedural skills. It couples these through an Executor, Distiller, and Planner, trained with a three-stage alternating GRPO paradigm to enable reward-guided skill distillation. At test time, distilled skills act as procedural priors for online Planner adaptation via skill-guided reinforcement learning, achieving state‑of‑the‑art results on seven benchmarks, outperforming GPT‑5.4 by 14.7 points and the best memory‑augmented baseline by 3.0 points.
The paper introduces Planned Test-Time Scaling (PTTS), a method that replaces independent sampling of reasoning branches with a coordinated joint policy. PTTS uses a planner to generate distinct solution outlines for each branch and an executor to produce full solutions, thereby improving coverage of complementary reasoning modes. Two variants—PTTS‑ZS (zero‑shot) and PTTS‑RL (reinforcement‑learned)—demonstrate significant gains on five mathematical reasoning benchmarks, with PTTS‑RL achieving up to a 13.4‑point improvement in pass@64 over repeated sampling.
EmbodiedMind introduces a three-stage training paradigm for embodied foundation models that tackles inefficient sample use, task imbalance, and credit assignment in long-horizon planning. The stages—Rejection Sampling-based Fine‑Tuning, Iterative Rejection GRPO, and Trie‑GRPO—filter low‑informative data, balance task difficulty, and use action prefix trees for step‑level advantage estimation. This approach yields a state‑of‑the‑art average performance of 70.02% across 18 benchmarks, notably improving long‑horizon task planning accuracy.
SPIRAL is a reinforcement‑learning framework that trains language models to employ three inference primitives—sequential reasoning within a trace, parallel sampling of independent traces, and aggregation of those traces—within a single compute pipeline. The model first generates multiple independent chain‑of‑thought traces in parallel, then produces a final aggregation trace conditioned on them, with all components optimized end‑to‑end for the reward of the aggregated response. Experiments on reasoning tasks demonstrate that SPIRAL scales efficiently with inference compute, achieving up to 11× better scaling efficiency and 15% higher performance compared to the GRPO baseline when all three primitives are scaled.
The paper introduces AnytimeReasoner, a framework that optimizes large language models for anytime reasoning by truncating the reasoning process to fit sampled token budgets and requiring the model to summarize the answer for verification. This approach adds verifiable dense rewards, improving credit assignment in reinforcement learning. The authors also propose Budget Relative Policy Optimization (BRPO), a variance reduction technique that enhances learning robustness and efficiency, and demonstrate that their method outperforms GRPO on mathematical reasoning tasks across various budgets and prior distributions.
DeepPlanner is an end-to-end reinforcement learning framework designed to enhance the planning capabilities of deep research agents. It introduces an entropy-based advantage shaping mechanism that allocates larger updates to high-entropy planning tokens and selectively upweights sample-level advantages during planning-intensive rollouts. Experiments on seven deep research benchmarks show that DeepPlanner improves planning quality and achieves state‑of‑the‑art results with a lower training budget.
GRASP is a multi-stage planning framework that improves the reliability of large language models on complex tasks. It separates planning into three specialized modules—GenPlan for global macro-guidelines, RevPlan for exploring localized strategies, and VerPlan for multi-criteria evaluation—allowing context isolation and strict macro-regularization. Experiments show GRASP outperforms direct LLM planners by significant margins on datasets such as Natural Plan Calendar Scheduling, ZebraLogic, and SciBench Math, and it mitigates performance collapse in multi-task and dual-task settings.
arXiv:2606. 00618v1 Announce Type: new Abstract: Generative models have emerged as a powerful paradigm for AI planning, yet their performance remains constrained by the training data distribution.
Language model reasoning can be substantially improved at test time via scaffolds that scale inference compute across different primitives -- sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During post-training, however, language models are optimized only for sequential reasoning within a single trace.