arXiv Machine Learning
LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
1d ago
CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning
arXiv:2603.00889v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, largely enabled by supervised fine-tuning (SFT)- and reinforce...
arXiv AI
Jun 18
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
arXiv:2606. 18831v1 Announce Type: cross Abstract: Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories.
arXiv Computation and Language
Sep 11
A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation
The paper proposes a new method for training large language models to handle long-context reasoning by combining Group Relative Policy Optimization (GRPO) with on‑policy distillation (OPD). It introduces a synthetic multilingual dataset called LongBlocks that tests multi‑hop reasoning, contextual grounding, and long‑form generation. Experiments show that the combined approach outperforms either GRPO or OPD alone while maintaining short‑context performance.
arXiv AI
Oct 1
StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning
StateTree is a reinforcement learning approach that improves long‑term dialogue reasoning by building a tree‑structured auxiliary task from limited dialogue data. The method embeds key‑value records across multiple sessions into a binary tree, requiring the model to traverse from root to leaf, retrieve records, compare timestamps, and identify a target question among distractors. Curriculum RL training increases tree depth, and a compositional variant trains the model to combine partial reasoning fragments, enabling cross‑session retrieval, temporal reasoning, knowledge updates, and multi‑hop reasoning while generalizing from 10K‑token to 128K‑token contexts.
arXiv AI
Jul 8
LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
arXiv:2607. 06160v1 Announce Type: cross Abstract: Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision.
arXiv Machine Learning
Sep 1
PERK: Long-Context Reasoning as Test-Time Learning
arXiv:2507.06415v3 Announce Type: replace-cross Abstract: Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK...