Randomized YaRN is a training method that enhances length generalization for large language models by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short-context data, tokens receive YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations. Evaluated on BABILong, Multi-Round Coreference Resolution, and LongBench v2, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K, outperforming standard fine‑tuning especially at far out‑of‑distribution lengths.
arXiv:2507.06415v3 Announce Type: replace-cross Abstract: Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK...
arXiv:2505. 17315v2 Announce Type: replace Abstract: Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored.
LongZero is a reinforcement learning method that enables large language models to improve long‑context reasoning without external data. It works by having the model first generate a compact, structured question‑answer pair with supporting evidence, then expanding that evidence into a long, multi‑document context. The approach combines self‑written multi‑hop reasoning and multi‑round retrieval tasks, trained with GRPO, and outperforms baseline RL methods on five long‑context reasoning benchmarks.
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.
The paper proposes a new method for training large language models to handle long-context reasoning by combining Group Relative Policy Optimization (GRPO) with on‑policy distillation (OPD). It introduces a synthetic multilingual dataset called LongBlocks that tests multi‑hop reasoning, contextual grounding, and long‑form generation. Experiments show that the combined approach outperforms either GRPO or OPD alone while maintaining short‑context performance.
arXiv:2607. 08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them.
arXiv:2607. 11614v1 Announce Type: cross Abstract: Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and linear memory scaling.
arXiv:2609.10441v1 Announce Type: cross Abstract: While Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context l...
arXiv:2607. 06160v1 Announce Type: cross Abstract: Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision.
arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.
While Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context limits. To address this, sequential approaches like...
arXiv:2610.19928v1 Announce Type: new Abstract: Reasoning over infinitely long documents is becoming a reality with recurrent memory agents. However, increasing context length exacerbates the trade-o...