← Back to all news
arXiv Computation and Language By Manas Mehta, Fangcong Yin, Greg Durrett

Randomized YaRN Improves Length Generalization for Long-Context Reasoning

Read the original on arXiv Computation and Language →

Randomized YaRN is a training method that enhances length generalization for large language models by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short-context data, tokens receive YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations. Evaluated on BABILong, Multi-Round Coreference Resolution, and LongBench v2, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K, outperforming standard fine‑tuning especially at far out‑of‑distribution lengths.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Oct 22

LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data

LongZero is a reinforcement learning method that enables large language models to improve long‑context reasoning without external data. It works by having the model first generate a compact, structured question‑answer pair with supporting evidence, then expanding that evidence into a long, multi‑document context. The approach combines self‑written multi‑hop reasoning and multi‑round retrieval tasks, trained with GRPO, and outperforms baseline RL methods on five long‑context reasoning benchmarks.

By Zhennan Wan, Jianfei Chen
llmsreinforcement-learningbenchmarks
More like this →
Hugging Face Trending Papers
Jul 9

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.

llmsnlpbenchmarks
More like this →
arXiv Computation and Language
Sep 11

A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation

The paper proposes a new method for training large language models to handle long-context reasoning by combining Group Relative Policy Optimization (GRPO) with on‑policy distillation (OPD). It introduces a synthetic multilingual dataset called LongBlocks that tests multi‑hop reasoning, contextual grounding, and long‑form generation. Experiments show that the combined approach outperforms either GRPO or OPD alone while maintaining short‑context performance.

By Miguel Moura Ramos, Duarte M. Alves, Andr\'e F. T. Martins
llmsreinforcement-learningfine-tuningefficiencysafety
More like this →