← Back to all news
arXiv Computation and Language By Manas Mehta, Fangcong Yin, Greg Durrett

Randomized YaRN Improves Length Generalization for Long-Context Reasoning

Read the original on arXiv Computation and Language →

Randomized YaRN is a training method that enhances length generalization for large language models by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short-context data, tokens receive YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations. Evaluated on BABILong, Multi-Round Coreference Resolution, and LongBench v2, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K, outperforming standard fine‑tuning especially at far out‑of‑distribution lengths.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

Hugging Face Trending Papers
Jul 9

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.

llmsnlpbenchmarks
More like this →
arXiv Computation and Language
2d ago

A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation

The paper proposes a new method for training large language models to handle long-context reasoning by combining Group Relative Policy Optimization (GRPO) with on‑policy distillation (OPD). It introduces a synthetic multilingual dataset called LongBlocks that tests multi‑hop reasoning, contextual grounding, and long‑form generation. Experiments show that the combined approach outperforms either GRPO or OPD alone while maintaining short‑context performance.

By Miguel Moura Ramos, Duarte M. Alves, Andr\'e F. T. Martins
llmsreinforcement-learningfine-tuningefficiencysafety
More like this →
arXiv AI
Jul 14

Extending LLM Context via Associative Recurrent Memory

arXiv:2607. 11614v1 Announce Type: cross Abstract: Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and linear memory scaling.

By Gleb Kuzmin, Ivan Rodkin, Aydar Bulatov, Yuri Kuratov, Lyudmila Rvanova, Mikhail Katkov, Ilia Sochenkov, Misha Tsodyks, Timothy Baldwin, Mikhail Burtsev, Artem Shelmanov
llmsfine-tuning
More like this →