The paper introduces EXACT, a supervision-allocation objective that reweights long-context targets by inverse frequency to address a token-level supervision mismatch in packed training. Across seven Qwen/LLaMA configurations, EXACT consistently improves NoLiMa and RULER metrics, with notable gains on Qwen2.5-0.5B and LLaMA-3.2-3B. Standard QA and reasoning performance remain largely unchanged, and a distance-resolved probe shows benefits primarily when evidence is thousands of tokens away.
arXiv:2505. 19293v2 Announce Type: replace-cross Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.
arXiv:2607. 09415v1 Announce Type: cross Abstract: Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs.
arXiv:2610.02404v1 Announce Type: new Abstract: We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simple...
arXiv:2602. 15257v3 Announce Type: replace-cross Abstract: We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text.
The paper studies how long‑context continual pre‑training (LCCP) scales with token budget and model size. An empirical additive power law is derived that predicts LCCP performance across five model scales and over a hundred budget configurations, and a new Constant‑Last4 protocol is introduced for stable monitoring. Experiments show that performance continues to improve without clear saturation, even after 128K context extension and fine‑tuning, indicating that scaling LCCP yields transferable gains for downstream tasks.
arXiv:2610.19928v1 Announce Type: new Abstract: Reasoning over infinitely long documents is becoming a reality with recurrent memory agents. However, increasing context length exacerbates the trade-o...
arXiv:2607. 06160v1 Announce Type: cross Abstract: Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision.
Reasoning over infinitely long documents is becoming a reality with recurrent memory agents. However, increasing context length exacerbates the trade-off between reasoning performance and inference ef...
Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this qu...
arXiv:2607. 02509v1 Announce Type: new Abstract: Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications.
arXiv:2605. 21850v2 Announce Type: replace-cross Abstract: Recent development of agents has renewed demand for long-context reasoning capacity of LLMs.
The paper introduces the Proximity Trap, a problem where long-context language models fail to attend to distant evidence because of competition from abundant, irrelevant proximal background. To mitigate this, the authors propose LYRA, a t‑distributed directional matching mechanism that reshapes the retrieval distribution to focus more on task‑relevant evidence while preserving positional information. Experiments on LongBench‑v2, RULER, and LongBench show consistent performance gains, and a new benchmark, ProxBench, is presented to evaluate distant evidence usage under varying proximal interference.