arXiv Computation and Language September 1, 2026 By Jinchang Zhu, Jindong Li, Chengyu Zou, Rong Fu, Chao Wang, Haowei He, Menglin Yang

Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing

Read the original on arXiv Computation and Language →

The paper introduces EXACT, a supervision-allocation objective that reweights long-context targets by inverse frequency to address a token-level supervision mismatch in packed training. Across seven Qwen/LLaMA configurations, EXACT consistently improves NoLiMa and RULER metrics, with notable gains on Qwen2.5-0.5B and LLaMA-3.2-3B. Standard QA and reasoning performance remain largely unchanged, and a distance-resolved probe shows benefits primarily when evidence is thousands of tokens away.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jun 4

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

arXiv:2505. 19293v2 Announce Type: replace-cross Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.

By Wang Yang, Hongye Jin, Shaochen Zhong, Song Jiang, Qifan Wang, Vipin Chaudhary, Xiaotian Han

llms benchmarks

arXiv AI

Jul 13

Self-Guided Test-Time Training for Long-Context LLMs

arXiv:2607. 09415v1 Announce Type: cross Abstract: Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs.

By Xinyu Zhu, Zhe Xu, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Kaushik Rangadurai, Hua Zhi, Frank Shyu, Sandeep Pandey, Luke Simon, Yu Meng, Xi Liu

llms benchmarks

arXiv Machine Learning

Oct 5

Trained Agentic Context Management

arXiv:2610.02404v1 Announce Type: new Abstract: We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simple...

By Bryce Sandlund

llms agents benchmarks

arXiv AI

Jun 30

How to Train Your Long-Context Visual Document Model

arXiv:2602. 15257v3 Announce Type: replace-cross Abstract: We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text.

By Austin Veselka

llms nlp fine-tuning benchmarks

arXiv Computation and Language

Oct 22

Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

The paper studies how long‑context continual pre‑training (LCCP) scales with token budget and model size. An empirical additive power law is derived that predicts LCCP performance across five model scales and over a hundred budget configurations, and a new Constant‑Last4 protocol is introduced for stable monitoring. Experiments show that performance continues to improve without clear saturation, even after 128K context extension and fine‑tuning, indicating that scaling LCCP yields transferable gains for downstream tasks.

By Ziqing Qiao, Xiaoyue Xu, Sikui Zhang, Yinuo Xu, Bingxiang He, Chaojun Xiao, Xu Han

agents fine-tuning

arXiv Computation and Language

Oct 15

MemDuet: Learning Dual-Process Memory for Long-Context Reasoning

arXiv:2610.19928v1 Announce Type: new Abstract: Reasoning over infinitely long documents is becoming a reality with recurrent memory agents. However, increasing context length exacerbates the trade-o...

By Jiangcheng Song, Haodong Jing, Yulin Mu, Lesong Tao, Jixin Wang, Jin Cui, Tianjing Ren, Runxuan Xie, Yongqiang Ma, Nanning Zheng

llms agents reinforcement-learning fine-tuning benchmarks

One email a morning, machine-written

Related stories

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

Self-Guided Test-Time Training for Long-Context LLMs

Trained Agentic Context Management

How to Train Your Long-Context Visual Document Model

Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

MemDuet: Learning Dual-Process Memory for Long-Context Reasoning