The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The paper studies how long‑context continual pre‑training (LCCP) scales with token budget and model size. An empirical additive power law is derived that predicts LCCP performance across five model scales and over a hundred budget configurations, and a new Constant‑Last4 protocol is introduced for stable monitoring. Experiments show that performance continues to improve without clear saturation, even after 128K context extension and fine‑tuning, indicating that scaling LCCP yields transferable gains for downstream tasks.
arXiv:2607. 14952v1 Announce Type: new Abstract: A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment.
arXiv:2609.14320v1 Announce Type: new Abstract: Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing con...
arXiv:2608. 01672v1 Announce Type: cross Abstract: Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later.
arXiv:2607. 07847v1 Announce Type: new Abstract: As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn?
arXiv:2606. 31208v1 Announce Type: new Abstract: Large tabular models (LTMs), i.