The study evaluates synthetic pre‑pretraining (PPT) across a wide range of models (500 M–7 B parameters) and training budgets (up to 100 B tokens). Results show that PPT consistently improves downstream performance and token efficiency, saving at least 21 B tokens at the 3 B scale, but these gains do not appear to stem from a grammatical prior. Instead, PPT benefits arise from tasks that enhance long‑range retrieval, and the improvements remain robust across diverse data mixtures, diminishing only when web text is omitted.
By Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus, Michal \v{S}tef\'anik, Aline Villavicencio, Nikolaos Aletras
arXiv:2608.28444v1 Announce Type: cross
Abstract: Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previo...
By Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, Emy Gervais
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
arXiv:2604. 26157v4 Announce Type: replace-cross Abstract: Structural generalization in semantic parsing requires systems to apply learned compositional rules to novel structural combinations.
By Zichao Wei
MS-GLA introduces Multi-Scale Gated Linear Attention, distributing attention heads across multiple temporal resolutions to address representational bottlenecks in GLA Transformers. Coarser resolutions pool longer token spans for long‑range dependencies, while finer heads capture local syntax, and a learnable fusion layer recombines outputs dynamically. Experiments on language modeling, recall‑intensive tasks, and long‑context generalization show MS‑GLA outperforms GLA with up to 18.9% improvement on recall tasks and 9.5% lower perplexity on language modeling benchmarks.
By Prasoon Dev, Anirudh Sankar, Vasudeva Varma
The paper investigates whether intrinsic tokenizer features can predict the performance of downstream language models. By training about 200 models across nine languages and evaluating numerous tokenizer metrics, the authors find that none of the existing metrics consistently forecasts model performance across different families, tasks, and languages. They propose a new linguistically grounded metric that captures aspects of tokenizer quality missed by prior measures, though its predictive power may still vary.
By Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava, Kevin Jiang, Mridul Sharma, Catherine Arnett
arXiv:2606. 25987v1 Announce Type: cross Abstract: Large language models (LLMs) attain remarkable surface fluency on code, yet they neither formally guarantee the syntactic validity of their output nor leverage the hierarchical structure defining the target language.
By Alexandre Bouayad
arXiv:2508. 16771v3 Announce Type: replace-cross Abstract: Code Language Models (CodeLLMs) learn token importance from data correlations, whereas human developers attend selectively to semantically salient code.
By Yifan Zhang, Chen Huang, Yueke Zhang, Jiahao Zhang, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, Yu Huang
The paper investigates how Spoken Language Models (SLMs) process speech compared to text, noting that current SLMs show weak alignment between speech and text representations despite strong downstream performance. The authors propose a framework that separates length mismatch from semantic alignment to better match speech and text representations. Experiments on multiple benchmarks demonstrate that this approach yields competitive results against strong baselines, highlighting the need to explicitly address structural differences between speech and text in SLM training.
By Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen
arXiv:2610.10114v1 Announce Type: new
Abstract: The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine differ...
By Xiaoran Liu, Ziwei He, Xipeng Qiu
Fine‑tuning reshapes internal representations of large language models, affecting attention patterns and layer‑wise activations. The study shows that components identified by EAP as important for task performance cluster in specific layers, yet these layers do not align with those undergoing the largest representational changes. Additionally, overlapping EAP components across different tasks do not guarantee cross‑task transfer and can even degrade performance when tasks differ in nature.
By Lingfang Li, Procheta Sen, Shubham Das, Danushka Bollegala
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y