arXiv:2608. 15454v1 Announce Type: new Abstract: Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization.
Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains...
arXiv:2606. 27550v1 Announce Type: cross Abstract: Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding.
arXiv:2601. 03043v4 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency.
arXiv:2608.27658v1 Announce Type: new Abstract: Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte...
arXiv:2605. 08696v4 Announce Type: replace-cross Abstract: Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput.
arXiv:2605.15508v3 Announce Type: replace Abstract: The quadratic complexity of attention imposes severe memory and computational bottlenecks on Large Language Model (LLM) inference. This challenge i...
The paper investigates why current byte‑level language models, which use a hierarchical structure that down‑samples to words and then upsamples back to bytes, struggle with precise character manipulation. Experiments show that pure byte‑level models outperform hierarchical variants on character‑level tasks, and that byte‑level attention is the key component driving this advantage. The study explains the trade‑off between computational efficiency and fine‑grained character understanding in hierarchical byte models.
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
The paper proposes a modular tokenizer framework for multilingual large language models, allowing the creation of language‑specific subtokenizers that match monolingual compression quality. It introduces a pretraining strategy that samples these subtokenizers to limit predictions to relevant vocabularies, enabling efficient training and inference. This approach reduces memory usage and speeds up inference without compromising performance.
arXiv:2603. 20843v3 Announce Type: replace-cross Abstract: Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches.
Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding...
arXiv:2511. 11346v2 Announce Type: replace Abstract: Multi-token prediction (MTP) is a prominent strategy to significantly speed up generation in large language models (LLMs), especially in byte-level LLMs, which are tokeniser-free but prohibitively slow.