Hugging Face Trending Papers
Toppling the Hierarchy in Byte-level Language Modeling
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Computation and Language
Sep 2
Toppling the Hierarchy in Byte-level Language Modeling
The paper investigates why current byte‑level language models, which use a hierarchical structure that down‑samples to words and then upsamples back to bytes, struggle with precise character manipulation. Experiments show that pure byte‑level models outperform hierarchical variants on character‑level tasks, and that byte‑level attention is the key component driving this advantage. The study explains the trade‑off between computational efficiency and fine‑grained character understanding in hierarchical byte models.
Hugging Face Trending Papers
Aug 16
Dynamic Multi-Byte Prediction With Hierarchical Language Models
Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains...
arXiv Computation and Language
Aug 31
When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
arXiv:2608.27658v1 Announce Type: new Abstract: Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte...
arXiv AI
Aug 18
Dynamic Multi-Byte Prediction With Hierarchical Language Models
arXiv:2608. 15454v1 Announce Type: new Abstract: Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization.
arXiv Computation and Language
Sep 22
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
arXiv:2601.06787v2 Announce Type: replace Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...
arXiv AI
Aug 25
Lexical Perturbations Disrupt LLM Reasoning: An Empirical Study of Attention Diversion
The study investigates how lexical perturbations—such as keyboard noise, character swaps, and filler insertion—affect large language models (LLMs) on reasoning benchmarks. Four open-weight instruction-tuned models and frontier models were evaluated, revealing that character-level perturbations significantly reduce accuracy, especially on multi-step reasoning tasks, while filler insertion has minimal impact. The authors attribute this asymmetry to Attention Diversion, where fragmented subword tokenization draws disproportionate attention in middle and final transformer layers; they demonstrate that both token content and attention allocation are coupled, making it difficult for inference-time repair strategies to fully recover performance.