← Back to all news
arXiv Machine Learning By Nicol\'as Vera Z\'u\~niga

Patch where the text is new: a compression view of where byte-level models should spend compute

Read the original on arXiv Machine Learning →

The paper introduces a novel patching strategy for byte‑level language models, called novelty patching, which starts a new patch when upcoming bytes are unlikely to repeat recent context, inspired by LZ77 compression. Experiments on a 1.9 GB math corpus show that novelty patching improves final‑answer accuracy from 47 % to 72 % at 12 M parameters and raises computed‑result accuracy from 21 % to 57 % at 50 M parameters, while having limited impact on prose and source code. The study also analyzes patch boundary distributions and reports several failed approaches, such as typed uncertainty and context‑aware novelty prediction. whyItMatters":"The findings demonstrate that a simple, data‑driven patching rule can significantly enhance the performance of byte‑level models on tasks involving computed results, offering a practical improvement for large language models that process raw text without explicit labels."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Oct 9

Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching misses

The paper investigates how byte‑level language models that use entropy‑based patching (like BLT) fail to compute certain predictable but value‑dependent positions, such as numbers following an equals sign in math solutions. By introducing a boundary‑dependence signal—measuring the model’s loss when a patch start is removed—they show that this signal, combined with entropy, significantly improves accuracy on computed outputs across various model sizes and budgets. The study demonstrates that entropy alone leaves a systematic blind spot that boundary dependence can fill, leading to higher exact‑match rates on challenging computed tasks.

By Nicol\'as Vera Z\'u\~niga
llmsfine-tuning
More like this →
arXiv Computation and Language
Aug 31

Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result

The paper demonstrates that a byte‑level BPE tokenizer can be sliced to create multiple vocabulary sizes from a single trained model, preserving exact logits while reducing deployed weights by 66%. Experiments on 30 models show that while sliced models match the full model numerically, they underperform fixed‑cap specialists by a few percentage points in bits‑per‑byte. Multi‑cap training improves robustness to typographical noise, suggesting benefits from training across multiple granularities rather than from control tokens alone.

By Christos Koutsiaris
llmsragnlpbenchmarks
More like this →
arXiv Machine Learning
Sep 14

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

The paper investigates whether small models distilled from larger ones behave similarly when using byte versus token tokenization. It introduces two methods—Marginalize‑It (approximate) and End‑Of‑Token (exact)—to convert token logits to byte logits, and conducts a large‑scale study on decoder‑only dense transformers ranging from 1 billion to 1 trillion bytes of data. Results show that while token‑based models excel early, byte‑based models eventually surpass them with more compute, achieving higher performance ceilings, greater data efficiency, and lower logit storage costs.

By Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz, Margaret Li, Mike Lewis, Luke Zettlemoyer, Srinivasan Iyer
llmsnlpefficiencybenchmarks
More like this →