The paper introduces a novel patching strategy for byte‑level language models, called novelty patching, which starts a new patch when upcoming bytes are unlikely to repeat recent context, inspired by LZ77 compression. Experiments on a 1.9 GB math corpus show that novelty patching improves final‑answer accuracy from 47 % to 72 % at 12 M parameters and raises computed‑result accuracy from 21 % to 57 % at 50 M parameters, while having limited impact on prose and source code. The study also analyzes patch boundary distributions and reports several failed approaches, such as typed uncertainty and context‑aware novelty prediction. whyItMatters":"The findings demonstrate that a simple, data‑driven patching rule can significantly enhance the performance of byte‑level models on tasks involving computed results, offering a practical improvement for large language models that process raw text without explicit labels."
Byte-level language models such as the Byte Latent Transformer (BLT) run their large global model once per patch and start a patch where a small model's next-byte entropy is high. This rule rarely sta...
The paper investigates how byte‑level language models that use entropy‑based patching (like BLT) fail to compute certain predictable but value‑dependent positions, such as numbers following an equals sign in math solutions. By introducing a boundary‑dependence signal—measuring the model’s loss when a patch start is removed—they show that this signal, combined with entropy, significantly improves accuracy on computed outputs across various model sizes and budgets. The study demonstrates that entropy alone leaves a systematic blind spot that boundary dependence can fill, leading to higher exact‑match rates on challenging computed tasks.
arXiv:2608.20873v2 Announce Type: new Abstract: A deployed language model is a file that many things depend on - a benchmark report, a certification, a fleet of devices - and every way of teaching it...
The paper demonstrates that a byte‑level BPE tokenizer can be sliced to create multiple vocabulary sizes from a single trained model, preserving exact logits while reducing deployed weights by 66%. Experiments on 30 models show that while sliced models match the full model numerically, they underperform fixed‑cap specialists by a few percentage points in bits‑per‑byte. Multi‑cap training improves robustness to typographical noise, suggesting benefits from training across multiple granularities rather than from control tokens alone.
The paper investigates whether small models distilled from larger ones behave similarly when using byte versus token tokenization. It introduces two methods—Marginalize‑It (approximate) and End‑Of‑Token (exact)—to convert token logits to byte logits, and conducts a large‑scale study on decoder‑only dense transformers ranging from 1 billion to 1 trillion bytes of data. Results show that while token‑based models excel early, byte‑based models eventually surpass them with more compute, achieving higher performance ceilings, greater data efficiency, and lower logit storage costs.
arXiv:2611.06598v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) adapts a frozen language model through a compact learned update, making it a natural setting for asking how much information...
Delta N-gram introduces a forward‑only write path into a language model’s native hashed N‑gram table, preserving the pretrained table while adding evidence through a sparse patch Δ. The method uses a 4.1 M‑parameter writer that emits gated targets per event and applies a block‑averaged delta rule to keep each row a convex combination of its old value and the new targets. Experiments on a 0.6 B prototype and Qwen3.8‑Flash‑Next show that the delta writer maintains more document‑specific memory and reduces drift compared to online gradient writes, achieving stable performance over thousands of writes. whyItMatters":"The approach allows continual updates to a language model’s memory without altering its pretrained parameters, enabling efficient, stable, and document‑specific knowledge retention during inference."
The study investigates how post‑training quantization (PTQ) affects proactive interference (PI) in large language models. Using bitsandbytes, the authors compare FP16, INT8, and INT4/NF4 precision across three instruction‑tuned models and find that INT4 quantization markedly degrades accuracy under high interference, with INT8 also incurring a smaller penalty in two of the three models. The degradation is linked to increased same‑key intrusion errors and originates in the quantized transformer backbone rather than the output layer.
ThinkBit is a two‑bit scalar quantization framework designed to preserve reasoning capabilities in language models. It assigns each layer a ternary, 2‑bit, or 3‑bit tier based on sensitivity, then uses a Gumbel‑softmax search to find optimal codewords block‑by‑block. After freezing codewords, it recalibrates shared scales via knowledge‑distillation and performs a brief quantization‑aware training, finally selecting the best checkpoint with paired reasoning tests.
arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This i...
arXiv:2608.20873v3 Announce Type: replace Abstract: A 4-bit quantized weight specifies a rounding cell rather than a single full-precision value. We introduce in-cell learning, a paradigm for writing...
arXiv:2607. 18305v1 Announce Type: cross Abstract: Some limits on what language models know are not gaps in data coverage but structural properties of learning from text.