The paper introduces a novel patching strategy for byte‑level language models, called novelty patching, which starts a new patch when upcoming bytes are unlikely to repeat recent context, inspired by LZ77 compression. Experiments on a 1.9 GB math corpus show that novelty patching improves final‑answer accuracy from 47 % to 72 % at 12 M parameters and raises computed‑result accuracy from 21 % to 57 % at 50 M parameters, while having limited impact on prose and source code. The study also analyzes patch boundary distributions and reports several failed approaches, such as typed uncertainty and context‑aware novelty prediction.
whyItMatters":"The findings demonstrate that a simple, data‑driven patching rule can significantly enhance the performance of byte‑level models on tasks involving computed results, offering a practical improvement for large language models that process raw text without explicit labels."
By Nicol\'as Vera Z\'u\~niga
Byte-level language models such as the Byte Latent Transformer (BLT) run their large global model once per patch and start a patch where a small model's next-byte entropy is high. This rule rarely sta...
The study re‑examines a reported advantage of a routed ternary (1.58‑bit) language model over a full‑precision transformer at 60K parameters. By running controlled experiments with multiple seeds and a fixed training recipe, the authors find that the apparent benefit largely stems from the choice of baseline model shape rather than the ternary architecture itself. While the routed model does outperform other shapes at a larger 130M‑byte budget, its advantage diminishes when a plain gated diagonal‑SSM block is used, and the ternary penalty varies with architecture and quantization details.
By Gautam Veldanda
RaReCache is a framework that allows a large target language model to decode accurately from a KV cache prefilled by a much smaller source model by selectively recomputing only critical token positions. It identifies these positions using a rank disagreement metric that scores tokens based on the energy of their mapped KV in output directions weakly supported by calibration data. Across two model families and five benchmarks, RaReCache achieves up to a 3.04× prefill speedup and significantly reduces prefill latency while maintaining 95–99% of target accuracy with a 30–40% recompute budget.
By Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel
Delta N-gram introduces a forward‑only write path into a language model’s native hashed N‑gram table, preserving the pretrained table while adding evidence through a sparse patch Δ. The method uses a 4.1 M‑parameter writer that emits gated targets per event and applies a block‑averaged delta rule to keep each row a convex combination of its old value and the new targets. Experiments on a 0.6 B prototype and Qwen3.8‑Flash‑Next show that the delta writer maintains more document‑specific memory and reduces drift compared to online gradient writes, achieving stable performance over thousands of writes.
whyItMatters":"The approach allows continual updates to a language model’s memory without altering its pretrained parameters, enabling efficient, stable, and document‑specific knowledge retention during inference."
By Hao Wu
The study measured the impact of a single training example on a GPT‑2 model by running 24 counterfactual experiments. 32 models were trained from scratch on OpenWebText, and at a specific training step a single batch row was replaced with a 194‑token passage under three conditions (fluent prose, fabricated subject, random characters) or left unchanged. Results showed that the passage was learned from one exposure and decayed, with measurable differences in cross‑entropy up to 50 steps after injection but no lasting effect at the final step.
By Zachary Speck, Asa Shepard