arXiv:2607. 21351v1 Announce Type: new Abstract: A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it.
arXiv:2610. 10552v1 Announce Type: cross Abstract: Comparing parameter-efficient fine-tuning recipes under a single, shared learning rate is a common but flawed practice: when the arms being compared have very different trainable-parameter counts, a shared rate can simultaneously depress the larger arms' means and inflate their variance, manufacturing a large, seemingly multi-seed-significant advantage for the smallest arm that is not a real effect.
A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it. We put that assumption on a scale.
arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.
The paper introduces a novel patching strategy for byte‑level language models, called novelty patching, which starts a new patch when upcoming bytes are unlikely to repeat recent context, inspired by LZ77 compression. Experiments on a 1.9 GB math corpus show that novelty patching improves final‑answer accuracy from 47 % to 72 % at 12 M parameters and raises computed‑result accuracy from 21 % to 57 % at 50 M parameters, while having limited impact on prose and source code. The study also analyzes patch boundary distributions and reports several failed approaches, such as typed uncertainty and context‑aware novelty prediction. whyItMatters":"The findings demonstrate that a simple, data‑driven patching rule can significantly enhance the performance of byte‑level models on tasks involving computed results, offering a practical improvement for large language models that process raw text without explicit labels."
The paper discusses how operator‑learning benchmarks are stored as full‑precision arrays that have grown to terabyte scale, and curators must decide how coarsely to compress them. It argues that the usual tolerance on reconstruction error of the stored field is measured in the wrong place, because what matters is the accuracy of a model trained on the compressed copy, not the raw field error. The authors propose a new metric: push a compressed field through a model trained at full precision and compare its output to the output on the clean field, which captures the portion of codec error that actually reaches the model’s output. This metric spans more than two orders of magnitude across six PDE families and correctly orders downstream cost where field error does not, reversing many cross‑dataset comparisons.
The paper introduces a label‑free mixed‑precision post‑training quantization technique for text embedders called Quantize by Drift. It uses quantization‑induced representation drift—measured by re‑encoding a corpus after quantizing a module—to estimate module sensitivity without relevance labels. Across five development embedders, drift‑based ordering of mixed‑precision plans correlates strongly with held‑out retrieval quality (macro Spearman 0.911) and outperforms uniform precision under a hard byte budget, though it does not fully replace fine‑grained redistribution strategies.
Byte-level language models such as the Byte Latent Transformer (BLT) run their large global model once per patch and start a patch where a small model's next-byte entropy is high. This rule rarely sta...
arXiv:2606. 27785v1 Announce Type: cross Abstract: Training-free compression methods for large language models (LLMs) often use calibration data to guide compression decisions.
arXiv:2609.01244v1 Announce Type: new Abstract: Every supervised fine-tuning run forces the same chain of decisions, such as learning rate, batch size, LoRA or full fine-tuning, how many epochs, whic...
arXiv:2602. 08686v3 Announce Type: replace-cross Abstract: Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction.
arXiv:2610.22438v1 Announce Type: new Abstract: Studies of what fine-tuning does to a language model often summarise the weight update $\Delta W$ by its rank. We compare the rank measures used for th...