arXiv Machine Learning
The Behavioral Rate of Fine-Tuning: Compression as a Measure of Useful Information
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 24
How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning
arXiv:2607. 21351v1 Announce Type: new Abstract: A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it.
arXiv AI
Oct 9
Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression
arXiv:2610. 10552v1 Announce Type: cross Abstract: Comparing parameter-efficient fine-tuning recipes under a single, shared learning rate is a common but flawed practice: when the arms being compared have very different trainable-parameter counts, a shared rate can simultaneously depress the larger arms' means and inflate their variance, manufacturing a large, seemingly multi-seed-significant advantage for the smallest arm that is not a real effect.
Hugging Face Trending Papers
Jul 23
How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning
A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it. We put that assumption on a scale.
arXiv Machine Learning
Jun 9
Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates
arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.
arXiv Machine Learning
Oct 16
Patch where the text is new: a compression view of where byte-level models should spend compute
The paper introduces a novel patching strategy for byte‑level language models, called novelty patching, which starts a new patch when upcoming bytes are unlikely to repeat recent context, inspired by LZ77 compression. Experiments on a 1.9 GB math corpus show that novelty patching improves final‑answer accuracy from 47 % to 72 % at 12 M parameters and raises computed‑result accuracy from 21 % to 57 % at 50 M parameters, while having limited impact on prose and source code. The study also analyzes patch boundary distributions and reports several failed approaches, such as typed uncertainty and context‑aware novelty prediction. whyItMatters":"The findings demonstrate that a simple, data‑driven patching rule can significantly enhance the performance of byte‑level models on tasks involving computed results, offering a practical improvement for large language models that process raw text without explicit labels."
arXiv Machine Learning
Oct 8
Transmission Factors for Lossy Compression of PDE Training Data: Measuring What Reaches a Trained Operator
The paper discusses how operator‑learning benchmarks are stored as full‑precision arrays that have grown to terabyte scale, and curators must decide how coarsely to compress them. It argues that the usual tolerance on reconstruction error of the stored field is measured in the wrong place, because what matters is the accuracy of a model trained on the compressed copy, not the raw field error. The authors propose a new metric: push a compressed field through a model trained at full precision and compare its output to the output on the clean field, which captures the portion of codec error that actually reaches the model’s output. This metric spans more than two orders of magnitude across six PDE families and correctly orders downstream cost where field error does not, reversing many cross‑dataset comparisons.