← Back to all news
arXiv Machine Learning By Areeb Ahmad, Pratinav Seth, Vinay Kumar Sankarapu

Every Ablation Is a Dose: Counterweights and the Semblance of Self-Repair

Read the original on arXiv Machine Learning →

The paper investigates the phenomenon of self‑repair in language models, proposing that it arises from a pre‑existing gain in components that act as counterweights when a component is ablated. By modeling interventions as points on a counterfactual axis, the authors derive an affine law for the causal repair response of fine‑grained units, showing that most downstream directions across several models follow this law. They further demonstrate that the slope of this law can be predicted from fixed weights, suggesting that self‑repair is a predictable, counterweight‑driven response rather than a noisy, unexplained effect.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 5

Sensitivity, Causality, and Repair Dissociate: A Layer-Wise Analysis of Perturbation Robustness and Its Scaling

arXiv:2608. 03842v1 Announce Type: cross Abstract: When a language model fails on surface-perturbed input (typos, OCR noise, homophones), "which layer is responsible" has three natural operationalizations: where representations diverge most (sensitivity), where restoring clean activations recovers the prediction (causality), and where a small adapter can repair the damage (compensatory capacity) - and we show these three layer maps dissociate.

By Nathan Labiosa, David Buff, Ena Nayak, Erica Donno
llmscomputer-vision
More like this →
arXiv Computation and Language
Sep 25

What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetition

The paper investigates neural text degeneration by measuring the fixed‑point structure of short‑window argmax maps across 17 pretrained models, using 96 random two‑token starts without prompts. It finds a stable four‑way classification that varies across model families and scales, with some models funneling to a single endpoint token while others do not, and shows that this behavior is not solely determined by training data or corpus frequency. The study demonstrates that repetition phenomena are not uniformly explained by either training data or network architecture alone, highlighting the complexity of neural text generation dynamics.

By Nicol\'as Vera Z\'u\~niga
llms
More like this →