Rethinking Neural Thickets Without Weight Perturbations
Read the original on arXiv AI →The paper revisits the Neural Thickets hypothesis, which claims that pretrained language models contain dense neighborhoods of task‑specific experts that can be accessed via random Gaussian weight perturbations. The authors reproduce the reported gains but demonstrate that similar improvements can be achieved by perturbing only a single decoder block or by keeping the model frozen and adding a random bias to its output logits. They propose the Perturb‑Select‑Vote (PSV) framework, which generates perturbed predictors, selects high‑performing ones on a small validation set, and aggregates their predictions, showing that output diversity, selection, and voting can explain the gains without relying on nearby weight‑space experts. PSV also improves vision tasks under domain shift, especially with logit perturbations, and offers a simple, training‑free alternative to full‑weight perturbation ensembles.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.