The paper studies GLU-based neurons in large language models by measuring the cosine similarity between each neuron's input and output weight vectors. A strong negative similarity identifies a "weakening neuron," which tends to appear in late layers, activates frequently, and exerts a large influence on model behavior. The authors also find that weakening neurons significantly affect outputs when gate values are negative, contrary to expectations.
By Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui, Hinrich Sch\"utze
We analyze the learned input-output behavior of GLU-based neurons in large language models (LLMs). We propose a simple analysis method: For each neuron, we compute the cosine similarities between its...
SYNAPSE is a training‑free framework that probes Transformer models by extracting per‑layer [CLS] representations, ranking neurons with a lightweight linear probe, and applying forward‑hook interventions during inference. It enables controlled, repeatable experiments on internal representations without altering the original model, revealing that task‑relevant information is encoded in broad, overlapping neuron subsets and that small weight or logit manipulations can redirect predictions. The framework highlights functional stability, class‑specific specialization, and complementary vulnerability profiles across domains.
By Jes\'us S\'anchez Ochoa, Enrique Tom\'as Mart\'inez Beltr\'an, Alberto Huertas Celdr\'an
The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.
By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
arXiv:2609.07965v1 Announce Type: cross
Abstract: Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev...
By Keren Artiaga, Sabyasachi Kamila, Haithem Afli, Conor Lynch, Mohammed Hasanuzzaman
arXiv:2606. 07414v1 Announce Type: new Abstract: Sparsity allows scaling model parameters without proportionally increasing computational cost.
By Simon Schug