We use GPT-4 to automatically write explanations for the behavior of neurons in large language models and to score those explanations. We release a dataset of these (imperfect) explanations and scores for every neuron in GPT-2.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at OpenAI Blog.
arXiv:2601. 22594v2 Announce Type: replace-cross Abstract: The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986).
A longstanding goal of research on interpretable deep learning is to replace opaque neural computations with human-meaningful symbolic descriptions. In this paper, we propose an approach for approximating the behavior of components of deep networks with executable programs.
The paper studies GLU-based neurons in large language models by measuring the cosine similarity between each neuron's input and output weight vectors. A strong negative similarity identifies a "weakening neuron," which tends to appear in late layers, activates frequently, and exerts a large influence on model behavior. The authors also find that weakening neurons significantly affect outputs when gate values are negative, contrary to expectations.
arXiv:2507. 00322v2 Announce Type: replace-cross Abstract: Despite remarkable advances in coding capabilities, language models (LMs) still struggle with simple syntactic tasks such as generating balanced parentheses.
arXiv:2606. 19317v1 Announce Type: cross Abstract: A longstanding goal of research on interpretable deep learning is to replace opaque neural computations with human-meaningful symbolic descriptions.