The study investigates the consistency and specificity of language model circuits across six tasks and five models, focusing on component-level (attention heads and MLP blocks) and neuron-level circuits. Component-level circuits are highly consistent and causally important but lack task specificity, as ablating a circuit for one task similarly harms performance on other tasks. Neuron-level circuits show higher task specificity but lower consistency, with overlap mainly between closely related tasks. The analysis of Llama‑3.2‑3B reveals that shared components are predominantly MLP blocks, while attention heads act as generic attention‑sink heads.
By Michael Li, Nishant Subramani
The study investigates whether large language models (LLMs) can generalize arithmetic reasoning across different input formats, such as numeric and verbal expressions in English, Spanish, and Italian. By using attribution patching, the researchers identified the specific internal circuits each model employs for numeric versus verbal arithmetic tasks. They found that the degree of overlap between a model’s numeric circuit and its verbal circuit predicts how well the model generalizes to verbal formats, explaining variations in performance across languages and formats without requiring labeled data.
By Andrea Gregor de Varda, Sana Pandey, Pengrui Han, Jacob Andreas, Evelina Fedorenko
arXiv:2608. 13567v1 Announce Type: new Abstract: The human brain exhibits a striking degree of functional specialization, with distinct networks supporting language, formal reasoning, reasoning about other minds, and reasoning about the physical world.
By Pengrui Han, Jacob Andreas, Evelina Fedorenko, Andrea Gregor de Varda
arXiv:2607. 05013v1 Announce Type: cross Abstract: Although LLMs have made significant progress in mathematical reasoning, determining whether a mathematical problem is solvable remains a fundamental yet challenging capability.
By Nikolaos Xiros, Maria-Eleni Zoumpoulidi, Georgios Paraskevopoulos
arXiv:2605.09129v2 Announce Type: replace
Abstract: Circuit study aims to explain how language models (LMs) implement a task by localizing and interpreting a circuit: a computational subgraph respons...
By Daking Rai, Mor Geva, Ziyu Yao
arXiv:2607. 17166v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) continue to achieve state-of-the-art performance across various natural language processing tasks.
By Luyu Qiu, Jianing Li, Hwanhee Kim, Xiaoyong Wei, Yueyuan Zheng, Janet Hsiao, Lei Chen
The paper introduces a probe‑free method called the Neuron Separability Index (NSI) to assess how individual neurons in large language models distinguish grammatical from ungrammatical sentences using linguistic minimal pairs. Across 68 linguistic paradigms and seven model checkpoints, the study finds that while raw separability for morphology and syntax peaks early, single‑unit selectivity is sparse and weak, with rare strongly selective "grandmother neurons." Moreover, the research shows a dissociation between whole‑vector linear separability, single‑neuron selectivity, and behavioral competence, and demonstrates that targeted ablations can further separate activation selectivity from causal reliance.
S^3martCirc is a self‑supervised framework that jointly discovers and interprets neural circuits in large language models, rather than treating circuit discovery and functional interpretation as separate stages. It abstracts node behavior into two general computational roles that generalize across tasks and introduces a quantitative metric for assigning these roles, enabling simultaneous identification of importance and function. Experiments demonstrate that S^3martCirc outperforms existing methods in circuit discovery.
By Wendy Zheng, Yinhan He, Liang Wu, Jundong Li
arXiv:2607.20995v2 Announce Type: replace
Abstract: Distinguishing animate from inanimate concepts in written language requires more than shallow text processing, as it involves recognizing complex s...
By Samuele Punzo, Giovanni Cin\`a, Sandro Pezzelle
Although LLMs have made significant progress in mathematical reasoning, determining whether a mathematical problem is solvable remains a fundamental yet challenging capability. While recent studies have probed internal representations of model solvability beliefs, verbalization has primarily been studied behaviorally rather than as an internal representation, limiting its analysis and manipulation.
arXiv:2605. 03058v2 Announce Type: replace-cross Abstract: A central goal of explainable AI is to express large language model (LLM) decision logic symbolically and ground it in internal mechanisms.
By Francesco Sovrano, Gabriele Dominici, Marc Langheinrich
arXiv:2610.33299v1 Announce Type: new
Abstract: To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds...
By Qing Yao, Sasha Boguraev, Tiago Pimentel, Kyle Mahowald