The paper introduces Circuit Condensation, a post‑training method that prunes low‑attribution edges from large causal graphs and trains a low‑rank adapter to preserve behavior. Across four behaviors and eight models, the condensed circuits are on average 8.1× smaller than the strongest frozen baseline, with reductions up to 316×. Experiments show that weight updates drive the size reduction, and detailed ablations reveal dependencies among remaining edges and a more focused set of heads for indirect object identification.
arXiv:2607. 18921v1 Announce Type: cross Abstract: Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior.
arXiv:2605. 26343v2 Announce Type: replace Abstract: Mechanistic interpretability seeks to explain a model's behaviour by finding its circuit: the sparse subgraph of the model's computation that is causally responsible for it.
The study investigates the consistency and specificity of language model circuits across six tasks and five models, focusing on component-level (attention heads and MLP blocks) and neuron-level circuits. Component-level circuits are highly consistent and causally important but lack task specificity, as ablating a circuit for one task similarly harms performance on other tasks. Neuron-level circuits show higher task specificity but lower consistency, with overlap mainly between closely related tasks. The analysis of Llama‑3.2‑3B reveals that shared components are predominantly MLP blocks, while attention heads act as generic attention‑sink heads.
The paper investigates whether structural differences in circuits discovered by circuit discovery methods reflect distinct mechanisms. By varying input-token frequency while keeping the task constant, the authors find that although circuits appear specialized by frequency structurally, functional and representational analyses reveal no reliable differences, a phenomenon they call phantom specialization. Across multiple models and tasks, structurally distinct circuits implement the same computation, with core shared subgraphs recovering most of the performance and interchangeable internal representations confirmed by causal interventions.
arXiv:2609.25602v1 Announce Type: new Abstract: In language models, the choice between believing the prompt and believing the weights is made by a handful of identifiable attention heads. Instruction...
arXiv:2606. 16920v1 Announce Type: cross Abstract: Circuit discovery is a key technique in mechanistic interpretability to pinpoint the model components that are crucial for performing a given task.
arXiv:2609.35890v1 Announce Type: new Abstract: Sparse-autoencoder decomposability and concentrated feature attribution are increasingly treated as evidence that a model's computation is easier to re...
The paper investigates mechanistic interpretability, focusing on how automated circuit discovery is evaluated. It shows that the commonly used faithfulness objective can favor circuits that reproduce a model’s behavior poorly, creating an objective-level recovery gap. Experiments on four human-reference tasks and InterpBench reveal that many discovery methods misrank candidate circuits, and that restoring excluded signals can correct most of these misrankings without altering the circuits’ behavior.
arXiv:2606. 05378v1 Announce Type: new Abstract: We test whether a single screen-and-ablate recipe -- identify attention-head circuits by task-pattern selectivity, then verify by causal ablation against a matched-random null -- produces consistent mechanistic claims across model families.
The paper introduces Concept-Targeted Attribution (CTA), a method that trains attribution graphs to explain the emergence of internal concept representations in language models, rather than just the final token prediction. CTA produces probe-specific circuits that reveal which internal computations drive a linear probe’s accuracy, and cross-layer transcoders demonstrate that these graphs contain predictive structure across multiple concept categories. Causal ablations show that probe-targeted and logit-targeted graphs capture distinct mechanisms, with probe-relevant features affecting internal concept scores and logit-relevant features altering generated tokens.
The paper introduces TopKLoRA, an interpretable parameter‑efficient fine‑tuning method that activates only the top‑k adapter latents per token. Using this sparse LoRA, the authors fine‑tune large language models on a poisoned dataset, identify and ablate the minimal circuit responsible for a backdoor, and show that the resulting circuits are an order of magnitude smaller than dense LoRA baselines. They also demonstrate how to construct model organisms with localized circuits via gradient‑routing, supporting the development of interpretable‑by‑design architectures.
arXiv:2605. 23393v2 Announce Type: replace-cross Abstract: Mechanistic interpretability of transformers requires identifying not just which components matter but how they compose into the computational route that produced a prediction.