arXiv Machine Learning
Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Sep 21
Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass
Finding circuits in language models usually means running many careful interventions. We try something simpler: treat attention as a routing map from one forward pass, keep a small set of routes that...
arXiv AI
Sep 28
FTB Graph: Determining and Validating First-token Broadcasters and Language-Identity Head Circuits in Multilingual Language Models
The paper introduces FTB Graph, a method for mapping the causal circuitry that determines the first-token language identity in multilingual language models. Using Edge Attribution Patching and exact activation patching across six architectures (GPT‑2, BLOOM‑560M, Pythia‑1B/2.8B, Qwen2.5‑1.5B Base/Instruct), the authors extract directed acyclic graphs that reveal deep or mid‑to‑deep broadcasting hubs, with notable differences among models. The study finds that first‑token routing is largely established during pretraining and largely preserved by instruction tuning, while linear gradient approximations can diverge from causal interventions, underscoring the need for exact‑patching verification.
arXiv AI
Jul 22
Circuit Claims Depend on What Is Extracted and How It Is Compared
arXiv:2607. 18921v1 Announce Type: cross Abstract: Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior.
arXiv Machine Learning
Aug 28
Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuit
The paper introduces Circuit Condensation, a post‑training method that prunes low‑attribution edges from large causal graphs and trains a low‑rank adapter to preserve behavior. Across four behaviors and eight models, the condensed circuits are on average 8.1× smaller than the strongest frozen baseline, with reductions up to 316×. Experiments show that weight updates drive the size reduction, and detailed ablations reveal dependencies among remaining edges and a more focused set of heads for indirect object identification.
arXiv AI
Jun 16
Multi-Granular Node Pruning for Causal Circuit Discovery
arXiv:2512. 10903v2 Announce Type: replace Abstract: Circuit discovery aims to identify minimal subnetworks that are responsible for specific behaviors in large language models (LLMs).
arXiv AI
Jun 2
Query Circuits: Explaining How Language Models Answer User Prompts
arXiv:2509. 24808v2 Announce Type: replace Abstract: Explaining why a language model produces a particular output requires local, input-level explanations.