arXiv:2609.25285v1 Announce Type: cross Abstract: Finding circuits in language models usually means running many careful interventions. We try something simpler: treat attention as a routing map from...
arXiv:2512. 10903v2 Announce Type: replace Abstract: Circuit discovery aims to identify minimal subnetworks that are responsible for specific behaviors in large language models (LLMs).
arXiv:2509. 24808v2 Announce Type: replace Abstract: Explaining why a language model produces a particular output requires local, input-level explanations.
The paper introduces Circuit Condensation, a post‑training method that prunes low‑attribution edges from large causal graphs and trains a low‑rank adapter to preserve behavior. Across four behaviors and eight models, the condensed circuits are on average 8.1× smaller than the strongest frozen baseline, with reductions up to 316×. Experiments show that weight updates drive the size reduction, and detailed ablations reveal dependencies among remaining edges and a more focused set of heads for indirect object identification.
arXiv:2609.36813v1 Announce Type: new Abstract: Large language models (LLMs) exhibit strong general capabilities that mechanistic interpretability has attributed to sparse computational circuits. How...
The paper introduces FTB Graph, a method for mapping the causal circuitry that determines the first-token language identity in multilingual language models. Using Edge Attribution Patching and exact activation patching across six architectures (GPT‑2, BLOOM‑560M, Pythia‑1B/2.8B, Qwen2.5‑1.5B Base/Instruct), the authors extract directed acyclic graphs that reveal deep or mid‑to‑deep broadcasting hubs, with notable differences among models. The study finds that first‑token routing is largely established during pretraining and largely preserved by instruction tuning, while linear gradient approximations can diverge from causal interventions, underscoring the need for exact‑patching verification.
arXiv:2607. 18921v1 Announce Type: cross Abstract: Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior.
arXiv:2607. 06503v1 Announce Type: new Abstract: Large language model (LLM) agents solving multi-step tasks frequently commit to trajectories that are doomed to fail, yet continue to consume substantial inference compute before the failure becomes observable.
arXiv:2605. 24033v2 Announce Type: replace Abstract: Mechanistic interpretability typically discovers circuits and then argues what they do from examples and ablations.
arXiv:2607. 08837v1 Announce Type: cross Abstract: Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers.
arXiv:2605. 26343v2 Announce Type: replace Abstract: Mechanistic interpretability seeks to explain a model's behaviour by finding its circuit: the sparse subgraph of the model's computation that is causally responsible for it.
The paper introduces FlashAttention with Statistics, a method that computes attention‑based detector features without materializing the full attention matrix, thereby reducing memory usage and latency. By accumulating statistics during the FlashAttention pass or via sparse lookups, the approach adds only a small overhead—16% at 2K tokens and 1.4% at 35K tokens on an H100 GPU—and stores just 22.5 KiB per response token. Experiments show that frozen Qwen3 proxy models, even as small as 0.6 B parameters, can serve as strong hallucination detectors, outperforming larger zero‑shot judges and specialized grounding checkers on several benchmarks, with attention features further boosting performance.