arXiv Machine Learning

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

Matryoshka Attribution (MAttr) is a mask‑learning method that identifies nested subsets of a language model’s internal components by minimizing downstream loss. It uses a differentiable sigmoid top‑k operator and randomizes sparsity during training to produce an attribution ordering of components. MAttr tops the Mechanistic Interpretability Benchmark leaderboard and can be applied via reinforcement learning to pinpoint weight changes that control behaviors such as refusal in Llama 3.1 8B Instruct, where restoring just 1% of weights removes refusals while preserving capabilities.

arXiv AI
Nov 5

From Outcome to Representation: Tracing Reasoning Mechanisms through Integrated Policy Gradient

The paper introduces Integrated Policy Gradient (IPG), a training‑free framework that applies policy‑based attribution to large language model reasoning. IPG backpropagates outcome‑level, non‑differentiable signals through entire inference trajectories, using path integration to isolate outcome‑relevant mediators at the component level. Empirical results show that IPG precisely localizes sparse internal components linked to specific reasoning behaviors and enables modulation of reasoning capabilities.

By Changming Li, Kaixing Zhang, Haoyun Xu, Yingdong Shi, Zheng Zhang, Kaitao Song, Kan Ren
arXiv Computation and Language
Oct 28

Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

The paper introduces Causal Attribution Pruning (CAP), a training‑free technique that identifies critical attention heads in large language models by measuring their causal impact on reasoning tasks. CAP estimates performance loss when each head is masked on a small calibration set, then translates these causal scores into weight‑level importance for pruning. Compared to magnitude‑only or activation‑based methods, CAP achieves up to 61% relative accuracy gains over Wanda on ARC‑Challenge at 20% sparsity and improves performance on GSM8K, StrategyQA, and ARC‑Challenge across Llama‑3‑8B‑Instruct and Mistral‑7B‑Instruct at 10%, 20%, and 50% sparsity.

By Amogh Sheth, Andrew Lin, Biruk Assefa, Yi Wen Huang, Yuhao Ge
arXiv Computation and Language
Nov 4

Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries

Circuit Oracle is an off‑the‑shelf large language model that, using a multi‑agent harness and task‑specific skills, can answer natural‑language questions about a target model by reading its attribution graph via tool calls. The system was evaluated on three safety‑relevant proxy tasks—detecting spurious features in probe circuits, eliciting hidden knowledge from taboo‑finetuned models, and suppression jailbreaking via feature‑level interventions—achieving 78.2 % accuracy on spurious‑feature detection and mixed results on the other tasks. In addition to task scores, Circuit Oracle produces pruned subgraphs and context‑specific feature explanations, which the authors release for further study.

By Shariar Kabir, Hong Kiat Tan, Swastik Agrawal, Sai V R Chereddy, Sriram Balasubramanian
arXiv AI
Oct 26

Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery

The paper introduces TopKLoRA, an interpretable parameter‑efficient fine‑tuning method that activates only the top‑k adapter latents per token. Using this sparse LoRA, the authors fine‑tune large language models on a poisoned dataset, identify and ablate the minimal circuit responsible for a backdoor, and show that the resulting circuits are an order of magnitude smaller than dense LoRA baselines. They also demonstrate how to construct model organisms with localized circuits via gradient‑routing, supporting the development of interpretable‑by‑design architectures.

By Marek Masiak, Andrzej Szablewski, Nicola Cancedda, Constantin Venhoff
arXiv AI
Nov 6

Teach the Way It Thinks: Guiding LLM Post-training Data Engineering with Sparse Autoencoders

The paper introduces SAERL, a method that uses Sparse Autoencoders (SAEs) to guide reinforcement learning data engineering for large language models. By mapping training samples into a feature space of interpretable SAE activations, SAERL selects samples with strong, non‑redundant gradient signals, organizes functional regions from easy to hard, and batches from single activation clusters with moderate mixing. Experiments on mathematical reasoning tasks show consistent performance gains across model scales and RL algorithms, with notable improvements on Qwen2.5-Math-1.5B.

By Yi Jing, Zao Dai, Jinwu Hu, Zijun Yao, Lei Hou, Juanzi Li, Xiaozhi Wang
arXiv AI
Oct 26

Learning to Be Monitored: Causal Bounds and Intervention-Aware Post-Training Recipes for Chain-of-Thought Monitorability

The paper introduces a refined evaluation framework for chain‑of‑thought (CoT) monitoring in large language models, showing that the commonly used minimal‑criterion sensitivity metric overestimates causal detection of targeted behaviors. It establishes upper and lower bounds on causal detection rates and uses these to assess monitorability. The authors then create a dataset of control–intervention task pairs and apply three training methods—off‑policy self‑distillation, reinforcement learning, and on‑policy self‑distillation—to produce model variants that consistently improve CoT monitorability across various configurations.

By Qiang Fu, Xinyue Li, Andre Wibisono, Arman Cohan
arXiv Machine Learning
Aug 31

How Do Linear Probes Emerge? A Circuit-Tracing Framework with Concept-Targeted Attribution

The paper introduces Concept-Targeted Attribution (CTA), a method that trains attribution graphs to explain the emergence of internal concept representations in language models, rather than just the final token prediction. CTA produces probe-specific circuits that reveal which internal computations drive a linear probe’s accuracy, and cross-layer transcoders demonstrate that these graphs contain predictive structure across multiple concept categories. Causal ablations show that probe-targeted and logit-targeted graphs capture distinct mechanisms, with probe-relevant features affecting internal concept scores and logit-relevant features altering generated tokens.

By Vedant Palit, Florent Draye, Terry Jingchen Zhang, Bernhard Sch\"olkopf, Zhijing Jin
arXiv AI
Oct 31

Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization

The paper investigates how to make Mixture-of-Experts (MoE) reward models more interpretable and useful for personalization by focusing on sparsity and diversity regularizers. By keeping the architecture and training data constant, the authors show that conventional MoE reward models learn entangled components that hinder both interpretability and personalization. Introducing a sparse MoE objective reduces entanglement, yielding experts that align with semantic categories and functional strengths, and enables composable, few‑shot personalization with a 25.81‑point improvement over entangled models.

By Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg