The paper demonstrates that a fixed‑weight transformer can exactly execute one update step of a standard continuous causal discovery algorithm, and by stacking such blocks it reproduces the algorithm’s optimization trajectory. The transformer maintains both the current graph and the algorithm’s multiplier, which is shown to be essential for accurate updates. Experiments confirm that the constructed block matches the reference solver to floating‑point precision and that arithmetic replay on synthetic and benchmark networks inherits the solver’s performance, while ordinary attention models fail to reliably execute the update or scale to larger graphs.
arXiv:2606. 17516v1 Announce Type: cross Abstract: Causal discovery from observational data remains challenging due to the need to recover directed structure and latent confounding without interventions.
arXiv:2610.16489v1 Announce Type: cross Abstract: Reinforcement learning (RL) can substantially improve the reasoning capabilities of large language models, but an effective training run requires con...
arXiv:2602. 08629v2 Announce Type: replace Abstract: Causal discovery is essential for advancing data-driven fields such as scientific AI and data analysis, yet existing approaches face significant time- and space-efficiency bottlenecks when scaling to large graphs.
arXiv:2601. 15158v4 Announce Type: replace-cross Abstract: Transformers trained via Reinforcement Learning (RL) with outcome-based supervision can spontaneously develop the ability to generate intermediate reasoning steps (Chain-of-Thought).
arXiv:2609.33149v2 Announce Type: replace Abstract: A common principle of effective learning is to practice material that is neither already mastered nor too difficult to permit progress. We ask how...
Matryoshka Attribution (MAttr) is a mask‑learning method that identifies nested subsets of a language model’s internal components by minimizing downstream loss. It uses a differentiable sigmoid top‑k operator and randomizes sparsity during training to produce an attribution ordering of components. MAttr tops the Mechanistic Interpretability Benchmark leaderboard and can be applied via reinforcement learning to pinpoint weight changes that control behaviors such as refusal in Llama 3.1 8B Instruct, where restoring just 1% of weights removes refusals while preserving capabilities.
arXiv:2609. 39892v1 Announce Type: new Abstract: Looped Transformers repeatedly apply the same set of Transformer layers, giving them a recurrent architecture for latent computation.
arXiv:2609.39967v1 Announce Type: cross Abstract: Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few...
The paper introduces the Counterfactual Graph Neural Process (CFGNP), an end‑to‑end method that directly answers counterfactual queries from observational data without first estimating a causal graph. CFGNP employs Graph Neural Networks to leverage a dual‑graph structure, limiting cross‑world message passing to matched factual–counterfactual variables. Experiments on synthetic and image‑based datasets, including the German Loan benchmark, show that CFGNP outperforms existing methods, reducing mean‑squared error from $4.51 imes10^{-2}$ to $0.19 imes10^{-2}$, and generalizes from 10‑node to 30‑node graphs without retraining.
The paper introduces recirculation, an inference‑time architectural enhancement for foundation models that reduces perplexity and improves accuracy on generation and reasoning tasks without adding significant latency. Recirculation adds a specific form of recurrence, enabling the model to function as a dynamical system that tracks belief states, and is distinct from chain‑of‑thought or depth‑recurrence methods. An adaptive variant requires minimal hyperparameter tuning and achieves notable gains on the Gemma3 family, including a 23% perplexity drop and a 21% accuracy increase on GSM8k.
The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.
arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.