Markovian Circuit Tracing (MCT) is a benchmark that tests whether transformer activations encode a compact state abstraction with recoverable transition structure. In experiments with small causal transformers trained on sequences from four‑state hidden Markov models, models achieve near‑optimal validation loss and recover transition matrices accurately under easy and medium observability, but performance collapses to control levels under hard observability. Various controls—including shuffled data, untrained activations, and sparse autoencoders—demonstrate that the recovered transition structure is genuinely causally useful rather than a trivial artifact.
arXiv:2609.30820v1 Announce Type: cross Abstract: Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes...
arXiv:2607. 18921v1 Announce Type: cross Abstract: Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior.
The paper introduces the concept of intervention fidelity in latent world models, measuring whether a model’s open‑loop transitions align with actual environment interventions. Experiments on TD‑MPC2, Cheetah, and DreamerV3 show that high reward fit does not guarantee fidelity, and that self‑supervised models can outperform task‑anchored ones in preserving intervention effects. The authors propose a capture‑gated audit to localize failures and argue that fidelity must be directly audited on the model’s native interface.
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
arXiv:2610.26866v1 Announce Type: new Abstract: Mechanistic studies of large language models localize function in parameters, attention heads or neurons, but those parts do not specify how one infere...
FLIP is a final‑layer inference‑time probe designed to test whether a logit‑facing intervention site in an open‑weight vision‑language model (VLM) supports structured, task‑linked computation rather than generic perturbation. The probe applies elementwise flooring to the final normalized hidden state before logit computation, leaving other model components unchanged. By sweeping intervention strength on a controlled detection/counting task, FLIP identifies three regimes—negligible change, a bounded interior regime with improved detection recall and reduced counting error, and over‑suppression—while a four‑criterion protocol ensures the observed effects are mechanistically interpretable.
arXiv:2607. 00871v1 Announce Type: new Abstract: Self-evolving agents violate the assumption behind most learning-theoretic guarantees: the data, evaluator, components, and hypothesis space are produced by the policy being updated.
arXiv:2607. 11958v1 Announce Type: new Abstract: Under the free energy principle, a predictive system does not observe reality directly; it maintains a generative model of the world and experiences that model's best current hypothesis.
The paper investigates the closed‑loop behavior of a recurrent state‑space model (RSSM) trained on human demonstrations in LunarLander‑v3. Using the model for zero‑shot CEM MPC and actor‑critic training in imagination, the authors find that a model‑based A2C policy trained on a single checkpoint achieves returns comparable to the best model‑free A2C policy while requiring far fewer real transitions. They introduce the Reward Observability Fraction (ROF) to quantify how much of the reward gradient lies in the observable subspace of the latent dynamics, but show that ROF does not predict the collapse of MPC in offline settings, leaving the problem of predicting closed‑loop failure unresolved.
arXiv:2607. 20519v1 Announce Type: new Abstract: Looped Transformers increase test-time computation by repeatedly applying a shared recurrent block.
SCIT (Suffix Cache Interchange Test) is a causal protocol designed to identify which transformer components carry counterfactual computations in latent chain-of-thought models. By constructing exact source‑recipient counterfactuals and applying sufficiency tests, K/V splits, hidden‑state controls, and semantic source controls, SCIT demonstrates that counterfactual arithmetic primarily transfers through value‑cache suffix trajectories rather than hidden states or keys. The method reveals carrier‑regime shifts across different GPT‑2 checkpoints, providing a cache‑level diagnostic and a competence‑gated carrier map for arithmetic mechanisms.
arXiv:2609.37717v1 Announce Type: new Abstract: Decoder-only transformers are trained only through a terminal next-token prediction loss, yet this loss constrains every intermediate hidden state thro...