arXiv Machine Learning
Aug 4
Feed-Forward Steering in Transformer Residual Dynamics
arXiv:2608. 02071v1 Announce Type: new Abstract: Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere.
Transformer Field Theory: A Response-Theoretic Approach to Mechanistic Interpretability
arXiv:2605. 25225v2 Announce Type: replace-cross Abstract: Mechanistic interpretability often studies Transformer behavior by intervening on internal activations through activation patching, causal tracing, path patching, and steering directions.
Related stories
arXiv Machine Learning
Jul 21
An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
arXiv:2607. 17696v1 Announce Type: cross Abstract: We develop an adjoint-sensitivity framework for positional influence in causal residual Transformers and separate unconditional analytic results from conditional boundary-shape conclusions.
arXiv Machine Learning
Jun 4
Vision Transformer Finetuning Benefits from Non-Smooth Components
arXiv:2602. 06883v3 Announce Type: replace Abstract: The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness.
arXiv Machine Learning
Jul 27
Interior interpretability with attention rollout: contraction and propagation profiles in Transformers
arXiv:2607. 22367v1 Announce Type: new Abstract: Feature-attribution methods assign scores relating input variables to a model's output, but do not by themselves characterize how explicitly defined interaction operators compose across its intermediate layers.
arXiv AI
23h ago
Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
arXiv:2502. 00213v5 Announce Type: replace-cross Abstract: Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam.
arXiv AI
Jul 29
Localized Adaptation Reveals Distinct Learning Signatures in Transformers
arXiv:2607. 25663v1 Announce Type: new Abstract: Transformer adaptation is typically distributed across model depth, even when the intended change is narrow.
arXiv AI
Jun 2
Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures
arXiv:2605. 27458v2 Announce Type: replace-cross Abstract: Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well.
arXiv AI
Jun 16
Adaptive inference and function vectors in deep transformers
arXiv:2606. 16694v1 Announce Type: cross Abstract: Transformers are widely used as a general-purpose substrate for learning complex correlations between a large collection of coupled variables, but their internal mechanisms have remained mysterious.
arXiv AI
Jul 7
FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity
arXiv:2607. 03763v1 Announce Type: cross Abstract: Federated Transformer training increasingly relies on local AdamW, whose adaptive updates can provide much stronger local progress than SGD-based training.
arXiv AI
Jun 10
When Attribution Patching Lies: Diagnosis and a Second-Order Correction
arXiv:2606. 09899v1 Announce Type: cross Abstract: A central goal of mechanistic interpretability is to identify which internal components causally drive a language model's behavior.
arXiv Machine Learning
Jul 22
Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posterior
arXiv:2607. 18804v1 Announce Type: new Abstract: In the \emph{latent posterior model} of transformer behavior, the next-token distribution arises from a posterior over latent predictive models conditioned on the context, mixed to generate continuations.
arXiv Machine Learning
Jul 9
Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation
arXiv:2603. 19742v2 Announce Type: replace Abstract: Understanding the internal mechanisms of transformer-based large language models (LLMs) is crucial for their reliable deployment and effective operation.