Towards Data Science June 24, 2026 By Subhanga Upadhyay

A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT

Read the original on Towards Data Science →

Activation patching reveals how facts are stored, routed, and read out across transformer layers, and why the residual stream does most of the work The post A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT appeared first on Towards Data Science .

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Towards Data Science.

llms

arXiv Machine Learning

Aug 18

A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

arXiv:2608. 14689v1 Announce Type: new Abstract: Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently.

By Pratikkumar Babariya

llms

arXiv AI

Jun 9

How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing

arXiv:2603. 13259v2 Announce Type: replace-cross Abstract: When a decoder-only transformer is forced to process matched correct and incorrect single-token continuations of a factual query, the two pathways through hidden-state space diverge in a specific way: displacement vectors from the query-only representation maintain approximately equal magnitude but rotate apart in direction.

By Javier Mar\'in

llms nlp

arXiv Machine Learning

Jun 3

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

arXiv:2606. 03780v1 Announce Type: cross Abstract: Causal tracing of factual recall has been studied predominantly in dense transformer language models, where interventions localize information flow to layers or feed-forward modules.

By Yuetian Lu, Ali Modarressi, Yihong Liu, Hinrich Sch\"utze

llms rag

arXiv AI

Jul 21

The Art of Not Forgetting

arXiv:2607. 17944v1 Announce Type: cross Abstract: We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network.

By Ashmith Atmuri, Akshay Kumar, Yashaswini Rao Bhogarajula

llms benchmarks

arXiv AI

Jun 3

FlashbackCL: Mitigating Temporal Forgetting in Federated Learning

arXiv:2606. 03939v1 Announce Type: cross Abstract: Federated Learning (FL) of foundation and edge models increasingly targets deployments where client data distributions drift over time, yet existing forgetting-mitigation methods assume each client's distribution is stationary.

By Mubarak A. Ojewale, Adriana E. Chis, Jorge M. Cortes-Mendoza, Bernardo Pulido-Gaytan, Horacio Gonzalez-Velez

efficiency benchmarks

arXiv AI

Aug 26

A Circuit, Not The Circuit: Non-Unique Causal Localisation of the Mamba-2 State Sink

arXiv:2606.00930v2 Announce Type: replace-cross Abstract: Mechanistic interpretability routinely reads a probe and labels its top-activating units as the circuit executing the computation. We test th...

By Yuhang Jiang, Bowen Zhang

llms benchmarks safety

One email a morning, machine-written

Related stories

A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

The Art of Not Forgetting

FlashbackCL: Mitigating Temporal Forgetting in Federated Learning

A Circuit, Not The Circuit: Non-Unique Causal Localisation of the Mamba-2 State Sink