Towards Data Science June 24, 2026 By Subhanga Upadhyay

A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT

Read the original on Towards Data Science →

Activation patching reveals how facts are stored, routed, and read out across transformer layers, and why the residual stream does most of the work The post A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT appeared first on Towards Data Science .

Summary generated by The Flow from the publisher's feed. The full article lives at Towards Data Science.

llms

arXiv Machine Learning

1d ago

A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

arXiv:2608. 14689v1 Announce Type: new Abstract: Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently.

By Pratikkumar Babariya

llms

arXiv AI

Jun 9

How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing

arXiv:2603. 13259v2 Announce Type: replace-cross Abstract: When a decoder-only transformer is forced to process matched correct and incorrect single-token continuations of a factual query, the two pathways through hidden-state space diverge in a specific way: displacement vectors from the query-only representation maintain approximately equal magnitude but rotate apart in direction.

By Javier Mar\'in

llms nlp

arXiv Machine Learning

Jun 3

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

arXiv:2606. 03780v1 Announce Type: cross Abstract: Causal tracing of factual recall has been studied predominantly in dense transformer language models, where interventions localize information flow to layers or feed-forward modules.

By Yuetian Lu, Ali Modarressi, Yihong Liu, Hinrich Sch\"utze

llms rag

arXiv AI

Jul 21

The Art of Not Forgetting

arXiv:2607. 17944v1 Announce Type: cross Abstract: We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network.

By Ashmith Atmuri, Akshay Kumar, Yashaswini Rao Bhogarajula

llms benchmarks

arXiv AI

Jun 3

FlashbackCL: Mitigating Temporal Forgetting in Federated Learning

arXiv:2606. 03939v1 Announce Type: cross Abstract: Federated Learning (FL) of foundation and edge models increasingly targets deployments where client data distributions drift over time, yet existing forgetting-mitigation methods assume each client's distribution is stationary.

By Mubarak A. Ojewale, Adriana E. Chis, Jorge M. Cortes-Mendoza, Bernardo Pulido-Gaytan, Horacio Gonzalez-Velez

efficiency benchmarks

Hugging Face Trending Papers

Jun 2

FlashbackCL: Mitigating Temporal Forgetting in Federated Learning

Federated Learning (FL) of foundation and edge models increasingly targets deployments where client data distributions drift over time, yet existing forgetting-mitigation methods assume each client's distribution is stationary. Flashback, the strongest recent FL method against cross-client (spatial) forgetting, uses monotonically accumulating per-class label counts as a knowledge proxy; this proxy becomes miscalibrated under temporal distribution shift and anchors the global model to an outdated class balance.

efficiency benchmarks

Related stories

A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

The Art of Not Forgetting

FlashbackCL: Mitigating Temporal Forgetting in Federated Learning

FlashbackCL: Mitigating Temporal Forgetting in Federated Learning