arXiv AI By Amirhossein Yousefiramandi

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
Oct 1

Unmerge: Efficient Machine Unlearning via Task Arithmetic

The paper introduces Unmerge, an efficient machine unlearning algorithm that treats unlearning as the inverse of task arithmetic. By representing the forget component as a low‑rank basis at each layer, Unmerge optimizes three goals—matching the merged vector, suppressing leakage, and bounding correction size—to limit forget leakage and retain damage. Experiments on ResNet‑50, ViT‑S/16, and Llama‑3.2‑3B show significant performance gains over existing methods while maintaining privacy and feature‑distribution fidelity.

By Haoran Tang, Andrew Tan, Rajiv Khanna
arXiv Machine Learning
Oct 19

Flat Loss, Evolving Features: Grokking in Modular Addition

The paper demonstrates that neural networks can maintain perfect training accuracy while their training cross‑entropy remains nearly flat at a positive level, a phenomenon observed on modular addition tasks. Using ReLU networks with supported‑Polar hidden updates, normalized output‑layer updates, and weight decay, the authors show that hidden features evolve and held‑out accuracy can rise from zero to above 90% even as cross‑entropy stays flat. Theoretical results for one‑hidden‑layer networks and empirical evidence from deeper networks and transformers confirm that correct classification need not coincide with minimal cross‑entropy, challenging conventional assumptions about loss plateaus.

By Akash Kumar, Karthik Sankaranarayanan
arXiv AI
Jul 22

Soft-TransFormers for Continual Learning

arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.

By Haeyong Kang, Chang D. Yoo