← Back to all news
arXiv AI By Itai Zehavi, Fanny Jourdan, Ulrich Aivodji

Neuralyzing the Trace: Selective Representation-Level Unlearning with Contrastive Sparse Autoencoders

Read the original on arXiv AI →

The paper introduces SCALPEL, a contrastive sparse autoencoder that selectively learns to unlearn target-specific information from neural network representations. It addresses the energy bias in standard reconstruction-based extractors, which tend to preserve dominant background structure while neglecting low-energy target components. The authors provide theoretical justification that contrastive training enhances target selectivity and demonstrate experimentally that SCALPEL outperforms NMF and standard SAE interventions on the TOFU benchmark, achieving performance comparable to Gradient Difference and RMU methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 16

Test-Time Unlearning via Sparse Autoencoder

arXiv:2609.16229v1 Announce Type: new Abstract: Machine unlearning aims to remove specific knowledge from a trained large language model (LLM) without retraining from scratch. Existing methods modify...

By Pingzhi Li, Jinhao Duan, Vaishnav Tadiparthi, Nakul Agarwal, Kwonjoon Lee, Ehsan Moradi Pari, Hossein Nourkhiz Mahjoub, Sijia Liu, Tianlong Chen
llmsfine-tuningbenchmarkssafety
More like this →
arXiv AI
Sep 10

SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs

The paper introduces Dynamic DAE Guardrails (DSG), a method that uses Dynamic Sparse Autoencoders to perform precision unlearning in large language models. DSG leverages principled feature selection and a dynamic classifier to target activation-based unlearning, outperforming existing gradient‑based methods in terms of computational efficiency, stability, sequential unlearning, resistance to relearning attacks, data efficiency, and interpretability.

By Aashiq Muhamed, Jacopo Bonato, Mona Diab, Virginia Smith
llmssafety
More like this →
arXiv AI
Aug 25

Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality

The paper introduces ADU, a fine‑grained training framework that unlearns sensitive information from large language models by decoupling contextual attention pathways instead of erasing tokens. ADU exploits the distinction between local and global attention heads to identify and suppress attention paths that retrieve persistent sensitive anchors, while preserving local‑attention structure and overall language modeling performance. Evaluation on the TOFU and WMDP benchmarks shows ADU achieves superior forget quality (0.93 on TOFU) and retains 92.9% of model utility compared to 81.9% for existing baselines, with fewer side effects in benign contexts.

By Xunlei Chen, Qirui Ye, Yuang Li, Yi Gong, Zhaokun Wang, Wenyi Li, Shiyao Guo, Jinyu Guo
llmsbenchmarkssafety
More like this →