← Back to all news
arXiv AI By Pawe{\l} Blicharz, Mi{\l}osz Grunwald

A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID

Read the original on arXiv AI →

The study investigates which neurons in a frozen BERT-base-uncased encoder support AI‑text detection using the RAID benchmark across six generators. By applying an L1‑to‑L2 sparse‑probing protocol to all 9,216 CLS hidden‑state dimensions, the authors identify a stable set of fewer than 1% of neurons per generator that largely preserves detection accuracy. Bidirectional activation patching confirms the causal relevance of this set, while mean‑ablating the neurons shows the signal is redundantly distributed, and cross‑generator analysis reveals a bipartite structure with instruction‑tuned generators concentrating more stable neurons in the final layer. "whyItMatters":"The findings demonstrate that a small, stable subset of BERT neurons can reliably support AI‑text detection across diverse generators, enabling efficient detector design without re‑identifying neurons for each new generator."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Aug 27

The Von-Neumann State-Space Transformer for neural decoding

The paper introduces the Von‑Neumann State‑Space Transformer (VN‑SST), a memory‑augmented Transformer that replaces the standard feed‑forward block with a low‑rank instruction bank. By decoding token‑specific operators from a low‑dimensional state‑space memory, VN‑SST achieves higher data‑efficiency and parameter‑efficiency on motor‑cortex neural‑decoding tasks and on small language‑model benchmarks. The model demonstrates that a compact instruction set can act as a control channel, improving performance without increasing accuracy through larger parameter counts.

By Morteza Sarafyazd
llmsbenchmarks
More like this →
arXiv Computation and Language
Sep 7

Large Language Models with At Most One Spike per Neuron

The paper presents a spiking neural network (SNN) approach that uses time-to-first-spike (TTFS) coding to limit each neuron to at most one spike per time window, enabling energy-efficient large language models (LLMs). A reference-based strategy is introduced to encode the four core LLM components—embedding layers, layer normalization, attention-related operations, and dropout—allowing the construction of a fully TTFS-based SNN architecture trained end-to-end. Experiments on BERT and GPT-2 show performance comparable to artificial neural network (ANN) counterparts on natural language understanding and common-sense reasoning, while achieving a 1.5‑billion‑parameter spiking LLM and providing an estimate of spike-related energy consumption.

By Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud
llmsrag
More like this →