← Back to all news
arXiv Computation and Language By Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud

Large Language Models with At Most One Spike per Neuron

Read the original on arXiv Computation and Language →

The paper presents a spiking neural network (SNN) approach that uses time-to-first-spike (TTFS) coding to limit each neuron to at most one spike per time window, enabling energy-efficient large language models (LLMs). A reference-based strategy is introduced to encode the four core LLM components—embedding layers, layer normalization, attention-related operations, and dropout—allowing the construction of a fully TTFS-based SNN architecture trained end-to-end. Experiments on BERT and GPT-2 show performance comparable to artificial neural network (ANN) counterparts on natural language understanding and common-sense reasoning, while achieving a 1.5‑billion‑parameter spiking LLM and providing an estimate of spike-related energy consumption.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 12

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

NeuronSpark is a self‑attention‑free language model that uses input‑dependent spiking membrane dynamics, allowing each neuron’s decay rate, gain, and firing threshold to adapt to the current token. This selective spiking approach enables billion‑parameter pretraining from random initialization without relying on backbone transfer, conversion, or distillation, achieving competitive zero‑shot performance and a fast–slow hierarchy. On a 1.1B‑parameter, 2.1B‑token training run, NeuronSpark attains a 40.0 eight‑task zero‑shot average, surpasses comparable‑budget Pythia/Mamba checkpoints, and demonstrates significant energy and cycle reductions on FPGA implementations.

By Zhengzheng Tang
llmsefficiency
More like this →