arXiv AI
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Encoder Spike Sparsity
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 18
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Activation Sparsity
arXiv:2605. 21333v2 Announce Type: replace-cross Abstract: Natively trained spiking language models must preserve information across time while operating through sparse binary activations, a combination that has produced a persistent quality gap relative to dense Transformers.
arXiv AI
Jun 3
Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs
arXiv:2606. 03026v1 Announce Type: cross Abstract: Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit.
arXiv AI
Oct 12
NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling
NeuronSpark is a self‑attention‑free language model that uses input‑dependent spiking membrane dynamics, allowing each neuron’s decay rate, gain, and firing threshold to adapt to the current token. This selective spiking approach enables billion‑parameter pretraining from random initialization without relying on backbone transfer, conversion, or distillation, achieving competitive zero‑shot performance and a fast–slow hierarchy. On a 1.1B‑parameter, 2.1B‑token training run, NeuronSpark attains a 40.0 eight‑task zero‑shot average, surpasses comparable‑budget Pythia/Mamba checkpoints, and demonstrates significant energy and cycle reductions on FPGA implementations.
arXiv Machine Learning
Oct 21
Compute-Optimal Scaling Laws for Spiking Language Models
arXiv:2610.15146v2 Announce Type: replace Abstract: Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiki...
arXiv Machine Learning
Aug 3
Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformers
arXiv:2601. 22876v2 Announce Type: replace Abstract: Spiking neural networks (SNNs) promise energy-efficient inference for large language models (LLMs), yet most reported savings rely on compute-operation counts that overlook data movement.
arXiv AI
Oct 9
Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models
The paper introduces Spora, a spiking language model that jointly designs spike encodings and attention operators to balance temporal encoding and nonlinear computation. Spora uses binary temporal weights and two encoding schemes—Unipolar Binary Spiking (UBS) and Bipolar Binary Spiking (BBS)—to represent compositional values efficiently, enabling accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora attains a 76.6 GLUE score and 44.1 CoLA MCC, surpassing SpikeLM, and further improvements are achieved with six steps.