arXiv Machine Learning

A Feature-Major Codebook for Memory-Efficient Sparse-Binary Self-Organizing Maps: Scaling a MEDLINE Atlas to 1.05 Million Neurons on a Single Consumer GPU

The paper presents a memory‑efficient sparse‑binary self‑organising map (SOM) that scales a MEDLINE atlas to over a million neurons on a single consumer GPU. By re‑ordering the codebook into a feature‑major layout, the authors accelerate the best‑matching‑unit search by 4.5–8.5× without increasing quantisation error, enabling training of a 1,048,576‑neuron SOM in 72 s on a 24 GB GPU. The approach outperforms existing cuSPARSE and CPU‑based SOM implementations, achieving the largest SOM reported to date and demonstrating that resolution limits are computational rather than data‑driven.

arXiv Machine Learning
Sep 7

From 80x to 385x: A Best-Matching-Unit Search at the L2 Roof, Measured Against a Symmetrically Tuned Baseline

The paper reports a comprehensive tuning of both a novel sparse self‑organizing map algorithm (SparseBin) and its baseline cuSPARSE implementation. By optimizing four key levers—tile size, tile‑membership clustering, neuron‑axis chunking, and vectorised loads—the authors achieved a 5.6‑10.1× speed‑up per epoch for map sizes ranging from 32×32 to 512×512, and increased the performance margin over the CUDA baseline from ~80× to ~385×. The tuned kernel saturated the L2 bandwidth at 77% of peak, indicating that further performance gains are unlikely without new hardware or fundamentally different approaches.

By Andrew James Amos
arXiv AI
Aug 18

Static Pruning Across Sparse Retrieval Regimes: What Transfers, What Breaks, and What Still Helps

arXiv:2608. 16309v1 Announce Type: cross Abstract: Static pruning is widely used to accelerate sparse neural retrieval, yet existing studies each validate their conclusions within a single custom pipeline, leaving it unclear which findings transfer to modern engines with different index organizations and dynamic pruning mechanisms.

By Zirui Song, Yuye Zhu, Yang Yang
arXiv Machine Learning
Oct 19

Peak-Memory Schedules You Can Trust: Exact Realization, Certified Optimality, and the Limits of Recomputation for Inference

The paper presents a method for reliably scheduling operator execution in real‑time inference on small devices, achieving exact peak‑memory predictions and certified optimality. By profiling the generated program and modeling its memory usage, the authors can permute statements to match the modeled peak, guaranteeing bitwise‑identical results on CPUs and characterizing GPU limitations. They also provide efficient lower‑bound traps and analyze selective recomputation, showing it is NP‑hard in general but can reduce peak memory on certain architectures.

By Ivan Fomin, Ivan Koryabkin, Boris Sheludko
arXiv AI
Jun 12

MiniMax Sparse Attention

arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.

By Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao
arXiv Machine Learning
Oct 9

PageWeaver: KV-Guided Query Unions for Sparse Attention

PageWeaver introduces a KV‑guided query union strategy for sparse attention, assembling query groups based on selected‑page affinity while preserving each query’s support and output ownership. The design employs a bounded GPU search to generate query IDs and an ID‑aware two‑CTA kernel that consumes them without materializing reordered Q tensors or cross‑page partial outputs. Experiments on the H200 Union8 with FP8 KV show a 1.70× speedup over FlashInfer, with additional latency reductions of 3.26–7.66% on selected captures and modest throughput gains for whole‑model prefill.

By Zhiyuan Li, Zihan Li, Zefang Yuan, Lei Wang, Hao Wang
arXiv Machine Learning
Oct 19

Memory Bandwidth, Not FLOPs: Profiling and Accelerating Query-Based Segmentation Decoders on Edge GPUs

The paper shows that query‑based segmentation decoders on edge GPUs are limited by memory bandwidth rather than FLOPs. Profiling reveals that non‑tensor‑core operations consume a large fraction of decoder time and bytes, saturating the DRAM bus. CompactInst mitigates this by using parameter‑free spatial pooling and a lightweight refinement head, achieving significant speedups on Jetson Orin Nano while maintaining accuracy.

By Oliver Busemann (Osnabr\"uck University, CLAAS E-Systems GmbH), Torben T\"oniges (CLAAS E-Systems GmbH), Martin Atzmueller (Semantic Information Systems Group, Osnabr\"uck University, German Research Center for Artificial Intelligence)
arXiv AI
Nov 6

Enhancing Local and Global Structure in Compressed AI Agent Memory

The paper introduces locality‑aware quantization (LAQ), a new compression technique for AI agent memory that preserves both local and global structure of embedded text. LAQ improves local fidelity (L) and, more importantly, raises global fidelity (G) across all tested runs, outperforming seven other quantizers while maintaining search efficiency and evidence recall. The method is validated on large‑scale datasets, including a million passages and external FinDER memory.

By Henry Han
arXiv Machine Learning
Oct 28

BellowsKV: One Latent for Many Layers, Expanded on Demand

BellowsKV introduces a training‑free approach that maps the hidden states of multiple adjacent layers in a large language model to a single shared latent representation per token, allowing each layer to retrieve its own keys and values via a layer‑specific projection. The method retains the latent at full dimension and offers two compression strategies: BellowsKV‑Quant, which stores the latent at low precision, and BellowsKV‑Evict, which makes a shared eviction decision per block. Experiments on Llama‑3.1‑8B, Mistral‑7B, and Qwen2.5‑7B show that BellowsKV‑Quant preserves about 99% of full‑cache accuracy at 3.6× compression and 97‑98% at 13‑14×, outperforming existing cross‑layer methods that drop to at most 84% accuracy at 16× compression.

By Jiamu Zhang, Hanjie Chen