arXiv AI
Jul 24
Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
arXiv:2607. 20538v1 Announce Type: cross Abstract: Long-context Transformer inference increasingly relies on KV-cache compression or quantization.
Understanding Rate-Distortion Performance in Distributed Transformer Inference
arXiv:2601. 22002v5 Announce Type: replace Abstract: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference.
Related stories
arXiv Machine Learning
Jul 14
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
arXiv:2607. 11883v1 Announce Type: new Abstract: Compression is fundamental to intelligence.
arXiv Machine Learning
Jul 17
Gibbs randomness-compression proposition
arXiv:2505. 23869v4 Announce Type: replace-cross Abstract: A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a compression process.
arXiv Machine Learning
Jun 2
Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition
arXiv:2411. 09816v5 Announce Type: replace Abstract: Large neural networks achieve state-of-the-art performance on many tasks, yet their sheer size hinders deployment on resource-constrained devices.
Hugging Face Trending Papers
Jul 13
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization.
arXiv AI
Aug 11
Full-bandwidth transformer
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
arXiv AI
Jul 24
Scaling Interpretable Transformers with Parity Bottleneck Layers
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
arXiv AI
Jun 18
The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior
arXiv:2604. 13082v2 Announce Type: replace-cross Abstract: Grokking in transformers trained on algorithmic tasks is characterized by a long delay between training-set fit and abrupt generalization, but the source of that delay remains poorly understood.
arXiv Machine Learning
Aug 4
UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction
arXiv:2608. 01298v1 Announce Type: cross Abstract: Diffusion Transformers (DiTs) have emerged as a core architecture in generative modeling due to their scalability and adaptability to multimodal tasks.
arXiv Machine Learning
Jul 14
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
arXiv:2607. 10034v1 Announce Type: new Abstract: Large language models (LLMs) store factual knowledge in their parameters.
arXiv Machine Learning
Jun 16
Descriptive versus Regulatory Uncertainty in Bounded Predictive Systems
arXiv:2605. 18909v2 Announce Type: replace Abstract: Any system that models the world under finite representational capacity must compress; any compression entails a prior; and the prior is the system's bias.
arXiv AI
Jul 7
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
arXiv:2607. 02964v1 Announce Type: cross Abstract: A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does.