arXiv Machine Learning
Jul 14
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
arXiv:2607. 11883v1 Announce Type: new Abstract: Compression is fundamental to intelligence.
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization.
Related stories
arXiv Machine Learning
Jul 17
Gibbs randomness-compression proposition
arXiv:2505. 23869v4 Announce Type: replace-cross Abstract: A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a compression process.
arXiv Machine Learning
Aug 4
Understanding Rate-Distortion Performance in Distributed Transformer Inference
arXiv:2601. 22002v5 Announce Type: replace Abstract: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference.
arXiv Machine Learning
Jun 8
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
arXiv:2606. 06888v1 Announce Type: new Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus.
arXiv Machine Learning
Aug 3
The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting
arXiv:2607. 29503v1 Announce Type: new Abstract: While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is.
arXiv Machine Learning
Jun 15
Can Deep Neural Networks Improve Compression of Very Large Scientific Data?
arXiv:2606. 14353v1 Announce Type: new Abstract: Error-bounded lossy compression is a fundamental technique for managing the rapidly growing volumes of scientific data produced by modern simulations and observational instruments.
arXiv AI
Aug 5
On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
arXiv:2602. 01997v3 Announce Type: replace-cross Abstract: Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning.
arXiv Machine Learning
Jul 21
EVOLVE: Efficient Learned Volume Compression with Variable-Rate Encoding on a Cross-Domain Database
arXiv:2607. 18187v1 Announce Type: cross Abstract: Large-scale scientific simulations generate volumetric data at rates that far outpace advances in storage and network bandwidth, making effective lossy compression increasingly critical.
Hugging Face Trending Papers
Jul 28
Bridging Compute- and Data-Optimal Pretraining
Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound.
arXiv AI
Jul 29
Bridging Compute- and Data-Optimal Pretraining
arXiv:2607. 25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data.
Hugging Face Trending Papers
Jun 22
Hybrid Compression: Integrating Pruning and Quantization for Optimized Neural Networks
Deep neural networks have witnessed remarkable advancements in recent years and have become integral to various applications. However, alongside these developments, training and deployment of neural network models on embedding and edge devices face significant challenges due to limited memory and computational resources.
arXiv AI
Jul 14
Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
arXiv:2607. 09693v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals.