Hugging Face Blog
Streaming datasets: 100x More Efficient
Read the original on Hugging Face Blog →The Flow has not summarised this story yet — read it at Hugging Face Blog.
Related stories
arXiv AI
Jul 2
scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics
arXiv:2506. 01883v3 Announce Type: replace-cross Abstract: Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory.
arXiv Machine Learning
Jul 23
Streaming Sliced Optimal Transport
arXiv:2505. 06835v5 Announce Type: replace Abstract: Sliced optimal transport (SOT), or sliced Wasserstein (SW) distance, is widely recognized for its statistical and computational scalability.
arXiv Machine Learning
Aug 3
LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data
arXiv:2607. 28880v1 Announce Type: cross Abstract: Modern multimedia machine learning workloads increasingly store large-scale datasets in cloud object storage services such as AWS S3.
arXiv Machine Learning
Aug 4
CascadeLUT: Information-Ordered Streaming Inference for Bandwidth-Constrained FPGAs
arXiv:2608. 00720v1 Announce Type: cross Abstract: Mapping neural networks to FPGAs enables low-latency, energy-efficient inference, particularly for lookup table (LUT)-based models that eliminate multipliers and map directly to reconfigurable fabric.
arXiv Machine Learning
Jun 5
Unifying Dataset Pruning and Distillation for Efficient Large-scale Compression
arXiv:2502. 06434v2 Announce Type: replace-cross Abstract: Dataset pruning (DP) and dataset distillation (DD) fundamentally differ in their outputs: DP selects original image subsets, while DD generates synthetic images.
arXiv AI
Aug 3
GPU-Accelerated ANNS: Quantized for Speed, Built for Change
arXiv:2601. 07048v5 Announce Type: replace-cross Abstract: Approximate nearest neighbor search (ANNS) is a core problem in machine learning and information retrieval applications.