From Files to Chunks: Improving HF Storage Efficiency

Introducing Storage Regions on the HF Hub

From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub

Smart Compaction: Predicting Compaction Utility from Lakehouse Table Metadata

arXiv:2608. 08639v1 Announce Type: new Abstract: Open lakehouse table formats accumulate small data files over time, which degrades query performance.

LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data

arXiv:2607. 28880v1 Announce Type: cross Abstract: Modern multimedia machine learning workloads increasingly store large-scale datasets in cloud object storage services such as AWS S3.

Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets

Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts. This workload differs from interactive scientific curation, where mutable records and ad hoc inspection are often more important than random indexed throughput.

Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets

arXiv:2606. 29975v1 Announce Type: new Abstract: Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts.