Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts. This workload differs from interactive scientific curation, where mutable records and ad hoc inspection are often more important than random indexed throughput.
arXiv:2506. 01883v3 Announce Type: replace-cross Abstract: Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory.
By Davide D'Ascenzo, Sebastiano Cultrera di Montesano
arXiv:2607. 18187v1 Announce Type: cross Abstract: Large-scale scientific simulations generate volumetric data at rates that far outpace advances in storage and network bandwidth, making effective lossy compression increasingly critical.
By Kaiyuan Tang, Maizhe Yang, Chaoli Wang
The paper introduces NIO Bench, a benchmarking framework that profiles storage I/O for six machine learning model types, using Python hooks and Linux strace to capture detailed access patterns. Experiments on a Ceph-backed Kubernetes cluster show that I/O is dominated by data preparation, model loading, and checkpointing, with training becoming compute-bound once data is staged. The study finds a power‑law distribution of file usage and identifies cache‑miss read tail latency as the main storage bottleneck, recommending aggressive prefetching, page‑cache pinning, and bursty write handling for ML‑optimized storage.
By Jonathan W. Morris, Ionut Mistreanu, Connor Louie
arXiv:2607. 28880v1 Announce Type: cross Abstract: Modern multimedia machine learning workloads increasingly store large-scale datasets in cloud object storage services such as AWS S3.
By Debopam Sanyal, Hongjie Chen, Alexey Tumanov, Joshua Kimball
arXiv:2608. 04942v1 Announce Type: cross Abstract: CheMLFlow is an open-source platform for building and executing end-to-end, high-throughput, and agentic workflows for scientific and technological applications.
By Brendan Smith, Susana Lopez-Moreno, Eric Dolores-Cuenca, Sangil Kim, Jose L. Mendoza-Cortes, Nijamudheen Abdulrahiman