NoPFS

NoPFS is the Near-optimal Prefetching System, an I/O middleware that optimizes distributed machine learning I/O at scale.

For more details, check our paper from Supercomputing 2021: Clairvoyant Prefetching for Distributed Machine Learning I/O.

@inproceedings{dryden2021clairvoyant,
  title={CLairvoyant Prefetching for Distributed Machine Learning {I/O}},
  author={Nikoli Dryden and Roman B\"{o}hringer and Tal Ben-Nun and Torsten Hoefler},
  booktitle={Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC)},
  year={2021},
  eprint={2101.08734},
  archivePrefix={arXiv},
  primaryClass={cs.DC}
}

---

We are currently preparing a more "production-ready" release, which will be out soon! If you are interested in the original "research" code, look at the old-nopfs branch.