How to load TNG snapshots in parallel by using Python module "mpi4py"?

Yun Wang
  • 29 Jan '21


The elapsed time needed to load the coordinates of dm from snapshot data is approximately 40 minutes if I use only one CPU.
So I want to know whether there are methods to parallel load the snapshot data by using the Python interface of MPI, i.e. mpI4py.

Dylan Nelson
  • 29 Jan '21

Hi Yun Wang,

Certainly yes, assuming you have a parallel filesystem to take advantage of a heavier read load. I personally use a multi-process approach, rather than MPI, but a MPI approach theoretically allows even more performant (multi-node) approaches.

There aren't any public examples of such approaches, but let me know if I can help further.

  • Page 1 of 1