How to load TNG snapshots in parallel by using Python module "mpi4py"?
Yun Wang
29 Jan '21
Hi,
The elapsed time needed to load the coordinates of dm from snapshot data is approximately 40 minutes if I use only one CPU.
So I want to know whether there are methods to parallel load the snapshot data by using the Python interface of MPI, i.e. mpI4py.
Dylan Nelson
29 Jan '21
Hi Yun Wang,
Certainly yes, assuming you have a parallel filesystem to take advantage of a heavier read load. I personally use a multi-process approach, rather than MPI, but a MPI approach theoretically allows even more performant (multi-node) approaches.
There aren't any public examples of such approaches, but let me know if I can help further.
Hi,
The elapsed time needed to load the coordinates of
dmfrom snapshot data is approximately 40 minutes if I use only one CPU.So I want to know whether there are methods to parallel load the snapshot data by using the Python interface of MPI, i.e. mpI4py.
Hi Yun Wang,
Certainly yes, assuming you have a parallel filesystem to take advantage of a heavier read load. I personally use a multi-process approach, rather than MPI, but a MPI approach theoretically allows even more performant (multi-node) approaches.
There aren't any public examples of such approaches, but let me know if I can help further.