I am working with a fixed sample of 2,314 FoF halos in TNG300-1 at snapshot 91. For a preregistered analysis step, I need to compute, for each halo, the total dark-matter mass within a fixed comoving sphere of radius 5 cMpc/h centred on its GroupPos:
I need all DM particles spatially inside each sphere, irrespective of their FoF membership.
I have verified that the halo cutout is not sufficient for this purpose: for a test halo (FoF 101, snapshot 91), the DM cutout contains exactly 3,311,403 particles, identical to its GroupLenType DM count. Thus it contains the FoF-associated DM particles rather than all DM particles within the required 5 cMpc/h spatial sphere.
Using the snapshot file endpoint with dm=Coordinates works, but for TNG300-1 snapshot 91 the 600 chunks would require roughly 380 GB of transfer. My current internet connection has a limited monthly data allowance.
Is there a recommended way to perform this calculation directly within the TNG JupyterLab/MPCDF environment, or another efficient method to access particles in arbitrary spatial regions without downloading the full snapshot?
I only need the resulting 2,314 density values, not a local copy of the snapshot.
Thank you very much for your help,
Konrad Seidlitz
Dylan Nelson
30 Oct
Since the statistic you are computing is just a sum, you can easily do this "chunked". That is, you only ever need to load 1% of the particles at a time, and so never run out of memory. You could just do a double for loop (halos, chunks), in the Lab. For chunks you can either go through the usual loading functions, or simply loop over the individual HDF5 files.
Dear Dylan and TNG team,
thank you for pointing me to the forum.
I am working with a fixed sample of 2,314 FoF halos in TNG300-1 at snapshot 91. For a preregistered analysis step, I need to compute, for each halo, the total dark-matter mass within a fixed comoving sphere of radius 5 cMpc/h centred on its GroupPos:
I need all DM particles spatially inside each sphere, irrespective of their FoF membership.
I have verified that the halo cutout is not sufficient for this purpose: for a test halo (FoF 101, snapshot 91), the DM cutout contains exactly 3,311,403 particles, identical to its GroupLenType DM count. Thus it contains the FoF-associated DM particles rather than all DM particles within the required 5 cMpc/h spatial sphere.
Using the snapshot file endpoint with dm=Coordinates works, but for TNG300-1 snapshot 91 the 600 chunks would require roughly 380 GB of transfer. My current internet connection has a limited monthly data allowance.
Is there a recommended way to perform this calculation directly within the TNG JupyterLab/MPCDF environment, or another efficient method to access particles in arbitrary spatial regions without downloading the full snapshot?
I only need the resulting 2,314 density values, not a local copy of the snapshot.
Thank you very much for your help,
Konrad Seidlitz
Since the statistic you are computing is just a sum, you can easily do this "chunked". That is, you only ever need to load 1% of the particles at a time, and so never run out of memory. You could just do a double for loop (halos, chunks), in the Lab. For chunks you can either go through the usual loading functions, or simply loop over the individual HDF5 files.