Efficient way to download snapshot data for environment?

Jonathan Diaz
  • 1
  • 12 Jun '17

Thanks for making it super easy to get a snapshot cutout for a particular group. But now I also want the snapshot data for the larger environment, everything within 1 or 2 Mpc of the halo in question - at each redshift. What is the most efficient way to do this?

For now I am using the API to get all FOF IDs within 2 Mpc of the CM of the target group, and then downloading the snapshot cutouts for each of those groups. That allows me to avoid massive downloads, but I'm finding it slow because there are more groups (~2000) than I was expecting to find. Is there a better way?

Dylan Nelson
  • 12 Jun '17

Hi Jonathan,

This is unfortunately the hardest (most expensive) type of request, especially since you want over multiple snapshots.

I assume then you only care about particles in (sub)halos? Your method won't be "particle complete" in the 2 Mpc sphere (e.g. you will miss IGM gas inbetween these halos). If so, then perhaps there is a mass cut to apply, such that you only need to get the N most massive?

Otherwise, unfortunately there isn't any spatial indexing of the snapshot data which can help here. You have to load the full positions from the snapshot to find all the particles in a given sphere. The one saving option may be that you can download only specific fields of snapshots, instead of the entire snapshots (see Task 11 in the API cookbook). This should let you download just the positions + the fields you want in the end, of all particle types you care about.

Jonathan Diaz
  • 13 Jun '17

I'll go with the mass cut for now but for the future you've given me a few other options to think about. Thanks!

  • Page 1 of 1