Public Data Access Overview / Discussion Forum

Hey there! I'm having some issues with the loadSubhalo() function.

import illustris_python as il
basePath='./TNG100-3/output'
snapNum, galid=99, 100
data=il.snapshot.loadSubhalo(basePath, snapNum, galid, 'gas')

returns the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/jacob/B6F04BDBF04BA08B/TNG/illustris_python/snapshot.py", line 196, in loadSubhalo
    return loadSubset(basePath, snapNum, partType, fields, subset=subset)
  File "/media/jacob/B6F04BDBF04BA08B/TNG/illustris_python/snapshot.py", line 54, in loadSubset
    nPart = getNumPart(header)
  File "/media/jacob/B6F04BDBF04BA08B/TNG/illustris_python/snapshot.py", line 26, in getNumPart
    nPart[j] = header['NumPart_Total'][j] | (header['NumPart_Total_HighWord'][j] << 32)
KeyError: 'NumPart_Total'

looking into the function loadSubset I see that it passes "dict(f['Header'].attrs.items())" to getNumPart, with f=h5py.File(snapPath(basePath, snapNum), 'r'). If I do

file=h5py.File(il.snapshot.snapPath(basePath, snapNum), 'r')
header=dict(file['Header'].attrs.items())
header.keys()

I see that this object does not contain the 'NumPart_Total' keyword expected by the function getNumPart().

How do I handle this?

PS: Also, perhaps related, say I wanted to create my own hdf5 file that contains only particles I have decided I am interested in. I would load snapshots/chunks one at a time, take the info on the particles I want, and put them into their own combined hdf5 file that has the same structure as regular TNG files. I'd need to keep the Header consistent as well. How do I edit the header? If not related I can post separately.

I just noticed that the header object also doesn't have 'NumPart_Total_HighWord', which I'd need next

Hi Jacob,

This works fine on the original data files (e.g. if you run these commands on the Lab). You must have different files: where are they from? Did you download just a subset of the data?

What is the output of the command h5dump -g Header snap_099.0.hdf5? For me this produces:

HDF5 "snap_099.0.hdf5" {
GROUP "Header" {
   ATTRIBUTE "BoxSize" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 75000
      }
   }
   ATTRIBUTE "Composition_vector_length" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 0
      }
   }
   ATTRIBUTE "Flag_Cooling" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 1
      }
   }
   ATTRIBUTE "Flag_DoublePrecision" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 0
      }
   }
   ATTRIBUTE "Flag_Feedback" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 1
      }
   }
   ATTRIBUTE "Flag_Metals" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 0
      }
   }
   ATTRIBUTE "Flag_Sfr" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 1
      }
   }
   ATTRIBUTE "Flag_StellarAge" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 0
      }
   }
   ATTRIBUTE "Git_commit" {
      DATATYPE  H5T_STRING {
         STRSIZE 40;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "d203ec8b07c7e2bdda5f608aa0babea46d603699"
      }
   }
   ATTRIBUTE "Git_date" {
      DATATYPE  H5T_STRING {
         STRSIZE 29;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "Thu Apr 7 14:14:27 2016 +0200"
      }
   }
   ATTRIBUTE "HubbleParam" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 0.6774
      }
   }
   ATTRIBUTE "MassTable" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SIMPLE { ( 6 ) / ( 6 ) }
      DATA {
      (0): 0, 0.0323568, 0, 0.00302063, 0, 0
      }
   }
   ATTRIBUTE "NumFilesPerSnapshot" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SCALAR
      DATA {
      (0): 7
      }
   }
   ATTRIBUTE "NumPart_ThisFile" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 6 ) / ( 6 ) }
      DATA {
      (0): 12723085, 13481846, 0, 27420840, 339778, 4381
      }
   }
   ATTRIBUTE "NumPart_Total" {
      DATATYPE  H5T_STD_U32LE
      DATASPACE  SIMPLE { ( 6 ) / ( 6 ) }
      DATA {
      (0): 88935326, 94196375, 0, 188392750, 2239134, 31309
      }
   }
   ATTRIBUTE "NumPart_Total_HighWord" {
      DATATYPE  H5T_STD_U32LE
      DATASPACE  SIMPLE { ( 6 ) / ( 6 ) }
      DATA {
      (0): 0, 0, 0, 0, 0, 0
      }
   }
   ATTRIBUTE "Omega0" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 0.3089
      }
   }
   ATTRIBUTE "OmegaBaryon" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 0.0486
      }
   }
   ATTRIBUTE "OmegaLambda" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 0.6911
      }
   }
   ATTRIBUTE "Redshift" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 2.22045e-16
      }
   }
   ATTRIBUTE "Time" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 1
      }
   }
   ATTRIBUTE "UnitLength_in_cm" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 3.08568e+21
      }
   }
   ATTRIBUTE "UnitMass_in_g" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 1.989e+43
      }
   }
   ATTRIBUTE "UnitVelocity_in_cm_per_s" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SCALAR
      DATA {
      (0): 100000
      }
   }
}
}

where you see the NumPart* entries.

For your second question, the info in the Header is made up of "attributes" (see h5py docs), rather than "datasets" (for large arrays). You can edit/create/delete these in h5py.

So I'm embarrassed to say that after looking into it I was basically loading a groupcat file like a snapshot file-- long story short I didn't realize how many different types of file there were and didn't keep them properly separated.

Thank you for the help!