Public Data Access Overview / Discussion Forum

I'm attempting download the public data to another computer. I'm downloading data specified through my terminal based off of the example scripts in the data access. All the directories are made for the data and the example scripts from bitbucket were also downloaded. I'm also using python.

Now that everything is said and done. I'm testing out retrieval of the data, which is also based of the example scripts.

import illustris_python as il

# Illustris-1 Public Data
basePath1 = 'users/alexander/Data/Illustris_1/groups1_135'
# Illustris-1 Dark Matter Public Data
basePath1DM = 'users/alexander/Data/Illustris_1_Dark/'
# Illustris-1 Hydro to Dark Matter matching Data
basePathMatch = 'users/alexander/Data/Illustris_1_Matching/'


halofields = ['GroupVel','Group_M_Crit200']
subhalofields = ['SubhaloMass']

subhalos = il.groupcat.loadSubhalos(basePath1,135,fields=subhalofields)

Testing out this scripts outputs this traceback.

Traceback (most recent call last):
  File "/Users/alexander/Illustris/test.py", line 14, in <module>
    subhalos = il.groupcat.loadSubhalos(basePath1,135,fields=subhalofields)
  File "/Users/alexander/Illustris/illustris_python/groupcat.py", line 92, in loadSubhalos
    return loadObjects(basePath,snapNum,"Subhalo","subgroups",fields)
  File "/Users/alexander/Illustris/illustris_python/groupcat.py", line 33, in loadObjects
    with h5py.File(gcPath(basePath,snapNum),'r') as f:
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/files.py", line 272, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/files.py", line 92, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2687)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2645)
  File "h5py/h5f.pyx", line 76, in h5py.h5f.open (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/h5f.c:1933)
IOError: Unable to open file (Unable to open file: name = 'users/alexander/data/illustris_1/groups1_135/groups_135/fof_subhalo_tab_135.0.hdf5', errno = 2, error message = 'no such file or directory', flags = 0, o_flags = 0)
[Finished in 0.5s with exit code 1]
[shell_cmd: python -u "/Users/alexander/Illustris/test.py"]
[dir: /Users/alexander/Illustris]
[path: /usr/bin:/bin:/usr/sbin:/sbin]

If i'm reading this correctly, I do not have fof_subhalo_tab_135.0.hdf5 in the directory where basePath1 leads to my downloaded data.

I've tried searching, but i'm having a bit of trouble looking for this missing file both in my directory and the released public data.

If their is any specifications I should mention that I did not include in the post, let me know.

Thank you.

Hi Alexandres,

Can you confirm you still have this problem? We had some filesystem maintenance and issues recently (Nov 18-23), depending on when you were trying to run this.

Thanks,

dylan

Hi Dylan, just re-ran this quick script. This still produced the same error as posted.

Thank you,

Alexander

Hi Alexandres,

The group catalog files you need are groups_135.0.hdf5 through groups_135.7.hdf5, which for Illustris-1 can be downloaded here:

http://www.illustris-project.org/api/Illustris-1/files/groupcat-135/

if those 8 files are in basePath1, do you still get this error?

Hi Dylan,

Yes, I checked and I have these files downloaded in my Illustris_1 directory. Do you think that I should remove these files and re-download these files again?

Hi Alexandres,

Sorry, the issue is simply the path convention. At the top of the example scripts page it describes how paths should be set up (for a hypothetical snapshot 68).

So you should use

basePath1 = 'users/alexander/Data/Illustris_1/'

and you should rename the groups1_135 folder to groups_135.

You are of course welcome to change any of these conventions, just edit the very top of illustris_python/groupcat.py.

Hi Dylan,

I re-downloaded the data just in case and re-routed my directory path. On my mac, it goes as this,

alexander-->Illustris-->Illustris_1-->groups_135

In which groups_135 is my folder that contains the hdf5 files.

So now I reconfigure my script as,

import illustris_python as il
import h5py 

# Illustris-1 Public Data
basePath1 = 'users/alexander/Illustris_1/'

subhalofields = ['SubhaloMass']

subhalos = il.groupcat.loadSubhalos(basePath1,135,fields=subhalofields)

Running this script again through my terminal produces the same traceback error.

IOError: Unable to open file (Unable to open file: name = 'users/alexander/illustris_1//groups_135/fof_subhalo_tab_135.0.hdf5', errno = 2, error message = 'no such file or directory', flags = 0, o_flags = 0)

The function which returns a path to a group catalog file is just:

def gcPath(basePath,snapNum,chunkNum=0):
    """ Return absolute path to a group catalog HDF5 file (modify as needed). """
    gcPath = basePath + '/groups_%03d/' % snapNum
    filePath1 = gcPath + 'groups_%03d.%d.hdf5' % (snapNum, chunkNum)
    filePath2 = gcPath + 'fof_subhalo_tab_%03d.%d.hdf5' % (snapNum, chunkNum)

    if isfile(filePath1):
        return filePath1
    return filePath2

You could simply this and change it to just:

def gcPath(basePath,snapNum,chunkNum=0):
    """ Return absolute path to a group catalog HDF5 file (modify as needed). """
    gcPath = basePath + '/groups_%03d/' % snapNum
    filePath1 = gcPath + 'groups_%03d.%d.hdf5' % (snapNum, chunkNum)

    return filePath1

If you indeed have a file called groups_135.0.hdf5 in the right directory, then I don't think there is anything else.

Perhaps basePath1 = '/users/...'

Thank you for the help Dylan, I greatly appreciate it.

I did what you have recommended, and now I am having

IOError: Unable to open file (Unable to open file: name = 'users/alexander/illustris_1/groups_135/groups_135.0.hdf5', errno = 2, error message = 'no such file or directory', flags = 0, o_flags = 0)

Which is odd since groups_135.0.hdf5 is definitely downloaded in that folder. So it could most likely by how I am setting up my path.

I'm providing a screenshot of my working directory from finder as well as my path, just to clarify any small issues.

basePath1 = '/users/alexander/Illustris_1'

subhalofields = ['SubhaloMass']
subhalos = il.groupcat.loadSubhalos(basePath1,135,fields=subhalofields)

Hey, I see Illustris/Illustris_1 in your screenshot but not your basePath. I think you'll just need to play with python/h5py and your paths until you can load (any) HDF5 file.

I see, I'll play around with it and see what I can get.

Thank you very much regardless, Dylan!