Create RUBIX data

Create RUBIX data#

The config#

The config contains all the information needed to run the pipeline. Those are run specfic configurations. Currently we just support Illustris as simulation, but extensions to other simulations (e.g. NIHAO) are planned.

For the config you can choose the following options:

  • particle_type: load only stars particle (“particle_type”: [“stars”]) or only gas particle (“particle_type”: [“gas”]) or both (“particle_type”: [“stars”,”gas”])

  • simulation: choose the Illustris simulation (e.g. “simulation”: “TNG50-1”)

  • snapshot: which time step of the simulation (99 for present day)

  • save_data_path: set the path to save the downloaded Illustris data

  • load_galaxy_args - id: define, which Illustris galaxy is downloaded

  • load_galaxy_args - reuse: if True, if in th esave_data_path directory a file for this galaxy id already exists, the downloading is skipped and the preexisting file is used

  • subset: only a defined number of stars/gas particles is used and stored for the pipeline. This may be helpful for quick testing

  • simulation - name: currently only IllustrisTNG is supported

  • simulation - args - path: where the data is stored and how the file will be named

  • output_path: where the hdf5 file is stored, which is then the input to the RUBIX pipeline

# NBVAL_SKIP
import os

config = {
    "logger": {
        "log_level": "DEBUG",
        "log_file_path": None,
        "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    },
    "data": {
        "name": "IllustrisAPI",
        "args": {
            "api_key": os.environ.get("ILLUSTRIS_API_KEY"),
            "particle_type": ["stars","gas"],
            "simulation": "TNG50-1",
            "snapshot": 99,
            "save_data_path": "data",
        },
        
        "load_galaxy_args": {
        "id": 12,
        "reuse": True,
        },

        "subset": {
            "use_subset": True,
            "subset_size": 1000,
        },
    },
    "simulation": {
        "name": "IllustrisTNG",
        "args": {
            "path": "data/galaxy-id-12.hdf5",
        },
    },
    "output_path": "output",

        
}

Convert data#

Convert the Data into Rubix Galaxy HDF5. This will make the call to the IllustrisAPI to download the data, and then convert it into the rubix hdf5 format using the input handler

# NBVAL_SKIP
from rubix.core.data import convert_to_rubix

convert_to_rubix(config)

2024-11-18 13:18:45,837 - rubix - INFO - 
   ___  __  _____  _____  __
  / _ \/ / / / _ )/  _/ |/_/
 / , _/ /_/ / _  |/ /_>  <
/_/|_|\____/____/___/_/|_|

2024-11-18 13:18:45,838 - rubix - INFO - Rubix version: 0.0.post134+g6444210.d20241106

2024-11-18 13:18:45,838 - rubix - INFO - Rubix galaxy file already exists, skipping conversion

'output'

Load data#

prepare_input loads the hdf5 file that was created and stored with the convert_to_rubix function. It loads the data into the rubixdata format and centers the particles. the rubixdata object has then the attributes stars and gas and both have then attributes with the relevant quantities for each particle. For example, if you want to access the coordinates of the stella rparticles, you can access them via rubixdata.stars.coords

# NBVAL_SKIP
from rubix.core.data import prepare_input

rubixdata = prepare_input(config)

#print, which attributes are available for rubixdata.stars
attr = [attr for attr in dir(rubixdata.stars) if not attr.startswith('__')]
print(attr)

2024-11-18 13:18:46,071 - rubix - INFO - Centering stars particles

2024-11-18 13:18:48,176 - rubix - WARNING - The Subset value is set in config. Using only subset of size 1000 for stars

2024-11-18 13:18:48,288 - rubix - INFO - Centering gas particles

2024-11-18 13:18:49,456 - rubix - WARNING - The Subset value is set in config. Using only subset of size 1000 for gas

['age', 'coords', 'datacube', 'mask', 'mass', 'metallicity', 'pixel_assignment', 'spatial_bin_edges', 'spectra', 'tree_flatten', 'tree_unflatten', 'velocity']

To have not to call two individual function to have the data ready to be passed into the pipeline, you can just use the get_rubix_data(config) from the rubix.core.data module

Overview over the hdf5 file structure#

# NBVAL_SKIP
from rubix.utils import print_hdf5_file_structure

print(print_hdf5_file_structure("output/rubix_galaxy.h5"))

File: output/rubix_galaxy.h5
Group: galaxy
    Dataset: center (float64) ((3,))
    Dataset: halfmassrad_stars (float64) (())
    Dataset: redshift (float64) (())
Group: meta
    Dataset: BoxSize (float64) (())
    Dataset: CutoutID (int64) (())
    Dataset: CutoutRequest (object) (())
    Dataset: CutoutType (object) (())
    Dataset: Git_commit (|S40) (())
    Dataset: Git_date (|S29) (())
    Dataset: HubbleParam (float64) (())
    Dataset: MassTable (float64) ((6,))
    Dataset: NumFilesPerSnapshot (int64) (())
    Dataset: NumPart_ThisFile (int32) ((6,))
    Dataset: Omega0 (float64) (())
    Dataset: OmegaBaryon (float64) (())
    Dataset: OmegaLambda (float64) (())
    Dataset: Redshift (float64) (())
    Dataset: SimulationName (object) (())
    Dataset: SnapshotNumber (int64) (())
    Dataset: Time (float64) (())
    Dataset: UnitLength_in_cm (float64) (())
    Dataset: UnitMass_in_g (float64) (())
    Dataset: UnitVelocity_in_cm_per_s (float64) (())
Group: particles
    Group: gas
        Dataset: coords (float64) ((13254, 3))
        Dataset: density (float32) ((13254,))
        Dataset: electron_abundance (float32) ((13254,))
        Dataset: internal_energy (float32) ((13254,))
        Dataset: mass (float64) ((13254,))
        Dataset: metallicity (float32) ((13254,))
        Dataset: sfr (float32) ((13254,))
        Dataset: velocity (float32) ((13254, 3))
    Group: stars
        Dataset: age (float64) ((649384,))
        Dataset: coords (float64) ((649384, 3))
        Dataset: mass (float64) ((649384,))
        Dataset: metallicity (float32) ((649384,))
        Dataset: velocity (float32) ((649384, 3))
Contents