Create RUBIX data#
The config#
The config contains all the information needed to run the pipeline. Those are run specfic configurations. Currently we just support Illustris as simulation, but extensions to other simulations (e.g. NIHAO) are planned.
For the config you can choose the following options:
particle_type: load only stars particle (“particle_type”: [“stars”]) or only gas particle (“particle_type”: [“gas”]) or both (“particle_type”: [“stars”,”gas”])
simulation: choose the Illustris simulation (e.g. “simulation”: “TNG50-1”)
snapshot: which time step of the simulation (99 for present day)
save_data_path: set the path to save the downloaded Illustris data
load_galaxy_args - id: define, which Illustris galaxy is downloaded
load_galaxy_args - reuse: if True, if in th esave_data_path directory a file for this galaxy id already exists, the downloading is skipped and the preexisting file is used
subset: only a defined number of stars/gas particles is used and stored for the pipeline. This may be helpful for quick testing
simulation - name: currently only IllustrisTNG is supported
simulation - args - path: where the data is stored and how the file will be named
output_path: where the hdf5 file is stored, which is then the input to the RUBIX pipeline
# NBVAL_SKIP
import os
config = {
"logger": {
"log_level": "DEBUG",
"log_file_path": None,
"format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
},
"data": {
"name": "IllustrisAPI",
"args": {
"api_key": os.environ.get("ILLUSTRIS_API_KEY"),
"particle_type": ["stars","gas"],
"simulation": "TNG50-1",
"snapshot": 99,
"save_data_path": "data",
},
"load_galaxy_args": {
"id": 12,
"reuse": True,
},
"subset": {
"use_subset": True,
"subset_size": 1000,
},
},
"simulation": {
"name": "IllustrisTNG",
"args": {
"path": "data/galaxy-id-12.hdf5",
},
},
"output_path": "output",
}
Convert data#
Convert the Data into Rubix Galaxy HDF5. This will make the call to the IllustrisAPI to download the data, and then convert it into the rubix hdf5 format using the input handler
# NBVAL_SKIP
from rubix.core.data import convert_to_rubix
convert_to_rubix(config)
2025-11-10 17:08:27,957 - rubix - INFO -
___ __ _____ _____ __
/ _ \/ / / / _ )/ _/ |/_/
/ , _/ /_/ / _ |/ /_> <
/_/|_|\____/____/___/_/|_|
2025-11-10 17:08:27,958 - rubix - INFO - Rubix version: 0.0.post482+g02b969039.d20251103
2025-11-10 17:08:27,958 - rubix - INFO - JAX version: 0.7.2
2025-11-10 17:08:27,959 - rubix - INFO - Running on [CpuDevice(id=0)] devices
2025-11-10 17:08:27,959 - rubix - INFO - Loading data from IllustrisAPI
2025-11-10 17:08:27,960 - rubix - INFO - Reusing existing file galaxy-id-12.hdf5. If you want to download the data again, set reuse=False.
2025-11-10 17:08:27,987 - rubix - INFO - Loading data into input handler
2025-11-10 17:08:27,988 - rubix - DEBUG - Loading data from Illustris file..
2025-11-10 17:08:27,989 - rubix - DEBUG - Checking if the fields are present in the file...
2025-11-10 17:08:27,989 - rubix - DEBUG - Keys in the file: <KeysViewHDF5 ['Header', 'PartType4', 'SubhaloData']>
2025-11-10 17:08:27,990 - rubix - DEBUG - Expected fields: ['Header', 'SubhaloData', 'PartType4', 'PartType0']
2025-11-10 17:08:27,990 - rubix - DEBUG - Matching fields: {'PartType4', 'SubhaloData', 'Header'}
2025-11-10 17:08:27,996 - rubix - DEBUG - Found 649384 valid particles out of 649384
2025-11-10 17:08:28,367 - rubix - DEBUG - Converting Stellar Formation Time to Age
2025-11-10 17:08:35,444 - rubix - DEBUG - Converting to Rubix format..
2025-11-10 17:08:35,446 - rubix - DEBUG - Checking if the fields are present in the particle data...
2025-11-10 17:08:35,447 - rubix - DEBUG - Keys in the particle data: dict_keys(['stars'])
2025-11-10 17:08:35,448 - rubix - DEBUG - Expected fields: {'PartType4': 'stars', 'PartType0': 'gas'}
2025-11-10 17:08:35,448 - rubix - DEBUG - Matching fields: {'stars'}
2025-11-10 17:08:35,448 - rubix - DEBUG - Required fields for stars: ['coords', 'mass', 'metallicity', 'velocity', 'age']
2025-11-10 17:08:35,449 - rubix - DEBUG - Available fields in particle_data[stars]: ['coords', 'mass', 'metallicity', 'age', 'velocity']
2025-11-10 17:08:35,449 - rubix - INFO - Rubix file saved at output/rubix_galaxy.h5
2025-11-10 17:08:35,450 - rubix - DEBUG - Creating Rubix file at path: output/rubix_galaxy.h5
2025-11-10 17:08:35,456 - rubix - DEBUG - Converting redshift for galaxy data into
2025-11-10 17:08:35,457 - rubix - DEBUG - Converting center for galaxy data into kpc
2025-11-10 17:08:35,458 - rubix - DEBUG - Converting halfmassrad_stars for galaxy data into kpc
2025-11-10 17:08:35,459 - rubix - DEBUG - Converting coords for particle type stars into kpc
2025-11-10 17:08:35,468 - rubix - DEBUG - Converting mass for particle type stars into Msun
2025-11-10 17:08:35,471 - rubix - DEBUG - Converting metallicity for particle type stars into
2025-11-10 17:08:35,473 - rubix - DEBUG - Converting age for particle type stars into Gyr
2025-11-10 17:08:35,475 - rubix - DEBUG - Converting velocity for particle type stars into km/s
2025-11-10 17:08:35,492 - rubix - INFO - Rubix file saved at output/rubix_galaxy.h5
Converted to Rubix format!
'output'
Load data#
prepare_input loads the hdf5 file that was created and stored with the convert_to_rubix function. It loads the data into the rubixdata format and centers the particles. the rubixdata object has then the attributes stars and gas and both have then attributes with the relevant quantities for each particle. For example, if you want to access the coordinates of the stella rparticles, you can access them via rubixdata.stars.coords
# NBVAL_SKIP
from rubix.core.data import prepare_input
rubixdata = prepare_input(config)
#print, which attributes are available for rubixdata.stars
attr = [attr for attr in dir(rubixdata.stars) if not attr.startswith('__')]
print(attr)
2025-11-10 17:08:35,537 - rubix - INFO - Centering stars particles
2025-11-10 17:08:36,285 - rubix - WARNING - The Subset value is set in config. Using only subset of size 1000 for stars
['age', 'age_unit', 'coords', 'coords_unit', 'datacube', 'mask', 'mass', 'mass_unit', 'metallicity', 'metallicity_unit', 'pixel_assignment', 'spatial_bin_edges', 'spectra', 'tree_flatten', 'tree_unflatten', 'velocity', 'velocity_unit']
To have not to call two individual function to have the data ready to be passed into the pipeline, you can just use the get_rubix_data(config) from the rubix.core.data module
Overview over the hdf5 file structure#
# NBVAL_SKIP
from rubix.utils import print_hdf5_file_structure
print(print_hdf5_file_structure("output/rubix_galaxy.h5"))
File: output/rubix_galaxy.h5
Group: galaxy
Dataset: center (float64) ((3,))
Dataset: halfmassrad_stars (float64) (())
Dataset: redshift (float64) (())
Group: meta
Dataset: BoxSize (float64) (())
Dataset: CutoutID (int64) (())
Dataset: CutoutRequest (object) (())
Dataset: CutoutType (object) (())
Dataset: Git_commit (|S40) (())
Dataset: Git_date (|S29) (())
Dataset: HubbleParam (float64) (())
Dataset: MassTable (float64) ((6,))
Dataset: NumFilesPerSnapshot (int64) (())
Dataset: NumPart_ThisFile (int32) ((6,))
Dataset: Omega0 (float64) (())
Dataset: OmegaBaryon (float64) (())
Dataset: OmegaLambda (float64) (())
Dataset: Redshift (float64) (())
Dataset: SimulationName (object) (())
Dataset: SnapshotNumber (int64) (())
Dataset: Time (float64) (())
Dataset: UnitLength_in_cm (float64) (())
Dataset: UnitMass_in_g (float64) (())
Dataset: UnitVelocity_in_cm_per_s (float64) (())
Group: particles
Group: stars
Dataset: age (float64) ((649384,))
Dataset: coords (float64) ((649384, 3))
Dataset: mass (float64) ((649384,))
Dataset: metallicity (float64) ((649384,))
Dataset: velocity (float64) ((649384, 3))