This repo contains python scripts designed to get the required metadata for data ingestion into UKSRC via Rucio.
At a minimum, each of these scripts will generate a json file containing the following metadata keys:
- namespace
- name
- lifetime
- calib_level
- obs_collection
- obs_id
- obs_publisher_did
- access_url
- access_format
- content_type
- dataproduct_type
Please see this confluence page for more details on the required metadata.
The scripts in this directory are written in python and have the following dependencies (also contained in the conda environment yaml, environment.yaml):
- astropy
- astropy-healpix
- casacore
- h5py
- numpy
get_metadata.py gets relevant header metadata from FITS (including uvfits), hdf5, and measurement set (still in development) files. It accepts a set of command line arguments as inputs including a path to at least one file. The full list of command line arguments and their descriptions can be seen via the -h or --help flags, i.e.
python get_metadata.py -h
Metadata for multiple files can be generated by passing multiple file paths as positional arguments. A metadata file will be created for each input file. For example, to create metadata files for two files, /path/to/file1.fits and /path/to/file2.uvfits, with a lifetime of 1 week (604800 seconds) belonging to the 'teal' namespace, you would execute
python get_metadata.py --namespace teal --lifetime 604800 /path/to/file1.fits /path/to/file2.fits
This script has been tested on FITS files containing 2D image and HEALPix data, uvfits files containing simulated and MWA visibilities, and a simple measurement set. It has been designed to work on data of arbitrary dimension as long as each dimension has a relevant 'CTYPE' header keyword.
If the file contains HEALPix data, this script only looks for the following header keywords:
- NSIDE
- ORDERING
- NPIX
- FIRSTPIX
- LASTPIX
If the file contains non-HEALPix data, which is assumed to be N-dimensional image data, this script examines all the 'CTYPEn' keywords looking for either 'GLON' and 'GLAT' or 'RA--', 'RA' and 'DEC-', 'DEC'. If galactic reference coordinates are found, these coordinates are converted to (RA, Dec). In this case, this script only looks for the following header keywords:
- NAXIS
- CTYPEn
- CRVALn
Additionally, for all types of FITS data, this script also looks for the following header keywords:
- NAXIS1
- NAXIS2
- OBJECT
- TELESCOP