This repository contains the implementation code for DSC037, including notebooks for quality assessment and scripts for visibility-based and image-based power spectra computation.
| Notebook | Purpose | Description |
|---|---|---|
plot_vis.ipynb |
step 3 of the WT: visibility inspection | Loads visibilities from LOFAR Measurement Sets (via casacore) or UVFITS/pyuvdata-compatible files and plots amplitude and phase versus time and frequency for individual baselines. Interactive controls let you choose polarisations, antennas, and plotting axes to verify temporal and spectral smoothness. |
plot_calibration_solutions.ipynb |
step 3 of the WT: calibration-solution QA | Opens complex gain solutions saved as FITS tables and plots the per-antenna, per-polarisation amplitude and phase versus frequency. Provides optional phase unwrapping and smoothness metrics to highlight jumps that may indicate cable reflections or calibration pathologies. |
plot_delay_spectra.ipynb |
step 3 of the WT: quick delay-spectra inspection | Computes and plots simple delay power spectra (FFT → absolute value → square). |
A script uses hera_pspec (the HERA power-spectra pipeline) to compute visibility-based delay power spectra (step 4 of the WT).
It creates:
- a time-averaged delay PS per antenna (averaging all baselines including a specific antenna), and
- a time-averaged delay power spectrum across all baselines.
A second script uses pspipe (the LOFAR power-spectra pipeline) to compute image-based cylindrically averaged power spectra (step 4 of the WT).
It creates:
- a Cylndrically averaged power-spectra figure.
- a Angular power-spectra figure.
Both power spectra scripts take a YAML file as input (with the same format).
The file specifies dataset location and input parameters such as instrument, data column, polarisation, and frequency/time ranges.
datafolder : '../data/'
datafile : 'hyp_1184702048_ionosub_ssins_30l_src8k_300it_8s_80kHz_i1000.uvfits'
data_col : None
# The polarizations numbers to include when reading data into the object
pol : [-5, -6]
# The time range in Julian Date to include when reading data into the object, must be length 2. If None, all times in datafile are used.
time_range : None
# The indices describing the range of frequencies to include (min and max only).
freq_range : [100, 200]
# The antennas numbers to include when reading data into the object. If None, all antennas are considered.
antenna_nums : [9, 10] #None
# The maximum baseline length to consider (in meters)
max_bl_len : 30.
# Choose type of analysis between auto- and cross-correlation of visibilities
analysis_type: 'cross'This script performs a direction-independent correction of bandpass effects, including the removal of spectral structures produced by cable reflections.
The script run_bpcal.py automates a two-step DI calibration:
- Bandpass gaincal using DP3 (steps
averagerandgaincal) - Applycal using DP3 (writing corrected visibilities to a new data column)
It also creates a clean working directory (*_run/) for each execution, generates all DP3 parset files based on the YAML configuration, logs all DP3 output to both screen and log files, and removes temporary intermediate Measurement Sets.
python run_bpcal.py config_bpcal.yaml
This produces:
config_bpcal_1/
DI_bandpass_cal.parset
DI_applycal.parset
bandpass-gaincal.h5
bandpass_cal.log
applycal.log
run_bpcal.py takes as input a YAML configuration file describing the Measurement Set location and calibration parameters.
datafolder: "../data/"
datafile: "L253456_SAP000_002_time1.flagged.5ch8s.dical.MS"
input_datacol: "DATA"
output_datacol: "BP_CORRECTED"
skymodelfile: "../skymodels/L253456_DI_10mJy.txt"
min_bl_len: 50.
max_bl_len: 5000.
avg_time_step: 4
time_sol_int: 450
The script uses this to generate DI_bandpass_cal.parset and DI_applycal.parset and executes DP3 accordingly.
Two notebooks are included to validate the calibration solutions and to compare the impact of the calibration on the final power spectra.
This notebook loads H5Parm gain solutions and visualises:
- per-antenna bandpass amplitude
- delay-domain FFT of the bandpass
- antenna-averaged responses
- optional sinusoidal fitting of cable-reflection signatures
It is designed to assess the smoothness and quality of the DI bandpass calibration and to check for residual cable-reflection terms.
This notebook compares two power-spectra products generated by image_ps/run_pspipe.py. It loads two output working directories (e.g. before and after DI bandpass calibration) and plots:
- 2D cylindrical power spectra for each dataset
- the 2D power spectrum of the visibility-difference
- 1D k∥ spectra for each dataset
This allows quick inspection of whether the bandpass calibration successfully suppresses spectral.
Each notebook assumes access to calibrated visibility data from either an MWA or LOFAR observation. The example datasets below are the ones used during the challenge and provide a good starting point when reproducing the analysis.
wget https://projects.pawsey.org.au/high0.uvfits/hyp_1184702048_ionosub_ssins_30l_src8k_300it_8s_80kHz_i1000.uvfitswget -O L253456_SAP000_002_time1.flagged.5ch8s.dical.MS.zip 'https://share.obspm.fr/s/Cek959sM3KRb4BQ/download'
unzip L253456_SAP000_002_time1.flagged.5ch8s.dical.MS.zipAll notebooks rely on a standard scientific Python stack plus packages commonly used for radio-interferometric data analysis:
numpymatplotlibastropypyuvdatahera_pspecpython-casacore(required only when reading Measurement Sets)
For the image-based power-spectra, a container is available at https://share.obspm.fr/s/nj3eB2bA9z9oBLd
For the bandpass calibratuon, DP3 is required which is available in the same container.
The following challenge documentation provides broader context and detailed walkthroughs:
- Chronological walkthrough: https://confluence.skatelescope.org/x/osw6F
- Implementation notes and software design: https://confluence.skatelescope.org/x/n8LMF
Questions about the notebooks can be directed to the Teal team members listed within the notebook headers.