run_woden.py

This is the main WODEN executable. It takes command line arguments, creates and array layout, visibility containers, read sky models, and launches GPU code to calculate the visibilities. Finally, it writes the outputs to uvfits files.

Command line running options

Run the WODEN simulator and profit. WODEN is setup to simulate MWA-style observations, where the full frequency bandwidth is split into 24 ‘coarse’ bands, each of which is split into fine channels. This naturally allows any simulation to be split across multiple GPUs as separate processes.

usage: run_woden.py [-h] [--ra0 RA0] [--dec0 DEC0] [--date DATE]
                    [--no_precession] [--band_nums BAND_NUMS]
                    [--lowest_channel_freq LOWEST_CHANNEL_FREQ]
                    [--coarse_band_width COARSE_BAND_WIDTH]
                    [--num_freq_channels NUM_FREQ_CHANNELS]
                    [--freq_res FREQ_RES] [--num_time_steps NUM_TIME_STEPS]
                    [--time_res TIME_RES] [--latitude LATITUDE]
                    [--longitude LONGITUDE] [--array_height ARRAY_HEIGHT]
                    [--array_layout ARRAY_LAYOUT]
                    [--primary_beam PRIMARY_BEAM] [--off_cardinal_dipoles]
                    [--telescope_name TELESCOPE_NAME]
                    [--hdf5_beam_path HDF5_BEAM_PATH]
                    [--MWA_FEE_delays MWA_FEE_DELAYS] [--use_MWA_dipflags]
                    [--use_MWA_dipamps] [--beam_ms_path BEAM_MS_PATH]
                    [--no_beam_normalisation] [--station_id STATION_ID]
                    [--eb_point_to_phase] [--eb_ra_point EB_RA_POINT]
                    [--eb_dec_point EB_DEC_POINT] [--move_array_to_latlon]
                    [--uvbeam_file_path UVBEAM_FILE_PATH]
                    [--cst_file_list CST_FILE_LIST]
                    [--gauss_beam_FWHM GAUSS_BEAM_FWHM]
                    [--gauss_beam_ref_freq GAUSS_BEAM_REF_FREQ]
                    [--gauss_ra_point GAUSS_RA_POINT]
                    [--gauss_dec_point GAUSS_DEC_POINT] [--IAU_order]
                    --cat_filename CAT_FILENAME
                    [--metafits_filename METAFITS_FILENAME]
                    [--output_uvfits_prepend OUTPUT_UVFITS_PREPEND]
                    [--sky_crop_components] [--sky_crop_sources] [--do_autos]
                    [--cpu_mode] [--num_threads NUM_THREADS]
                    [--max_sky_directions MAX_SKY_DIRECTIONS]
                    [--precision PRECISION] [--chunking_size CHUNKING_SIZE]
                    [--dry_run] [--remove_phase_tracking] [--version]
                    [--verbose] [--save_log] [--profile]

OBSERVATION OPTIONS

--ra0

RA of the desired phase centre (deg). If not set, must be using a metafits file or measurement set.

Default: nan

--dec0

Dec of the desired phase centre (deg). If not set, must be using a metafits file or measurement set.

Default: nan

--date

Initial UTC date of the observatio in format YYYY-MM-DDThh:mm:ss This is used to set the LST and array precession. This is set automatically when reading a metafits but including this will override the date in the metafits

Default: False

--no_precession

By default, WODEN rotates the array back to J2000 to match the input sky catalogue. Add this to switch off precession

Default: False

FREQUENCY OPTIONS

--band_nums

Defaults to running 24 coarse bands. Alternatively, enter required numbers delineated by commas, e.g. –band_nums=1,7,9

Default: 'all'

--lowest_channel_freq

Set the frequency (Hz) of the lowest channel for band 1. If using a metafits file, this will override the frequency in the metafits

Default: False

--coarse_band_width

Set the width of each coarse band If using a metafits file, this will override the frequency in the metafits

Default: 'obs'

--num_freq_channels

Number of fine frequency channels to simulate - defaults to –coarse_band_width / –freq_res

Default: 'obs'

--freq_res

Fine channel frequnecy resolution (Hz) - will default to what is in the metafits

Default: False

TIME OPTIONS

--num_time_steps

The number of time steps to simualte. Defaults to how many are inthe metafits if using metafits

Default: False

--time_res

Time resolution (s) - will default to what is in the metafits if the metafits if using metafits

Default: False

TELESCOPE OPTIONS

--latitude

Central Latitude (deg) of the array: if –primary_beam=EB_LOFAR, this defaults to LOFAR at 52.905329712, elif –primary_beam=UVB_HERA, this defaults to HERA at -30.72152612068926, else defaults to MWA at -26.703319405555554. –latitude will override these defaults.

Default: nan

--longitude

Central Longitude (deg) of the array: if –primary_beam=EB_LOFAR, this defaults to LOFAR at 6.867996528, elif –primary_beam=UVB_HERA, this defaults to HERA at 21.42830382686301, else defaults to MWA at 116.67081523611111. –longitude will override these defaults.

Default: nan

--array_height

Central height (m) of the array: if –primary_beam=EB_LOFAR, this defaults to LOFAR at 0.0, elif –primary_beam=UVB_HERA, this defaults to HERA at 1051.6900000007606, else defaults to MWA at 377.827. –array_height will override these defaults.

Default: nan

--array_layout

Instead of reading the array layout from the metafits file, read from a text file. Store antenna positions as offset from array centre, in east, north, height coords (metres)

Default: False

--primary_beam

R|Which primary beam to use in the simulation. Options are:

• MWA_FEE (hyperbeam MWA fully embedded element model;

  defaults to using env variable $MWA_FEE_HDF5 as input; use –hdf5_beam_path to specifiy otherwise)

• MWA_FEE_interp (hyperbeam MWA fully embedded element model that has had

  spherical harmonics interpolated over frequency via ; defaults to using env variable $MWA_FEE_HDF5_INTERP as input; use –hdf5_beam_path to specifiy otherwise)

• Gaussian (Analytic symmetric Gaussian)

  see –gauss_beam_FWHM and and –gauss_beam_ref_freq for fine control)

• EDA2 (Analytic dipole with a ground mesh)

• MWA_analy (MWA analytic model)

• everybeam_OSKAR (EveryBeam OSKAR model; requires an OSKAR measurement set via –beam_ms_path)

• everybeam_LOFAR (EveryBeam LOFAR model; requires a LOFAR measurement set via –beam_ms_path)

• everybeam_MWA (EveryBeam MWA model; does NOT require a measurement set.

  defaults to using env variable $MWA_FEE_HDF5 as input; use –hdf5_beam_path to specifiy otherwise)

• uvbeam_MWA (pyuvdata.uvbeam MWA model;

  defaults to using env variable $MWA_FEE_HDF5 as input; use –hdf5_beam_path to specifiy otherwise)

• uvbeam_HERA (pyuvdata.uvbeam HERA beam model either from CST simulations

  via –cst_file_list or some other UVBeam file via –uvbeam_file_path)

• none (Don’t use a primary beam at all)

Defaults to –primary_beam=none

Default: 'none'

--off_cardinal_dipoles

Add this to force the dipole orientations to be at 45, 135 degrees (aka NE-SW and SE-NW orientated) rather than 0, 90 (a.k.a. N-S, E-W). This changes the mixing matrix that applies the beam gains to the Stokes IQUV values to create instrumental Stokes visibilities. Currently always set to False, pending further investigation.

Default: False

--telescope_name

Name of telescope written out to the uvfits file. Defaults to something “sensible” based on the primary beam being used.

Default: False

MWA PRIMARY BEAM OPTIONS

--hdf5_beam_path

Location of the hdf5 file holding the MWA FEE beam coefficients

Default: False

--MWA_FEE_delays

R|A list of 16 delays to point the MWA FEE primary beam model enter as as list like: –MWA_FEE_delays=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0] for a zenith pointing. This is read directly from the metafits if using a metafits file. Used for all MWA model variants.

Default: False

--use_MWA_dipflags

Apply the dipole flags stored in the metafits file. Only effects hyperbeam and uvbeam models, does not work on analytic or everybeam models.

Default: False

--use_MWA_dipamps

Attempt to use bespoke MWA dipole amplitudes stored in the metafits file. Must be stored under the DipAmps column. Only effects hyperbeam and uvbeam models, does not work on analytic or everybeam models.

Default: False

EVERYBEAM PRIMARY BEAM OPTIONS

--beam_ms_path

When using a LOFAR/OSKAR everybeam primary beam option, must provide a path to the measurement set

Default: False

--no_beam_normalisation

When using an everybeam primary beam option, do not normalise the primary beam to the beam centre. By default, WODEN calculates the beam response at the beam centre, and multiplies all beam values by the inverse of this value. Add this option to switch that off. Note this does not have an effect for the MWA beam

Default: False

--station_id

When using any everybeam primary beam option, default is to simulate a unique beam per station.Include this index to use a specific station number for all stations instead. (Does not apply to MWA beam)

Default: nan

--eb_point_to_phase

Lock the EveryBeam pointing to the phase centre. EveryBeam reads the beam pointing in from a measurement set, so this defaults to whatever is in –beam_ms_path. If added, –eb_point_to_phase will trigger WODEN to create a minimal copy of the measurement set with the pointing set to the phase centre

Default: False

--eb_ra_point

The RA (deg) to lock the EveryBeam primary beam centre to. EveryBeam reads this pointing in from a measurement set, so this defaults to whatever is in –beam_ms_path. If added, –eb_ra_point will trigger WODEN to create a minimal copy of the measurement set with the pointing set to this value

Default: nan

--eb_dec_point

The Declination (deg) to lock the EveryBeam primary beam centre to. EveryBeam reads this pointing in from a measurement set, so this defaults to whatever is in –beam_ms_path. If added, –eb_dec_point will trigger WODEN to create a minimal copy of the measurement set with the pointing set to this value

Default: nan

--move_array_to_latlon

Use the arguments in –latitude and –longitude to move the array to a new location. Does a number of rotations to the linked measuerment in –beam_ms_path. Writes results to a new measurement set to pass to EveryBeam.

Default: False

PYUVDATA UVBEAM PRIMARY BEAM OPTIONS

--uvbeam_file_path

When using –primary_beam=uvbeam_HERA, use this to point to a single UVBeam file to use, e.g. –uvbeam_file_path=/path/to/beamfile.fits. Assumes the file is a functional UVBeam file; error messages associated from file will come from pyuvdata.

Default: False

--cst_file_list

When using –primary_beam=uvbeam_HERA, can also provide a list of CST simulation file locations. Provided via a csv file, where the first column is the path to the CST file, and the second column is the frequency in Hz. Must be comma separated. If there are commas in the path/filenames this will fail.

Default: False

GAUSSIAN PRIMARY BEAM OPTIONS

--gauss_beam_FWHM

The FWHM of the Gaussian beam in deg - WODEN defaults to using 20 deg if this is not set

Default: 20

--gauss_beam_ref_freq

The frequency at which the gauss beam FWHM is set at. If not set, WODEN will default to 150MHz.

Default: 150000000.0

--gauss_ra_point

The initial RA (deg) to point the Gaussian beam at. This will be used to calculate an hour angle at which the beam will remain pointed at for the duration of the observation. Defaults to the RA of the metafits if available, or the RA of the phase centre if not

Default: False

--gauss_dec_point

The initial Dec (deg) to point the Gaussian beam at. Defaults to the Dec of the metafits if available, or the Dec of the phase centre if not

Default: False

INPUT/OUTPUT OPTIONS

--IAU_order

NEW IN WODEN versions >= 1.4.0. By default, the XX pol is now from the East-West aligned dipoles. Add –IAU_order to define the XX pol as North-South. Background: the first polaristaion output in the uvfits is called “XX”. The IAU defines “XX” as aligned to North-South, however typically it is assumed that “XX” in a uvfits is from the East-West dipoles. Le sigh. For WODEN versions < 1.4.0, XX was always the N-S dipoles.

Default: False

--cat_filename

Path to WODEN style sky model

--metafits_filename

MWA style metafits file to base the simulation on. Array layout, frequency and time parameters are all set by this option, but can be overridden using other arguments

Default: False

--output_uvfits_prepend

Prepend name for uvfits - will append band%02d.uvfits %band_num at the end. Defaults to “output”.

Default: 'output'

--sky_crop_components

This option is deprecated, but held here for compatibility.

Default: True

--sky_crop_sources

WODEN will crop out sky model information that is below the horizon for the given LST. By default, WODEN will include any COMPONENT above the horizon, regardless of which SOURCE it belongs to. If –sky_crop_source is included for each SOURCE in the sky model, if any COMPONENT is below the horizon, the entire source will be flagged

Default: False

--do_autos

By default, WODEN only calculates cross-correlations. Add this flag to include auto-correlations.

Default: False

SIMULATOR OPTIONS

--cpu_mode

Switch to CPU only mode. By default, WODEN will attempt to use a GPU. Add this flag to force CPU only mode

Default: False

--num_threads

Number of threads to run the sky model reading / EveryBeam calculations on. Defaults to the number of physical cores on the machine. Add to set a specific number e.g. –num_threads=8

Default: 0

--max_sky_directions

Maximum number of directions on the sky to calculate per sky model chunk. Only useful for controlling EveryBeam calculations. For EveryBeam, defaults to 200, which means a maximum of 200 components go into each chunk. For other primary beams, finds a maximum based on –chunking_size.

Default: 0

--precision

What precision to run WODEN at. Options are “double” or “float”. Defaults to “double”

Default: 'double'

--chunking_size

The chunk size to break up the point sources into for processing - defaults to 1e10

Default: 10000000000.0

--dry_run

Add this to NOT call the WODEN executable - this will just write out the .json file and do nothing else

Default: False

--remove_phase_tracking

By adding this flag, remove the phase tracking of the visibilities - use this to feed uvfits into the RTS

Default: False

LOGGING OPTIONS

--version

Just print the version of WODEN and exit

Default: False

--verbose

Add to increase the verbosity of the logging. Extra information will be prefaced with DEBUG

Default: False

--save_log

By default, WODEN just logs to stdout. Add this flag to save a log file to disk

Default: False

--profile

By adding this flag, profile the WODEN code using line_profiler Must also run the code via LINE_PROFILE=1 run_woden.py

Default: False

Function documentation

Wrapper script to run the WODEN simulator. Author: J.L.B. Line

run_woden.get_future_result(future: Future, logger: Logger, function: Callable)

This function gets the result of a future, and handles any exceptions that may occur. It logs the error and exits the program if an exception occurs. :param future: The future to get the result from, as returned by the ProcessPoolExecutor. :type future: Future :param logger: The logger to use for logging. :type logger: Logger :param function: The function that was run in the future. Used for logging. :type function: Callable

Returns:

The result of the future, or None if an exception occurred.

Return type:

Any

run_woden.main(argv=None)

Runs the WODEN simulator, given command line inputs. Does fancy things like reading in the sky model and running the GPU code in parallel; the sky model lazy load allows us to simulate sky models that cannot fit into RAM.

Parameters:

argv (_type_, optional) – Will be parsed from the command line if run in main script. Can be passed in explicitly for easy testing

run_woden.read_skymodel_worker(thread_id: int, num_threads: int, chunked_skymodel_map_sets: List[Skymodel_Chunk_Map], lsts: ndarray, latitudes: ndarray, args: Namespace, beamtype: int, main_table: Table, shape_table: Table, v_table: Table = False, q_table: Table = False, u_table: Table = False, p_table: Table = False, logger: Logger = False, uvbeam_objs: ndarray = None) → Tuple[List[Source_Python], int]

Reads a list of Skymodel_Chunk_Map types in chunked_skymodel_map_sets from the given astropy tables, into a list of Source_Python types. Depending on the type of primary beam specified by beamtype

Parameters:

• thread_id (int) – The ID of the current thread.

• num_threads (int) – The total number of threads.

• chunked_skymodel_map_sets (List[Skymodel_Chunk_Map]) – A list of chunked skymodel map sets.

• lsts (np.ndarray) – Local Sidereal Times (LSTs) for all time steps.

• latitudes (np.ndarray) – Latitudes of the array at all time steps; when doing array precession, these will all be slightly different.

• args (argparse.Namespace) – Command-line arguments namespace.

• beamtype (int) – The value of the type of beam being used, e.g. BeamTypes.FEE_BEAM.value

• main_table (Table) – Main table containing skymodel data.

• shape_table (Table) – Table containing Shapelet data.

• v_table (Table, optional) – Table containing V polarization data (default is False).

• q_table (Table, optional) – Table containing Q polarization data (default is False).

• u_table (Table, optional) – Table containing U polarization data (default is False).

• p_table (Table, optional) – Table containing P polarization data (default is False).

• logger (Logger, optional) – The logger to use for logging (default is False). If not provided, a default logger will be created.

• uvbeam_objs (np.ndarray, optional) – The UVBeam objects to use for the primary beam calculations. Only needed if the beamtype is in BeamGroups.uvbeam_beams.

Returns:

tuple – A tuple containing the list of Source_Python`s and the thread number, where thead number is `thread_id % num_threads; this can be used to match the order of recovered data to the order of the chunked maps.

Return type:

Tuple[List[Source_Python], int]

run_woden.run_woden_processing(num_threads, num_rounds, chunked_skymodel_map_sets, lsts, latitudes, args, beamtype, main_table, shape_table, v_table, q_table, u_table, p_table, woden_settings_python, array_layout_python, visi_sets_python, logger=False, serial_mode=False, uvbeam_objs=None)

This function runs the WODEN processing, either in serial or parallel mode. It sets up the WODEN settings, array layout, and visibility set arrays, and then runs the WODEN processing in either serial or parallel mode. It also handles the logging and profiling of the processing. :param num_threads: The number of threads to use for processing. :type num_threads: int :param num_rounds: The number of rounds of processing to perform. :type num_rounds: int :param chunked_skymodel_map_sets: A list of chunked skymodel map sets. :type chunked_skymodel_map_sets: List[Skymodel_Chunk_Map] :param lsts: Local Sidereal Times (LSTs) for all time steps. :type lsts: np.ndarray :param latitudes: Latitudes of the array at all time steps; when doing array precession,

these will all be slightly different.

Parameters:

• args (argparse.Namespace) – Command-line arguments namespace.

• beamtype (int) – The value of the type of beam being used, e.g. BeamTypes.FEE_BEAM.value

• main_table (Table) – Main table containing skymodel data.

• shape_table (Table) – Table containing Shapelet data.

• v_table (Table) – Table containing V polarisation data (if not needed, can be boolean False)

• q_table (Table) – Table containing Q polarisation data (if not needed, can be boolean False)

• u_table (Table) – Table containing U polarisation data (if not needed, can be boolean False)

• p_table (Table) – Table containing P polarisation data (if not needed, can be boolean False)

• woden_settings_python (Woden_Settings_Python) – The WODEN settings to be used.

• array_layout_python (Array_Layout_Python) – The array layout to be used.

• visi_sets_python (List[List[Visi_Set_Python]]) – Place to output the visibilities to. Should be of shape (num_visi_threads, num_bands) where num_visi_threads is the number of threads to use for processing visibilities, and num_bands is the number of frequency bands.

• logger (Logger) – The logger to use for logging. If not provided, a default logger will be created.

• serial_mode (bool) – Whether to run in serial mode (True) or parallel mode (False). Default is False.

• uvbeam_objs (np.ndarray) – The UVBeam objects to use for the primary beam calculations. Only needed if the beamtype is in BeamGroups.uvbeam_beams.

run_woden.sum_components_in_chunked_skymodel_map_sets(chunked_skymodel_map_sets)

Sums the number of components in a list of Skymodel_Chunk_Map types.

Parameters:

chunked_skymodel_map_sets (List[Skymodel_Chunk_Map]) – A list of chunked skymodel map sets.

Returns:

int

Return type:

The total number of components in the list of Skymodel_Chunk_Map types.

run_woden.woden_worker(thread_ind: int, all_loaded_python_sources: List[List[Source_Python]], all_loaded_sources_orders: List[int], round_num: int, woden_settings_python: Woden_Settings_Python, array_layout_python: Array_Layout_Python, visi_sets_python: List[Visi_Set_Python], beamtype: int, logging_level: int = 10, precision: str = 'double', logger: Logger = False, profile=False) → Tuple[List[Visi_Set_Python], int, int]

This worker function runs WODEN CPU/GPU code, to process chunked source catalogues from a queue until the queue is empty.

Parameters:

• thread_ind (int) – The index of the thread this worker is running in. Just used for logging here.

• all_loaded_python_sources (List[List[Source_Python]]) – A list of lists of Source_Python to be processed, as ouput by read_skymodel_worker, where each list of Source_Python matches a chunk_map in chunked_skymodel_map_set. Each entry is a list itself as the Shapelet chunking can result in multiple sources per thread per round, as the chunking happens by sky direction as well as number of shapelet coefficients.

• all_loaded_sources_orders (List[int]) – The order of the sources in all_loaded_python_sources as matched to chunked_skymodel_map_set; orders also output by read_skymodel_worker (different threads finish in different times so the order of the sources in all_loaded_python_sources may not match the order of the chunk_maps)

• round_num (int) – The round number of the processing

• woden_settings_python (Woden_Settings_Python) – The WODEN settings to be used.

• array_layout_python (Array_Layout_Python) – The array layout to be used.

• visi_sets_python (List[Visi_Set_Python]) – The existing visibility sets information. The results of this worker will be added to this and returned. Should have length of num_bands, where num_bands is the number of frequency bands.

• beamtype (int) – The value of the type of beam being used, e.g. BeamTypes.FEE_BEAM.value

• logging_level (int) – The logging level to use for logging. Default is logging.DEBUG.

• precision (str) – The precision to use for the WODEN processing. Default is “double”. Should be either “float” or “double”.

• logger (Logger) – The logger to use for logging. If not provided, a default logger will be created.

• profile (bool) – Whether to profile the function. This is a VERY experimental way of doing things, and only includes some specific functions. Use with caution. Default is False.

run_woden.woden_worker_into_queue(q: Queue, all_loaded_python_sources: List[List[Source_Python]], all_loaded_sources_orders: List[int], round_num: int, woden_settings_python: Woden_Settings_Python, array_layout_python: Array_Layout_Python, visi_sets_python: List[Visi_Set_Python], beamtype: int, args: Namespace, logger: Logger)

This function launches the woden_worker() function in a separate process, putting outputs into the queue q. Specifically, it places

• visi_set_python

• completed_round

into the queue.

Do this to keep the casacore version linked to libwoden_*.so separate from the one linked to python-casacore. When both casacore versions are initialised in the main process, they clash, and can cause segfaults.

This wrapper is only intended for use when running WODEN in serial mode. As such, we pass 0 as the thread index and visi_sets_python[0,:] into woden_worker(), as there should only be one thread’s worth of visibility sets.

Parameters:

• q (Queue) – The queue to put the outputs into.

• all_loaded_python_sources (List[List[Source_Python]]) – A list of lists of Source_Python to be processed, as ouput by read_skymodel_worker, where each list of Source_Python matches a chunk_map in chunked_skymodel_map_set. Each entry is a list itself as the Shapelet chunking can result in multiple sources per thread per round, as the chunking happens by sky direction as well as number of shapelet coefficients.

• all_loaded_sources_orders (List[int]) – The order of the sources in all_loaded_python_sources as matched to chunked_skymodel_map_set; orders also output by read_skymodel_worker (different threads finish in different times so the order of the sources in all_loaded_python_sources may not match the order of the chunk_maps)

• round_num (int) – The round number of the processing

• woden_settings_python (Woden_Settings_Python) – The WODEN settings to be used.

• array_layout_python (Array_Layout_Python) – The array layout to be used.

• visi_sets_python (List[Visi_Set_Python]) – The exisiting visibility sets information. The results of this round of processing will be added to this and put in the queue; visi_sets_python will then need to be updated in turn with this visi_sets_python in the main process.

• beamtype (int) – The value of the type of beam being used, e.g. BeamTypes.FEE_BEAM.value

• args (argparse.Namespace) – Command-line arguments namespace as returned by get_parser().

• logger (Logger) – The logger to use for logging.