toirex package

Submodules

toirex.toirex_main module

toirex.grouping_frames module

toirex.grouping_frames.grouping_items(config, dirname, catalogue_dict=None, open_editor=True)

Group catalogue files by observing setup while excluding flat frames.

This function reads the catalogue for the specified directory, groups frames according to the instrument-specific grouping keys, and associates each frame with its classification flag. A human-readable summary of the groups is written to Grouped_txtfile.txt, and the grouped catalogue is returned as a nested dictionary.

Parameters:

• config (dict) –

  Configuration dictionary.

  The following keys are required:

  • config['inits']['DICTKW']: Instrument identifier used to select the appropriate grouping rules.

  • config['outputs']['OP_DIR']: Output directory where the grouping summary is written.

• dirname (str or pathlib.Path) – Subdirectory of the output directory corresponding to the dataset.

Returns:

Nested dictionary containing the grouped catalogue.

The outer dictionary maps group numbers to dictionaries of frame types. Each inner dictionary maps classification flags (e.g. OBJECT, ARGON, FLAT) to lists of filenames.

Return type:

dict

Notes

Flat frames are excluded using the instrument-specific flat_kw keyword.

Grouping relies on the following components:

• read_catalog() to load the catalogue.

• instruments[dictkw]['grouping_keys'] to define observing groups.

• instruments[dictkw]['flat_grouping_keys'] to associate flat frames.

• ordered_keys() to construct the grouped catalogue.

The function also writes Grouped_txtfile.txt to <config['outputs']['OP_DIR']>/<dirname>.

A placeholder exists for automatically associating continuum flats in a future version.

Examples

>>> config = {
...     "inits": {"DICTKW": "SpecTANSPEC"},
...     "outputs": {"OP_DIR": "Reduced_data"},
... }
>>> groups = grouping_items(config, "20240908")
>>> sorted(groups.keys())
[0, 1, 2]

toirex.grouping_frames.grouping_with_re(config, dirname)

Group catalogue entries using user-specified regular expressions.

This function reads a catalogue of image frames from the given directory and filters them based on user-defined regular expressions. Users are first prompted to input regex patterns for science frames and flat frames, and optionally for lamp frames (depending on the configuration). The matched frames are collected into a reduced catalogue and then grouped by instrument-specific keywords (e.g., SLIT, GRATING, FILTER) via grouping_items.

Parameters:

• config (dict) –

  Configuration dictionary with at least the following keys:

  • config['inits']['TODO']str

    Indicates the processing mode. If set to 'S', the user will be prompted to enter regex rules for lamp frames as well.

  • config['inits']['DICTKW']str

    Key used to identify the instrument class from instrument_class. This is needed to obtain instrument-specific keywords (e.g., lamp keywords).

• dirname (str) – Path to the directory containing the catalogue and associated files.

Returns:

grouped_dict – A dictionary of grouped catalogue entries, as produced by grouping_items. The grouping is based on instrument-specific parameters (SLIT, GRATING, FILTER, etc.) after filtering by the user-provided regex rules.

Return type:

dict

Notes

• The function is interactive: it prompts the user to enter regular expressions for selecting science, flat, and (if applicable) lamp frames.

• Frames that match at least one of the provided regex rules are collected.

• Grouping is performed by calling the external grouping_items function.

• Regex documentation reference: http://docs.python.org/2/howto/regex.html#regex-howto

Examples

>>> config = {
...     'inits': {
...         'TODO': 'S',
...         'DICTKW': 'TIRSPEC'
...     }
... }
>>> dirname = "/path/to/data"
>>> grouped = grouping_with_re(config, dirname)
Enter the regular expression for SCIECNE frames: .*M31.*
Enter the regular expression for FLAT frames: .*continuum.*
Enter the regular expression for LAMP frames: .*argon.*
>>> print(grouped.keys())
dict_keys(['FNAME', 'SLIT', 'FILTER', 'GRATING', ...])

toirex.grouping_frames.ordered_keys(catalog_dict, config, grouping_keys, flats_keys, flat_flag)

Group files by specified catalogue keys and append matching flats.

This function groups catalogue entries according to a set of header keys (e.g., grism, filter, order). For each unique group, it collects the corresponding file names and appends any flat frames that match based on flats_keys. The result is a dictionary mapping group indices to arrays of file names (science + flats).

Parameters:

• catalog_dict (dict) – Dictionary containing the catalogue. Must include: - 'FNAME' : list or array of file names. - Additional keys specified in grouping_keys and flats_keys.

• config (dict) – Configuration dictionary (currently unused in this function, but passed for consistency with other pipeline functions).

• grouping_keys (list of str) –

  Catalogue dictionary keys used to define the main grouping of science

  files (e.g., [“OBJECT”, “FILTER”, “GRISM”]).

  flats_keyslist of str

  Catalogue dictionary keys used to group flat frames.

• flat_flag (list of str) – List of flags identifying flat frames in the catalogue.

Returns:

grouped_files – Dictionary where: - keys are integer group indices (0, 1, 2, …), - values are arrays of file names corresponding to that group,

including any matching flat frames.

Return type:

dict

Notes

• Groups are formed by unique combinations of values in grouping_keys.

• Flat frames are grouped separately using select_flats and then merged into the corresponding science groups if their key values match.

• Duplicate groups are removed using remove_repeated_values.

• The function preserves the order of first occurrences.

Examples

>>> catalog_dict = {
...     "FNAME": ["sci1.fits", "sci2.fits", "flat1.fits"],
...     "FLAG":  ["SCIENCE", "SCIENCE", "FLAT"],
...     "OBJECT": ["StarA", "StarA", "Lamp"],
...     "FILTER": ["R", "R", "R"],
...     "GRISM":  ["G1", "G1", "G1"]
... }
>>> grouping_keys = ["OBJECT", "FILTER", "GRISM"]
>>> flats_keys = ["FILTER", "GRISM"]
>>> flat_flag = ["FLAT"]
>>> grouped = ordered_keys(catalog_dict, {}, grouping_keys, flats_keys,
    flat_flag)
>>> list(grouped.keys())
[0]   # one science group
>>> grouped[0]
array(['sci1.fits', 'sci2.fits', 'flat1.fits'], dtype='<U10')

toirex.grouping_frames.reading_re(entered_re)

Function to read the entered regular expression and separate FNUM from the entered item.

toirex.grouping_frames.remove_repeated_values(candidate_list)

Remove duplicate elements from a list, with special handling for NumPy arrays.

This function iterates through the input list and removes repeated elements. Unlike the built-in set, it preserves the original order and works with elements that are NumPy arrays by comparing their contents (via tolist()) rather than object identity.

Parameters:

candidate_list (list) – Input list potentially containing repeated elements. Elements can be of arbitrary type, including NumPy arrays.

Returns:

• filtered_list (list) – A new list with repeated elements removed, preserving the first

  occurrence of each unique element.

  Notes

• —–

• - For NumPy arrays, equality is determined by comparing the result of – .tolist(), so arrays with the same contents but different memory locations are considered duplicates.

• • The function preserves the order of elements, unlike set().

Examples

>>> import numpy as np
>>> arr1 = np.array([1, 2, 3])
>>> arr2 = np.array([1, 2, 3])
>>> candidate_list = [arr1, arr2, [4, 5], [4, 5], "a", "a"]
>>> remove_repeated_values(candidate_list)
[array([1, 2, 3]), [4, 5], 'a']

toirex.grouping_frames.select_flats(catalog_dict, flats_keys, flat_flag)

Select and group flat frames from a catalogue.

This function extracts flat frames from the input catalogue based on classification flags and groups them according to a set of catalogue keys (e.g., grism, filter, order). Duplicate groups are removed using remove_repeated_values. The grouped flats are returned as a dictionary mapping a concatenated string of key values to the corresponding flat file names.

Parameters:

• catalog_dict (dict) – Dictionary containing the catalogue. Must include: - 'FNAME' : list or array of file names - 'FLAG' : list or array of classification flags - Additional keys specified in flats_keys.

• flats_keys (list of str) – List of catalogue dictionary keys used to define unique groups (e.g., [“GRISM”, “FILTER”]).

• flat_flag (list of str) – List of flags that identify flat frames in the catalogue.

Returns:

grouped_flats – Dictionary where: - keys are strings obtained by joining the values of the group-defining

keys with a space (e.g., “GRISM1 FILTER2”),

• values are arrays of file names corresponding to that group of flats.

Return type:

dict

Notes

• The grouping is done by extracting values from catalog_dict at

positions

where FLAG matches flat_flag.

• Duplicate groups are removed using remove_repeated_values.

• The function assumes that the catalogue dictionary values can be indexed and broadcast into NumPy arrays.

Examples

>>> catalog_dict = {
...     "FNAME": ["flat1.fits", "flat2.fits", "sci1.fits"],
...     "FLAG":  ["FLAT", "FLAT", "SCIENCE"],
...     "FILTER": ["R", "R", "R"],
...     "GRISM":  ["G1", "G1", "G1"]
... }
>>> flats_keys = ["FILTER", "GRISM"]
>>> flat_flag = ["FLAT"]
>>> select_flats(catalog_dict, flats_keys, flat_flag)
{'R G1': array(['flat1.fits', 'flat2.fits'], dtype='<U10')}

toirex.obscatalog module

toirex.obscatalog.create_catalog(dirname, config)

Function to create the catalogue of files available in the list. Inputs —— dirname: Name of the directory. config: The main config file.

toirex.obscatalog.extract_catalog_entries(fname, dictkw)

This function will take the required kewords from the header and enter it into a list. This list will be used to enter into the catalog.

toirex.obscatalog.flag_sky(catalogue_entries)

The function to identify sky frames and flag them.

toirex.obscatalog.read_catalog(dirname, config=None, showcatname=True)

if config is not None, the main catalog will be read. Otherwise, give the required text editor ffor dirname.

toirex.selecting_frames module

toirex.selecting_frames.feed_to_txt_file(grouped_files, config, dirname, group)

Write grouped flat and lamp files to separate text files and open them in an editor.

This function takes the grouped files dictionary (produced during grouping), identifies the flat-field and lamp calibration frames associated with each science object, and writes them into two text files:

• Objects_flats_group{group}.txt

• Objects_lamps_group{group}.txt

Each line in the output file contains one science object filename, followed by the list of associated flat or lamp calibration files. After writing, each file is opened in the editor specified in the config.

Parameters:

• grouped_files (dict) – Dictionary containing grouped science object and calibration file

• names. – Must include at least the keys ‘OBJECT’, instrument.flat_kw, and

• instrument.lamp_kw.

• config (dict) –

  Configuration dictionary. Expected to contain:

  • config[‘inits’][‘DICTKW’]: key for selecting the instrument class

  • config[‘outputs’][‘OP_DIR’]: output directory path

  • config[‘editor’]: preferred text editor (used by open_in_editor)

• dirname (str or Path) – Subdirectory under the output directory where the text files will be

• written. (Each file is opened in the configured text editor after being)

• group (int) – Group number identifier used in the output file names.

• Outputs

• -------

• directory (Creates two text files in the) – <OP_DIR>/<dirname>/Objects_flats_group{group}.txt <OP_DIR>/<dirname>/Objects_lamps_group{group}.txt

• written.

Notes

• If no calibration files exist for a given object, only the object

filename

will be written on that line.

• The mapping between calibration type (flat or lamp) and grouped_files

keys

is provided by the instrument class (instrument.flat_kw,

instrument.lamp_kw).

toirex.dithering module

toirex.photometry module

toirex.spectral_reduction module

toirex.setups module

toirex.setups.add_dict_keywords(config)

toirex.setups.create_config(configfilename, entries)

This function is to write a new config file. Inputs —— configfilename: Name of the config file with path. entries: Keywords and values as a dictionary.

toirex.setups.create_dir(dirname)

This function will create the directory. Input —– dirname: Name of the directory that need to be created. If the directory already exists, it will be skipped.

toirex.setups.get_logger(name=None) → Logger

Return the configured logger. If name is given, returns a child logger.

toirex.setups.log_separator(char='=', length=60)

toirex.setups.print_banner()

This function is to print the banner of the pipeline.

toirex.setups.read_args()

Read the argument while execution. This function will take the config file.

toirex.setups.read_config(configfile)

Function to read the config file.

Input

configfile: Name of the config file.

Output

config: Read config file. dict.

toirex.setups.read_dirs(dirname='.')

toirex.setups.setup_logger_from_config(config: dict, log_file='TOIREx_logging.log') → Logger

Configure logger using entries from the config. The logger name is taken from config[‘INSTRUMENT’].

toirex.utils module

---

toirex.instrument.call_masterflat_tirspec(frame, instrument_config='config/instrument_templates.config')

toirex.instrument.catalog_flag_spectanspec(flog_list: list, headers_list: list) → list

This function is to flag the frames as object, argon, neon, cont1 or cont2. Specific for instrument. Input —— flog_list: catalog entries. :rtype: catalog list with flag.

toirex.instrument.catalog_flag_tirspec(flog_list: list, headers_list: list) → list

toirex.instrument.frame_select_spectanspec(fname: str) → bool

This function will decide weather to select a frame or not. Since TANSPEC have both image and spectroscopy mode, need to decied weather to select certain files or not.

toirex.instrument.frame_select_tirspec(fname: str) → bool

toirex.instrument.get_response_spectanspec(fname, instrument_config='config/instrument_templates.config')

toirex.instrument.get_stdsky_spectanspec(fname, instrument_config='config/instrument_templates.config')

toirex.instrument.get_template_spectanspec(lampfname, index, instrument_config='config/instrument_templates.config')

toirex.instrument.load_badpixelmask_tirspec(instrument_config='config/instrument_templates.config')

toirex.instrument.makemasterflat_tanspec(normcontdata)

This function will create a master flat using the continuum flat of each night and already generated master flat. This is to take care of noise for higher orders (orders 10, 11 nd 12) in XD mode. Basically, we will use the master flat to remove noise in higher orders and for the lower orders, the pipeline will use the continuum lamp observed in each night for the flat correction. In the end it will return the data for the new continuum flat.

toirex.instrument.masterflat_combination(flat_fname)

toirex.instrument.pixel_offset_spectanspec(lampspectra)

toirex.instrument.select_trace_spectanspec(dataframe, instrument_config='config/instrument_templates.config')

toirex.instrument.select_trace_tirspec(dataframe, instrument_config='config/instrument_templates.config')

toirex.instrument.sort_filename_key(fname: str, regexp='(\\d+)\\.Z\\.fits')

toirex.instrument.standardise_header_spectanspec(fname: str) → dict

toirex.instrument.standardise_header_tirspec(fname: str) → dict

Module contents