mangadap.proc.templatelibrary module

Class that reads and prepares template libraries for use in fitting the stellar-continuum of a spectrum. See available_template_libraries() for the list of default template libraries.

The “raw” template libraries are expected to consist of 1D fits files. The wavelength sampling can be either linear or logarithmic in wavelength. The reference frame of the template wavelengths must be defined as either vacuum or air. It is expected that the object spectra to be fit are calibrated to vacuum wavelengths. When preparing the template spectra for analysis, this class will use pydl.goddard.astro.airtovac to convert the template wavelengths to vacuum. Finally, one can specify that the template library is only valid within a certain wavelength range and above a certian flux limit; see TemplateLibraryDef.

Preparation of the template library for use in stellar-continuum fitting consists of a number of steps as described in the documentation of TemplateLibrary.process_template_library().

A template library that has been prepared for analysis is automatically written to disk for later recovery.

Two support classes are also provided. One is a derived mangadap.par.parset.KeywordParSet instance that provides the defining parameters of a DAP template library. The second is a derived mangadap.util.bitmask.BitMask instance that defines the bitmasks for the template library spectra.

Class usage examples

TODO: Update this for new datacube usage

Assuming you have the default directory structure setup, you can do:

# Imports
from mangadap.drpfits import DRPFits
from mangadap.proc.templatelibrary import TemplateLibrary
from matplotlib import pyplot

# Define the DRP file
drpf = DRPFits(7495, 12703, 'CUBE')

# Build the template library
tpl_lib = TemplateLibrary('M11MILES', drpf=drpf, directory_path='.')
# Writes: ./manga-7495-12703-LOGCUBE_M11-MILES.fits

# Plot one of the spectra
pyplot.plot(tpl_lib.hdu['WAVE'].data, tpl_lib.hdu['FLUX'].data[0,:])
pyplot.show()

As part of the instantiation of the TemplateLibrary object in the above call, the template library is prepared for use in fitting the specified DRP file. If the required processing has already been done, the instantiation of the TemplateLibrary object simply reads the existing file. If you do not have the default directory structure setup, you’ll need to define the paths to, e.g., the DRP file; see mangadap.util.drpfits.DRPFits.

If you do not want to process the template (match the spectral resolution and sampling to that of the DRP data), you can force the TemplateLibrary object to only provide the “raw” spectra:

# Imports
from mangadap.proc.templatelibrary import TemplateLibrary
from matplotlib import pyplot

# Read the raw template library
tpl_lib = TemplateLibrary('M11MILES', process=False)
# Nothing should be written to disk

# Plot one of the spectra
pyplot.plot(tpl_lib.hdu['WAVE'].data[0,:], tpl_lib.hdu['FLUX'].data[0,:])
pyplot.show()

Note that in the previous example, the wavelength data was already one-dimensional, whereas for the raw library, the wavelength vector can be spectrum-dependent.

In the above examples, the user has not provided a list of template libraries, meaning that the default set available to the DAP is used. The default set is defined in available_template_libraries(). If you want to use your own template library, you have to define its parameters using TemplateLibraryDef. Currently, the template library spectra are expected to be 1D fits files with WCS header keywords defining the wavelength solution; see above. Using an existing DAP library as an example:

# Imports
import os
from mangadap.proc.templatelibrary import TemplateLibraryDef
from mangadap.proc.templatelibrary import TemplateLibrary
from mangadap.config.defaults import dap_data_root
from matplotlib import pyplot

# Define the search string for the library
search_str = os.path.join(dap_data_root(), 'stellar_templates/miles/*.fits')

# Define the template library parameters
tpl_par = TemplateLibraryDef(key='MILES',    # Unique keyword for the library
                        file_search=search_str, # Search string
                        fwhm=2.50,              # FWHM of resolution element (assumed const)
                        in_vacuum=False,        # Wavelength in vacuum?
                        wave_limit=numpy.array([ 3575., 7400. ]),   # Range of valid lambda
                        lower_flux_limit=0.0)   # Lower limit for valid flux

# Read the raw template library
tpl_lib = TemplateLibrary('MILES', tpllib_list=tpl_par, process=False)
# Nothing should be written to disk

# Plot one of the spectra
pyplot.plot(tpl_lib.hdu['WAVE'].data[0,:], tpl_lib.hdu['FLUX'].data[0,:])
pyplot.show()

Note that the keyword you use must be provided both to the parameter set and when instantiating the TemplateLibrary object. In the example above, I have not processed the library, but you can by following a similar approach to the first example.

You can also process the spectra to a user-provided resolution and pixel sampling, toggle on/off the renormalization of the library to a mean flux of unity and define various paths if you’re not working within the nominal DAP directory structure. See the optional instantiation arguments for TemplateLibrary.

class mangadap.proc.templatelibrary.TemplateLibrary(library, cube=None, match_resolution=True, velscale_ratio=None, sres=None, velocity_offset=0.0, min_sig_pix=0.0, no_offset=True, spectral_step=None, log=True, wavelength_range=None, renormalize=True, output_path=None, output_file=None, read=True, process=True, hardcopy=False, symlink_dir=None, overwrite=False, checksum=False, loggers=None, quiet=False)[source]

Bases: object

TODO: Revisit these docs.

Object used to read, store, and prepare template libraries for use in analyzing object spectra.

The default list of available libraries provided by the MaNGA DAP defined in available_template_libraries(). The user can provide their own library for use with this class provided they are contained in 1D fits spectra, sampled linearly in wavelength with the wavelength coordinates available via the WCS keywords (CRPIX1, CRVAL1, CDELT1), and they have an appropriately defined spectral resolution. See TemplateLibraryDef and _build_raw_hdu().

The class is optimized for use in analyzing MaNGA DRP files; however, one can provide the necessary information so that the class can be used with a non-DRP spectrum. In the latter case, the user must supply the velocity scale of the pixel for the logarithmically resampled template library, and a mangadap.util.resolution.SpectralResolution object the defines the instrumental resolution of the spectrum/spectra to be analyzed.

Todo

Docs are out of date.
Update the list of attributes.
Allow for velocity scale input instead of spectral_step?
Only works with DRP files that have log-linear wavelength binning!
Allow to process, where process is just to change the sampling or the resolution (not necessarily both).
Need to make this more general, removing all dependence on DRPFits object. This would simplify the functionality to change how the resolution and sampling matching is specified.

On initialization, if the datacube object is not provided (is None), the default behavior is to read the raw template library if read=True. If the datacube is provided, the routine will check for the resolution matched fits file; if it doesn’t exist and read is True, it will prepare the template library for use in analyzing the datacube and write the prepared library file (if hardcopy is True). If overwrite=True, the preparation and writing of the template library will be done even if the library already exists.

Todo

Change sres to be a standard input object?

Parameters:

library (str, TemplateLibraryDef) – A keyword selecting one of the template libraries distributed with the DAP or the definition of a user-provided library.
cube (mangadap.datacube.datacube.DataCube, optional) – The datacube to be analyzed using the template library.
match_resolution (bool, optional) – Match the spectral resolution of the template library to the resolution provided by the cube; the latter must be provided for this argument to have any use.
velscale_ratio (int, optional) – Resample the template spectra such that the ratio of the pixel scale in the provided cube is this many times larger than the pixels in the resampled template spectrum.
sres (mangadap.util.resolution.SpectralResolution, optional) – The object is used simply to access the spectral resolution and associated wavelength coordinate vector needed when matching the spectral resolution of the template library; this is used in lieu of a datacube.
velocity_offset (float, optional) – Velocity offset to use when matching the spectral resolution between the template library and the galaxy spectra.
min_sig_pix (float, optional) – Minimum value of the LSF standard deviation in pixels allowed before assuming the kernel is a Delta function. See mangadap.util.resolution.match_spectral_resolution().
no_offset (bool, optional) – Prohibit the spectral resolution matching to introduce an offset in spectral resolution required to match the relative shape of the resolution vectors. See mangadap.util.resolution.match_spectral_resolution() and mangadap.util.resolution.SpectralResolution.GaussianKernelDifference().
spectral_step (float, optional) – Logarithmic (log=True) or linear (log=False) step in wavelength for the template library.
log (bool, optional) – Flag to force the library to be logarithmically sampled in wavelength.
wavelength_range (array-like, optional) – Force the template library to cover this spectral range. Unobserved spectral regions will be masked.
renormalize (bool, optional) – After processing, renormalize the flux of the full library to unity. The relative fluxes of the templates are maintained.
output_path (str, Path, optional) – The path for the output file. If None, the current working directory is used.
output_file (str, optional) – The name of the output file. The full path of the output file will be directory_path/output_file. If None, the default is to combine cube.output_root and the library key; if cube is None, it is simply the library key.
read (bool, optional) – Flag to read the template library data.
process (bool, optional) – Process (spectral resolution and sampling operations) the template library. See process_template_library().
hardcopy (bool, optional) – Flag to keep a hardcopy of the processed template library.
symlink_dir (str, optional) – Create a symlink to the file in this directory. If None, no symbolic link is created.
overwrite (bool, optional) – Overwrite any saved, processed template library file.
checksum (bool, optional) – Use the checksum in the fits header to confirm that the data has not been corrupted. The checksum is always written to the fits header when the file is created.
loggers (list, optional) – List of logging.Logger objects to log progress; ignored if quiet=True. Logging is done using mangadap.util.log.log_output(). If None, no logging is performed and output is just written to stdout.
quiet (bool, optional) – Suppress all terminal and logging output.

bitmask (class: mangadap.util.bitmask.BitMask): Object used to toggle mask values; see TemplateLibraryBitMask().

library

Parameter set required to read and prepare the library.

Type:: TemplateLibraryDef

file_list

The list of files found using glob.glob and file_search.

Type:: list

ntpl

Number of template spectra in the library

Type:: int

cube

The datacube to be analyzed using the template library.

Type:: mangadap.datacube.datacube.DataCube

sres

The object is used simply to access the spectral resolution and associated wavelength coordinate vector needed when matching the spectral resolution of the template library; this is used in place of the attributes in any provided DRP file object.

Type:: mangadap.util.resolution.SpectralResolution

velocity_offset

Velocity offset to use when matching the spectral resolution between the template library and the galaxy spectra.

Type:: float

spectral_step

Target logarithmic (log10_sampling=True) or linear (log10_sampling=False) step in wavelength for the template library.

Type:: float

log10_sampling

Flag that the processed template library is logarithmically sampled in wavelength.

Type:: bool

directory_path

The exact path to the processed template library file.

Type:: str

output_file

The name of the file containing (to contain) the prepared template library output file. The file should be found at directory_path/output_file.

Type:: str

processed

Flag that the template library has been prepared for use in the DAP.

Type:: bool

hardcopy

Flag to keep a hardcopy of the processed template library.

Type:: bool

symlink_dir

Symlink created to the file in this directory

Type:: str

hdu

HDUList read from the DAP file

Type:: astropy.io.fits.HDUList

_build_raw_hdu(npix)[source]

Build the “raw” template library arrays.

This simply reads the provided list of fits files and puts them into arrays of size \(N_{\rm tpl} \times N_{\rm pix}\).

This will force reading of the data, even if the hdu is already initialized.

The hdu will contain the appropriate extensions, but it is important to note that the wavelength vectors will not necessarily be the same. That is, reading of the raw template spectra can accommodate spectra that have different wavelength coordinates. Any pixels that have no data are masked using the ‘NO_DATA’ bitmask flag; see TemplateLibraryBitMask().

The spectral resolution is set using the fwhm or sres_ext values in the library parameters.

Warning

Currently no errors are saved because none are expected for the template libraries.

Parameters:: npix (int) – Number of spectral channels for the output arrays

_get_nchannels()[source]

Get the maximum number of wavelength channels needed to store all the template spectra in a single array.

Todo

What happens if the spectrum has an empty primary extension?

Returns:: Maximum number of pixels used by the listed spectra.
Return type:: int
Raises:: ValueError – Raised if the input template spectra are not one-dimensional.

_minimum_sampling()[source]

Return the minimum sampling of the available wavelength vectors.

Returns:: Minimum sampling of all (will just be one if the library has been processed) wavelength vectors.
Return type:: float

_modify_spectral_resolution()[source]

Modify the spectral resolution to match the provided sres.

Returns:: Redshift used when matching the spectral resolution.
Return type:: float

_process_library(wavelength_range=None, renormalize=True)[source]

Process the template library for use in analyzing object spectra. See process_template_library().

Todo

Add wavelength coordinate WCS information to the appropriate extension headers.
Include ‘step’ as an argument
Include limit on covering fraction used to mask data as an argument

Parameters:

wavelength_range (array-like, optional) – A two-element object with the lower and upper limits for the wavelength range of the library.
renormalize (bool, optional) – Renormalize the flux of the full library to unity. Relative fluxes of spectra in the library are maintained.

_read_raw()[source]

Read the ‘raw’ versions of the template library.

The “raw” library is the data before its sampling or resolution have been altered.

This is a simple wrapper for _get_file_list(), _get_channels() and _build_raw_hdu().

_rebin_masked(i, flag, fullRange, rmsk_lim=0.5)[source]

Determine the mask value to adopt for a rebinned spectrum by rebinning the mask pixels, setting a masked pixel to unity, and an unmasked pixel to zero. After rebinning, any pixel with a value of larger than rmsk_lim is masked; otherwise it is left unmasked.

Although the function can be used with multiple flags, its intended use is to determine which pixels should be masked with a specific flag.

Parameters:

i (int) – Index of the spectrum to be rebinned.
flag (str, list) – Flags to consider when determining which pixels to mask; see mangadap.util.bitmasks.BitMask.flagged().
fullRange (numpy.ndarray) – Two-element array with the wavelength range for the rebinned spectrum.
rmsk_lim (float) – Limit of the rebinned mask value that is allowed before considering the pixel as masked.

Returns:

Boolean array indicating which pixels in the rebinned spectrum should be masked.

Return type:

numpy.ndarray

_reset_hdu(wave, flux, mask, sres, soff)[source]

(Re)Set hdu to a new HDUList object using the input arrays. Also sets the header items indicating the version of the template reader and the keyword for the library.

Warning

No checking is done concerning the size of each array!

Parameters:

wave (numpy.ndarray) – Array with the wavelengths of each pixel.
flux (numpy.ndarray) – Array with the flux in each pixel.
mask (numpy.ndarray) – Bitmask values for each pixel.
sres (numpy.ndarray) – Spectral resolution, \(R=\lambda/\delta\lambda\), at each pixel.
soff (numpy.ndarray) – The spectral resolution offset for each spectrum (see mangadap.util.resolution.GaussianKernelDifference()).

_wavelength_range(flag=None)[source]

Return the valid wavelength range for each spectrum based on the first and last unmasked pixel; interspersed masked regions are not considered.

Parameters:: flag (str, list, optional) – Flags to consider when determining the wavelength range; see mangadap.util.bitmasks.BitMask.flagged().
Returns:: Two-element vector with wavelengths of the first and last valid pixels.
Return type:: numpy.ndarray

static default_paths(library_key, cube=None, output_path=None, output_file=None)[source]

Set the default directory and file name for the output file.

Parameters:

library_key (str) – Unique library keyword.
cube (mangadap.datacube.datacube.DataCube, optional) – Datacube to analyze. The output is specific to the cube if the spectral resolution and spectral sampling are matched to the cube data. If None, the output file name is simply the library key.
output_path (str, Path, optional) – The path for the output file. If None, the current working directory is used.
output_file (str, optional) – The name of the output file. The full path of the output file will be directory_path/output_file. If None, the default is to combine cube.output_root and the library key.

Returns:

Returns a Path with the output directory and a str with the output file name.

Return type:

tuple

file_name()[source]: Return the name of the processed file.

file_path()[source]: Return the full path to the processed file.

process_template_library(library=None, cube=None, match_resolution=True, velscale_ratio=None, sres=None, velocity_offset=0.0, min_sig_pix=0.0, no_offset=True, spectral_step=None, log=True, wavelength_range=None, renormalize=True, output_path=None, output_file=None, hardcopy=False, symlink_dir=None, overwrite=False, loggers=None, quiet=False)[source]

Process the template library for use in analyzing spectra.

Todo

Docs need to be updated
type checking
If a DRP file is provided, the processing to a logarithmic binning is done by default (log=True). But linearly sampled DRP data are available, so need to have mangadap.util.drpfits.DRPFits return the spectral sampling type.
Documentation needs updating!

Primary steps are to:

Read the raw 1D fits files; see _read_raw().

Convert the wavelengths to vacuum, if necessary; see pydl.goddard.astro.airtovac.

Mask wavelengths outside the rest wavelength range of the datacube spectra, due to need to extrapolate these values; see mangadap.util.bitmasks.HDUList_mask_wavelengths().

Match the spectral resolution of the template to that of the datacube (if match_resolution is True); see mangadap.util.resolution.match_spectral_resolution().

Mask the template pixels where the spectral resolution was too low to match to datacube; see mangadap.util.bitmasks.BitMask.turn_on().

Force a common wavelength range and sampling for all templates, where the sampling is forced to match the sampling of the DRP spectra; see mangadap.util.sampling.grid_pix(). The masks are appropriately resampled as well; see _rebin_masked().

Warning

The routine does not check that that an existing processed file or the existing object has been processed using the same datacube, velocity_offset, velscale, or sres input. If unsure, use overwrite=True.

Parameters:

library (str, TemplateLibraryDef, optional) – Keyword selecting the library to use or the full set of defining parameters.
cube (mangadap.datacube.datacube.DataCube, optional) – The datacube to be analyzed using the template library.
match_resolution (bool, optional) – Match the spectral resolution of the template library to the resolution provided by the cube; the latter must be provided for this argument to have any use.
velscale_ratio (int, optional) – Resample the template spectra such that the ratio of the pixel scale in the provided cube is this many times larger than the pixels in the resampled template spectrum.
sres (mangadap.util.resolution.SpectralResolution, optional) – The object is used simply to access the spectral resolution and associated wavelength coordinate vector needed when matching the spectral resolution of the template library; this is used in lieu of a datacube.
velocity_offset (float, optional) – Velocity offset to use when matching the spectral resolution between the template library and the galaxy spectra.
min_sig_pix (float, optional) – Minimum value of the LSF standard deviation in pixels allowed before assuming the kernel is a Delta function. See mangadap.util.resolution.match_spectral_resolution().
no_offset (bool, optional) – Prohibit the spectral resolution matching to introduce an offset in spectral resolution required to match the relative shape of the resolution vectors. See mangadap.util.resolution.match_spectral_resolution() and mangadap.util.resolution.SpectralResolution.GaussianKernelDifference().
spectral_step (float, optional) – Logarithmic (log=True) or linear (log=False) step in wavelength for the template library.
log (bool, optional) – Flag to force the library to be logarithmically sampled in wavelength.
wavelength_range (array-like, optional) – Force the template library to cover this spectral range. Unobserved spectral regions will be masked.
renormalize (bool, optional) – After processing, renormalize the flux of the full library to unity. The relative fluxes of the templates are maintained.
output_path (str, Path, optional) – The path for the output file. If None, the current working directory is used.
output_file (str, optional) – The name of the output file. The full path of the output file will be directory_path/output_file. If None, the default is to combine cube.output_root and the library key; if cube is None, it is simply the library key.
hardcopy (bool, optional) – Flag to keep a hardcopy of the processed template library.
symlink_dir (str, optional) – Create a symlink to the file in this directory. If None, no symbolic link is created.
overwrite (bool, optional) – Overwrite any saved, processed template library file.
loggers (list, optional) – List of logging.Logger objects to log progress; ignored if quiet=True. Logging is done using mangadap.util.log.log_output(). If None, no logging is performed and output is just written to stdout.
quiet (bool, optional) – Suppress all terminal and logging output.

Raises:

ValueError – If the sampling is undefined, or if a hardcopy is requested but the output file could not be defined.

read_raw_template_library(library=None)[source]

Read the identified template library. If all the arguments are the default, the preset attributes from the initialization of the object are used.

Parameters:: library (str, TemplateLibraryDef, optional) – Keyword selecting the library to use or the full set of defining parameters.

single_spec_to_fits(i, ofile, overwrite=True)[source]

Write one of the template spectra to a 1D fits file.

Output is a one-dimensional fits file with a single extension containing the flux measurements and header keywords needed to determine the wavelength of each pixel.

Parameters:

i (int) – Index of the spectrum in the avaiable list to output.
ofile (str) – Name of the file to write
overwrite (bool, optional) – Overwrite any existing file.

Raises:

ValueError – Raised if the selected index is not available.

supported_libraries = ['BC03', 'BPASS', 'M11ELODIE', 'M11MARCS', 'M11MILES', 'M11STELIB', 'M11STELIBZSOL', 'MASTARHC', 'MASTARHC2', 'MASTARSSP', 'MILES', 'MILESAVG', 'MILESHC', 'MILESTHIN', 'MIUSCAT', 'MIUSCATTHIN', 'STELIB']: Provides the keywords of the supported libraries.

class mangadap.proc.templatelibrary.TemplateLibraryBitMask[source]

Bases: DAPBitMask

Derived class that specifies the mask bits for the template library data. The maskbits defined are:

Key	Bit	Description
NO_DATA	0	Pixel has no data
WAVE_INVALID	1	Used to designate pixels in the 1D spectra that are outside the valid wavelength range defined by default_template_libraries().
FLUX_INVALID	2	Used to designate pixels in the 1D spectra that are below the valid flux limit defined by default_template_libraries().
SPECRES_EXTRAP	3	The spectral resolution has been matched to a value that was an extrapolation of the target spectral resolution samples.
SPECRES_LOW	4	The spectral resolution was not matched to the target value because the target value was higher than the existing spectral resolution.
SPECRES_2PIXEL	5	Resampling has lead to the resolution being below the two pixel limit. The spectral resolution has been changed to be exactly the two pixel limit for these pixels.
SPECRES_NOFLUX	6	Resolution matching has lead a template flux <= 0.

cfg_root = 'spectral_template_bits'

class mangadap.proc.templatelibrary.TemplateLibraryDef(key='MILESHC', file_search='miles_cluster/*.fits', fwhm=2.5, sres_ext=None, in_vacuum=False, wave_limit=None, lower_flux_limit=None, log10=False)[source]

Bases: KeywordParSet

Class with parameters used to define the template library. See mangadap.par.parset.ParSet for attributes.

The defined parameters are:

Key	Type	Default	Description
`key`	str	`MILESHC`	Keyword to distinguish the template library.
`file_search`	str	`miles_cluster/*.fits`	Search pattern used to find the 1D fits spectra to include in the template library. The search string must either include the full path or be within the template directory of the DAP source distribution.
`fwhm`	int, float	2.5	FWHM of the resolution element in angstroms.
`sres_ext`	str		Extension in the fits files with measurements of the spectral resolution as a function of wavelength. If present, this supercedes any provided FWHM.
`in_vacuum`	bool	False	Flag that the wavelengths of the spectra are in vacuum, not air.
`wave_limit`	ndarray, list		Two-element array with the starting and ending wavelengths for the valid spectral range of the templates.
`lower_flux_limit`	int, float		Minimum valid flux in the template spectra.
`log10`	bool	False	Flag that the template spectra have been binned logarithmically in wavelength.

_validate()[source]: Validate the template library parameters.

classmethod from_dict(d)[source]

Instantiate from a dictionary.

If the dictionary only contains the keyword for the template library, the function assumes the user is trying to define the library based on that keyword. Otherwise, the function calls the base-class from_dict method.

classmethod from_key(key)[source]: Instantiate the template parameters using a keyword only.

get_file_list()[source]

Use the search string to find the template library fits files.

The search string must be via the default location in the DAP distribution or the full path to a local library.

The file list read by the search key is sorted for consistency.

Returns:: The sorted list of files.
Return type:: list
Raises:: ValueError – Raised if no files are found.