mangadap.proc.spatiallybinnedspectra module

A class hierarchy that perform that spatially bins a set of two-dimensional data.

The base class allows for user-defined definitions of binning procedures.

class mangadap.proc.spatiallybinnedspectra.SpatiallyBinnedSpectra(method, cube, rdxqa, reff=None, ebv=None, output_path=None, output_file=None, hardcopy=True, symlink_dir=None, overwrite=False, checksum=False, loggers=None, quiet=False)[source]

Bases: object

Class that holds spatially binned spectra.

Parameters:

method (SpatiallyBinnedSpectra) – Object that defines the main parameters used for the spatial binning.
cube (mangadap.datacube.datacube.DataCube) – The datacube with the spectra to bin.
rdxqa (mangadap.proc.reductionassessments.ReductionAssessments) – The basic assessments of the datacube that are used for the binning procedures.
reff (float, optional) – The effective radius of the galaxy in arcsec.
output_path (str, Path, optional) – The path for the output file. If None, the current working directory is used.
output_file (str, optional) – The name of the output “reference” file. The full path of the output file will be directory_path/output_file. If None, the default is to combine cube.output_root and the method keys of both rdxqa and the one for this class. The order of the keys is the order of operations (rdxqa key, then the binning key). See default_paths().
hardcopy (bool, optional) – Flag to write the astropy.io.fits.HDUList (hdu) to disk. If False, the object data is only kept in memory.
symlink_dir (str, optional) – Create a symbolic link to the created file in the supplied directory. If None, no symbolic link is created.
overwrite (bool, optional) – Overwrite any existing files. If False, any existing files will be used. If True, the analysis is redone and any existing output is overwritten.
checksum (bool, optional) – Use the checksum in the fits header to confirm that the data has not been corrupted. The checksum is always written to the fits header when the file is created.
loggers (list, optional) – List of logging.Logger objects to log progress; ignored if quiet=True. Logging is done using mangadap.util.log.log_output(). If None, no logging is performed and output is just written to stdout.
quiet (bool, optional) – Suppress all terminal and logging output.

loggers

List of logging.Logger objects to log progress; ignored if quiet=True. Logging is done using mangadap.util.log.log_output().

Type:: list

quiet

Suppress all terminal and logging output.

Type:: bool

Todo

Allow velocity offsets for registration.
Fill in attributes.

_add_method_header(hdr)[source]

Add method-specific metadata to the header.

Note that the header object is both edited in-place and returned.

Parameters:: hdr (astropy.io.fits.Header) – Input base header for added keywords. Expected to have been initialized using _initialize_primary_header().
Returns:: Edited header object.
Return type:: astropy.io.fits.Header

_add_reddening_header(hdr)[source]

Set the relevant reddening information to the header.

Note that the header object is both edited in-place and returned.

Parameters:: hdr (astropy.io.fits.Header) – Input header object to edit.
Returns:: Edited header object.
Return type:: astropy.io.fits.Header

_apply_reddening(flux, ivar, sdev, covar, deredden=True)[source]

Correct the spectra for Galactic reddening.

Largely a wrapper for executing mangadap.util.extinction.GalacticExtinction.apply() on the provided arrays.

If the reddening law is undefined (galext.form is None), the method simply returns the input.

Otherwise, the input arrays are both modified directly and returned.

Parameters:

flux (numpy.ndarray) – Flux array
ivar (numpy.ndarray) – Inverse variance array. Can be None.
sdev (numpy.ndarray) – The standard deviation of the stacked spectra, if relevant to flux. Can be None.
covar (mangadap.util.covariance.Covariance) – Spatial covariance in the binned spectra. Assumed to be a 3D covariance cube. Can be None.

Returns:

Returns four numpy.ndarray objects with the reddening corrected flux, inverse variance, standard deviation in the stack, and spatial covariance. Any of the latter three can be None if the corresponding input is None.

Return type:

tuple

_assign_image_arrays()[source]: Set image_arrays, which contains the list of extensions in hdu that are on-sky image data.

_assign_spectral_arrays()[source]: Set spectral_arrays, which contains the list of extensions in hdu that contain spectral data.

_binned_data_table(bin_indx, stack_flux, stack_ivar, per_pixel=True)[source]

Construct the output data table for the binned spectra.

Parameters:

bin_indx (numpy.ndarray) – The integer vector with the bin associated with each spectrum in the DRP cube. This is the flattened BINID array.
stack_flux (numpy.ndarray) – The stacked spectra with shape \(N_{\rm spec}\times\N_\lambda}\).
stack_ivar (numpy.ndarray) – The stacked inverse variance with shape \(N_{\rm spec}\times\N_\lambda}\).
per_pixel (bool, optional) – Base the flux statistics on per-pixel measurements. Set to False for a per-angstrom calculation.

Returns:

The record array that is put in the BINS extension of hdu.

Return type:

numpy.recarray

_check_snr()[source]

Determine which spectra in rdxqa have a S/N greater than the minimum set by method. Only these spectra will be included in the binning.

Returns:: Boolean array for the spectra that satisfy the criterion.
Return type:: numpy.ndarray

_construct_2d_hdu(bin_indx, good_fgoodpix, good_snr, bin_data, stack_flux, stack_sdev, stack_ivar, stack_npix, stack_mask, stack_sres, stack_covar)[source]

Construct hdu that is held in memory for manipulation of the object. See construct_3d_hdu() to convert the object into a datacube.

Parameters:

bin_indx (numpy.ndarray) – 2D array with the bin associated with each spaxel.
good_fgoodpix (numpy.ndarray) – Boolean array selecting spaxels with good spectral coverage.
good_snr (numpy.ndarray) – Boolean array selecting spaxels with good S/N.
bin_data (SpatiallyBinnedSpectraDataTable) – Data table with relevant metadata for each binned spectrum.
stack_flux (numpy.ndarray) – Array with the stacked spectral flux. Shape is \((N_{\rm bin}, N_{\rm wave})\).
stack_sdev (numpy.ndarray) – Array with the standard deviation in the stacked spectral flux. Shape is \((N_{\rm bin}, N_{\rm wave})\).
stack_ivar (numpy.ndarray) – Array with the inverse variance in the stacked spectral flux. Shape is \((N_{\rm bin}, N_{\rm wave})\).
stack_npix (numpy.ndarray) – Integer array with the number of spectral channels that were stacked to construct the binned spectrum. Shape is \((N_{\rm bin}, N_{\rm wave})\).
stack_mask (numpy.ndarray) – Integer array with the bitmasks associated with pixel in the stacked spectra. Shape is \((N_{\rm bin}, N_{\rm wave})\).
stack_sres (numpy.ndarray) – Array with the spectral resolution of the stacked spectra. Shape is \((N_{\rm bin}, N_{\rm wave})\). If None, the spectral resolution is set to be the median of the resolution in the datacube. If the datacube also has no spectral resolution data, the method faults.
stack_covar (mangadap.util.covariance.Covariance) – Spatial covariance in the stacked spectra. Can be None.

_finalize_cube_mask(mask)[source]

Finalize the mask after the 2D mask has been reconstructed into a 3D cube.

This mostly handles the masks for regions outside the datacube field of view.

This propagates the bits flagging pixels that should not be included in the stacked spectra, and sets the the LOW_SPECCOV and LOW_SNR bits based on the results from check_fgoodpix() and _check_snr().

Note that the input mask is both edited in-place and returned.

Todo

This needs to be abstracted for non-DRP datacubes.

Parameters:: mask (numpy.ndarray) – 3D array with the current bitmask data.
Returns:: Edited bitmask data.
Return type:: numpy.ndarray

static _get_missing_bins(unique_bins)[source]: Return bin IDs omitted from a sequential list.

_initialize_primary_header(hdr=None)[source]

Construct the primary header for the reference file.

Parameters:: hdr (astropy.io.fits.Header, optional) – Input base header for added keywords. If None, uses the cube header (if there is one) and then cleans the header using mangadap.util.fitsutil.DAPFitsUtil.clean_dap_primary_header().
Returns:: Initialized header object.
Return type:: astropy.io.fits.Header

_unbinned_data_table(bin_indx)[source]

Construct the output data table for the unbinned spectra.

Parameters:: bin_indx (numpy.ndarray) – The integer vector with the bin associated with each spectrum in the DRP cube. This is the flattened BINID array.
Returns:: The record array that is put in the BINS extension of hdu.
Return type:: numpy.recarray

above_snr_limit(sn_limit, original_spaxels=False, debug=False)[source]

Flag bins above a provided S/N limit.

Parameters:

sn_limit (float) – S/N threshold.
original_spaxels (bool, optional) – Instead of returning flags for the binned spectra S/N, return flags for the original spaxels. Note the difference in the returned shape!!
debug (bool, optional) – Run in debug mode. This just selects the first 2 spectra that meet the S/N criterion so that the rest of the analysis is just done using those two bins.

Returns:

Boolean array selecting those bins that meet the S/N threshold.

Return type:

numpy.ndarray

bin_spectra(cube, rdxqa, reff=None, ebv=None, output_path=None, output_file=None, hardcopy=True, symlink_dir=None, overwrite=False, loggers=None, quiet=False)[source]

Bin and stack the spectra.

This is the core funtion of this class, constructing its main data container, hdu.

Todo

Describe algorithm.

Parameters:

cube (mangadap.datacube.datacube.DataCube) – The datacube with the spectra to bin.
rdxqa (mangadap.proc.reductionassessments.ReductionAssessments) – The basic assessments of the datacube that are used for the binning procedures.
reff (float, optional) – The effective radius of the galaxy in arcsec.
ebv (float, optional) – The E(B-V) Galactic reddening to be removed
output_path (str, Path, optional) – The path for the output file. If None, the current working directory is used.
output_file (str, optional) – The name of the output “reference” file. The full path of the output file will be directory_path/output_file. If None, the default is to combine cube.output_root and the method keys of both rdxqa and the one for this class. The order of the keys is the order of operations (rdxqa key, then the binning key). See default_paths().
hardcopy (bool, optional) – Flag to write the astropy.io.fits.HDUList (hdu) to disk. If False, the object data is only kept in memory.
symlink_dir (str, optional) – Create a symbolic link to the created file in the supplied directory. If None, no symbolic link is created.
overwrite (bool, optional) – Overwrite any existing files. If False, any existing files will be used. If True, the analysis is redone and any existing output is overwritten.
loggers (list, optional) – List of logging.Logger objects to log progress; ignored if quiet=True. Logging is done using mangadap.util.log.log_output(). If None, no logging is performed and output is just written to stdout.
quiet (bool, optional) – Suppress all terminal and logging output.

Raises:

ValueError – Raised if cube or rdxqa is None, or if no spectra in the datacube meet the S/N or spectral coverage criteria. If the spectra are actually being binned (i.e., the binning type is not ‘none’), this error is also raised if the output file cannot be defined, if no spectra are assigned to any bin, or if the stacking function results in a correlation matrix instead of a full covariance matrix.
TypeError – Raised if the input cube is not derived from mangadap.datacube.datacube.DataCube or if rdxqa is not a mangadap.proc.reductionassessments.ReductionAssessment.

check_fgoodpix(minimum_fraction=None)[source]

Determine which spaxels in rdxqa have a fractional spectral coverage of greater than the provided minimum fraction. Only these spectra will be included in the binning.

Parameters:: minimum_fraction (float, optional) – The minimum fraction of the spectrum that must be valid for the spectrum to be included in any bin. If None, self.method['minimum_frac'] is used.
Returns:: Boolean array for the spectra that satisfy the criterion. Shape is \((N_{\rm spaxel},)\).
Return type:: numpy.ndarray

construct_3d_hdu()[source]: Reformat the binned spectra into a cube matching the shape of the datacube from which it was derived.

copy_to_array(ext='FLUX', waverange=None, include_missing=False)[source]

Wrapper for mangadap.util.fitsutil.DAPFitsUtil.copy_to_array() specific for SpatiallyBinnedSpectra.

Return a copy of the selected data array. The array size is always \(N_{\rm bins} \times N_{\rm wavelength}\); i.e., the data is always flattened to two dimensions and the unique spectra are selected.

Parameters:

ext (str, optional) – Name of the extension from which to draw the data. Must be allowed for the current mode; see data_arrays.
waverange (array-like, optional) – Two-element array with the first and last wavelength to include in the computation. If None, use the full wavelength range.
include_missing (bool, optional) – Create an array with a size that accommodates the missing models.

Returns:

A 2D array with a copy of the data from the selected extension.

Return type:

numpy.ndarray

copy_to_masked_array(ext='FLUX', flag=None, waverange=None, include_missing=False)[source]

Wrapper for mangadap.util.fitsutil.DAPFitsUtil.copy_to_masked_array() specific for SpatiallyBinnedSpectra.

Return a copy of the selected data array as a masked array. This is functionally identical to copy_to_array(), except the output format is a numpy.ma.MaskedArray. The pixels that are considered to be masked can be specified using flag.

Parameters:

ext (str, optional) – Name of the extension from which to draw the data. Must be allowed for the current mode; see data_arrays.
flag (str, list, optional) – (List of) Flag names that are considered when deciding if a pixel should be masked. The names must be a valid bit name as defined by bitmask. If not provided, ANY non-zero mask bit is omitted.
waverange (array-like, optional) – Two-element array with the first and last wavelength to include in the computation. If None, use the full wavelength range.
include_missing (bool, optional) – Create an array with a size that accommodates the missing models.

Returns:

A 2D array with a copy of the data from the selected extension.

Return type:

numpy.ma.MaskedArray

static default_paths(cube, method_key, rdxqa_method, output_path=None, output_file=None)[source]

Set the default directory and file name for the output file.

Parameters:

cube (mangadap.datacube.datacube.DataCube) – Datacube to analyze.
method_key (str) – Keyword designating the method used for the reduction assessments.
rdxqa_method (str) – The method key for the basic assessments of the datacube.
output_path (str, Path, optional) – The path for the output file. If None, the current working directory is used.
output_file (str, optional) – The name of the output “reference” file. The full path of the output file will be directory_path/output_file. If None, the default is to combine cube.output_root and the method keys of both rdxqa and the one for this class. The order of the keys is the order of operations (rdxqa key, then the binning key).

Returns:

Returns a Path with the output directory and a str with the output file name.

Return type:

tuple

static do_not_fit_flags()[source]

file_name()[source]: Return the name of the output file.

file_path()[source]: Return the full path to the output file.

find_nearest_bin(input_bins, weighted=False, indices=False)[source]

Use a scipy.spatial.KDTree to find the bins nearest to and excluding a list of input bins.

Parameters:

input_bins (array-like) – One or more bin ID numbers to use to locate the bin nearest to it based on its on-sky coordinates. The list must be a unique set.
weighted (bool, optional) – Use the weighted coordinates (LW_SKY_COO) instead of the unweighted coordinates (SKY_COO) when finding the nearest bin.
indices (bool, optional) – Return the indices of the nearest bins instead of their ID number (default).

Returns:

The bin IDs, one per input bin, of the bin closest to each input bin. Any bins that are in missing_bins have a return value of -1; there are no coordinates for these bins.

Return type:

numpy.ndarray

Raises:

ValueError – Raised if the set of input bins is not unique or contains bin IDs that are not present.

get_bin_indices(bins)[source]

Return the indices of the bins in the BIN table.

Parameters:: bins (array-like) – The bin ID numbers to find
Returns:: Integer array with the index of each bin ID in the BINID columns of the BINS extension.
Return type:: numpy.ndarray

get_mask_array_from_cube(select=None)[source]

Convert the datacube mask for individual spaxels to the binned spectra mask.

Any pixel with bit flags in the list returned by mangadap.datacube.datacube.DataCube.do_not_stack_flags() for cube are consolidated into the DIDNOTUSE flag. Any FORESTAR flags are propagated.

Parameters:: select (numpy.ndarray, optional) – Flattened, boolean array with the spaxels in the datacube to select. Shape is \((N_{\rm spaxel},)\). If None, all spaxels are included.
Returns:: Integer array with the consolidated mask bits.
Return type:: numpy.ndarray

info()[source]

property is_unbinned: Determine if the spectra are unbinned.

read(ifile=None, strict=True, checksum=False)[source]

Read an existing file with a previously binned set of spectra.

Parameters:

ifile (str, optional) – Name of the file with the data. Default is to use the name provided by file_path().
strict (bool, optional) – Force a strict reading of the file to make sure that it adheres to the expected format. At the moment, this only checks to make sure the method keyword printed in the header matches the expected value in method.
checksum (bool, optional) – Use the checksum in the header to check for corruption of the data.

Raises:

FileNotFoundError – Raised if the file does not exist.
ValueError – Raised if strict is true and the header keyword BINKEY does not match the method keyword.

replace_with_data_from_nearest_bin(data, bad_bins)[source]

Replace data in the list of provided bad bins with the data from the nearest good bin.

Parameters:

data (array-like) – Data for each bin. The length must be the same as nbins.
bad_bins (array-like) – The list of indices (must not be a boolean array) with bad values to be replaced.

Returns:

A new array with the bad data filled with the data from the nearest bin.

Return type:

numpy.ndarray

Raises:

ValueError – Raised if the input array doesn’t have the correct shape or if the list of bad bins has numbers outside the viable range (0,self.nbins-1).

write(match_datacube=False, overwrite=False)[source]

Write the hdu object to the file.

Parameters:

match_datacube (bool, optional) – Match the shape of the data arrays to the input datacube. I.e., convert them to 3D and replicate the binned spectrum to each spaxel in the bin.
overwrite (bool, optional) – Overwrite any existing file.

class mangadap.proc.spatiallybinnedspectra.SpatiallyBinnedSpectraBitMask[source]

Bases: DAPBitMask

Derived class that specifies the mask bits for the spatially binned spectra. The maskbits defined are:

Key	Bit	Description
DIDNOTUSE	0	Pixel was ignored because it was flagged as either DONOTUSE or FORESTAR by the data reduction pipeline (DRP).
FORESTAR	1	Pixel was ignored because it was flagged as FORESTAR by the data reduction pipeline (DRP).
LOW_SPECCOV	2	Pixel was ignored because the fraction of valid spectral channels limited the spectral coverage below the set threshold; see header keyword FSPECCOV.
LOW_SNR	3	Pixel was ignored because the S/N estimate of the spectrum was below the set threshold; see header keyword BINMINSN.
NONE_IN_STACK	4	No valid pixels were available or valid for the stacked spectrum.
NO_STDDEV	5	Insufficient pixels to calculate the standard deviation in the stacked spectrum.
IVARINVALID	6	Pixel ignored because inverse variance invalid.

cfg_root = 'spatially_binned_spectra_bits'

class mangadap.proc.spatiallybinnedspectra.SpatiallyBinnedSpectraDataTable(shape=None)[source]

Bases: DataTable

Primary data table with the results of the spatial binning.

Table includes:

Key	Type	Description
BINID	`int64`	Bin ID number
NBIN	`int64`	Number of spaxels in the bin
SKY_COO	`float64`	The mean on-sky coordinates of the binned spaxels.
LW_SKY_COO	`float64`	The luminosity-weighted mean on-sky coordinates of the binned spaxels.
ELL_COO	`float64`	The mean elliptical coordinates of the binned spaxels.
LW_ELL_COO	`float64`	The luminosity-weighted mean elliptical coordinates of the binned spaxels.
AREA	`float64`	Total on-sky area of the bin
AREA_FRAC	`float64`	Fraction of the expected area covered by good spaxels (not relevant to all binning schemes)
SIGNAL	`float64`	Mean flux per pixel in the binned spectrum.
VARIANCE	`float64`	Mean variance per pixel in the binned spectrum.
SNR	`float64`	Mean S/N per pixel in the binned spectrum.

Todo

There currently is no mask; however, could add masks for:

Insufficient good wavelength channels in the spectrum.
Variance in flux in bin is too large.

Parameters:: shape (int, tuple, optional) – The shape of the initial array. If None, the data array will not be instantiated; use init() to initialize the data array after instantiation.

class mangadap.proc.spatiallybinnedspectra.SpatiallyBinnedSpectraDef(key='SPX', galactic_reddening='ODonnell', galactic_rv=3.1, minimum_snr=1.0, minimum_frac=0.8, binpar=None, binclass=None, binfunc=None, stackpar=None, stackclass=None, stackfunc=None, overwrite=False)[source]

Bases: KeywordParSet

A class that holds the two parameter sets and the key designator for the binning scheme.

The provided binfunc method must have a call with the following form:

id = bin(x, y, par=par)

where id is the index of the bin to place each spectrum and x and y are the on-sky Cartesian coordinates; e.g.:

x = ReductionAssessment.hdu['SPECTRUM'].data['SKY_COO'][:,0]
y = ReductionAssessment.hdu['SPECTRUM'].data['SKY_COO'][:,1]

The provided stackfunc method must have a call with the following form:

stack_wave, stack_flux, stack_sdev, stack_npix, stack_ivar,                 stack_sres, stack_covar = stack(cube, id, par=par)

where cube is a mangadap.datacube.datacube.DataCube object. Note that the wavelengths are not returned because the input and output wavelength ranges are expected to be the same!

As long as they are mutable, the values in par can change, meaning that some products of the bin algorithm can be passed to the stack algorithm. For example, if you want to weight the inclusion of the spectrum in the bin, you’d have to provide both the binning and stacking routines. Actually, if that’s the case, you’re better off providing a class object that will both bin and stack the spectra!

The defined parameters are:

Key	Type	Default	Description
`key`	str	`SPX`	Keyword used to distinguish between different spatial binning schemes.
`galactic_reddening`	str	`ODonnell`	The string identifier for the Galactic extinction curve to use. See `valid_forms()` for the available curves.
`galactic_rv`	int, float	3.1	Ratio of V-band extinction to the B-V reddening.
`minimum_snr`	int, float	1.0	Minimum S/N of spectra to include in any bin.
`minimum_frac`	int, float	0.8	Minimum fraction of unmasked pixels in each spectrum included in any bin.
`binpar`	ParSet, dict		The spatial-binning parameters.
`binclass`	Undefined		Instance of the spatial-binning class. Needed in case binfunc is a non-static member function of the class.
`binfunc`	Undefined		The spatial-binning function that determines which spectra go into each bin.
`stackpar`	ParSet, dict		The spectral-stacking parameter set.
`stackclass`	Undefined		Instance of spectral-stacking class to use. Needed in case stackfunc is a non-static member function of the class.
`stackfunc`	Undefined		The spectral-stacking function that stacks the spectra in a given bin.
`overwrite`	bool	False	If the output file already exists, redo all the calculations and overwrite it.

classmethod from_dict(d)[source]

Instantiate the object from a nested dictionary.

The dictionary can have any of the keywords that are identical to the instantiation keyword arguments, except for the spatial binning and spectral stacking parameters. These are assumed to be construced from a set of nested dictionaries. The keywords/nested dictionaries used to define these parameters are:

'spatial_method': selects a string that identifies the spatial binning method. If not present, no spatial binning is performed.

'spatial': selects a nested dictionary with the parameters needed to instantiate the spatial binning method. May not be necessary for some binning methods. If not present, will use defaults for binning method selected.

'spectral': selects a nested dictionary with spectral stacking parameters. If not present, will use defaults.

Parameters:: d (dict) – Dictionary used to instantiate the class.