mangadap.survey.drpcomplete module
Defines a class used to produce and provide an interface for data required to create The DAP Datacube Configuration File.
The drpcomplete file/data is primarily required for the survey-level Execution of the DAP. However, it does provide useful information regarding the completed DRP files.
Copyright © 2019, SDSS-IV/MaNGA Pipeline Group
- class mangadap.survey.drpcomplete.DRPComplete(platelist=None, ifudesignlist=None, drpall=None, platetargets=None, catid=None, drpver=None, redux_path=None, dapver=None, analysis_path=None, directory_path=None, readonly=False)[source]
Bases:
object
Database with information needed by the DAP to analyze the completed DRP files.
This class searches the defined paths for files resulting from the 3D phase of the MaNGA DRP, and then collates the information necessary to run those files through the MaNGA DAP. The necessary parameters are pulled from the provided platetargets files; see
update()
.- Parameters
platelist (
str
,list
, optional) – List of plates to search for. Default is to search the full DRP path.ifudesignlist (
str
,list
, optional) – List of ifudesign to search for. Default is to search the full DRP path.drpall (
str
, optional) – The full path to the DRPall fits file. Default is set bymangadap.config.defaults.drpall_file()
.platetargets (
str
,list
, optional) – List of platetargets files to search through to find any given plate ifudesign combination. Default is returned as the first element inmangadap.config.defaults.plate_target_files()
.catid (
str
,list
, optional) – List of target catalog ID numbers. Default is returned as the second element inmangadap.config.defaults.plate_target_files()
.drpver (
str
, optional) –DRP version, which is:
used to define the default DRP redux path
used when declaring a drpfits instance
used in the name of the drpcomplete fits file
included as a header keyword in the output file
Default is defined by
mangadap.config.defaults.drp_version()
.redux_path (
str
, optional) – The path to the top level directory containing the DRP output files; this is the same as theredux_path
in themangadap.util.drpfits.DRPFits
class. Default is defined bymangadap.config.defaults.drp_redux_path()
.dapver (
str
, optional) –DAP version, which is:
used to define the default DAP analysis path
included as a header keyword in the output drpcomplete fits file
Default is defined by
mangadap.config.defaults.dap_version()
.analysis_path (
str
, optional) – The path to the top level directory for the DAP output files; this is different from the directory_path in themangadap.dapfile
class. Default is defined bymangadap.config.defaults.dap_analysis_path()
directory_path (
str
, optional) – Direct path to the output file produced usingmangadap.config.defaults.dap_common_path()
readonly (
bool
, optional) – Flag that the drpcomplete fits file is only opened for reading, not for updating. If True, many of the attributes will be set to None.
- platelist
List of plates to search for, see above.
- Type
list
- ifudesignlist
List of IFU designs to search for, see above.
- Type
list
- drpall
The DRPall file, see above.
- Type
str
- platetargets
List of platetargets files to search through, see above.
- Type
numpy.ndarray
- catid
List of target catalog IDs, see above
- Type
numpy.ndarray
- drpver
DRP version, see above.
- Type
str
- redux_path
Path to the top level of directory for the DRP output, see above.
- Type
str
- dapver
DAP version, see above.
- Type
str
- analysis_path
Path to the top level directory for the DAP output files, see above.
- Type
str
- directory_path
Direct path to the output file produced using
mangadap.config.defaults.dap_common_path()
- Type
str
- readonly
Flag that the drpcomplete fits file is only opened for reading, not for updating.
- Type
bool
- hdu
Fits data with binary table data.
- nobs
Number of observations in the file
- Type
int
- _all_data_exists()[source]
Determines if the data for all the plates/ifudesigns selected in the current compilation is already present in the current drpcomplete fits file.
- Returns
Flag that all data has already been collated for the requested plate/ifudesign list.
- Return type
bool
- _confirm_access(reread=False)[source]
Check the drpcomplete fits file at
file_path()
is accessible, and read the data if not yet read.- Parameters
reread (
bool
, optional) – Force the file to be re-read- Returns
Flag that
hdu
is read and valid.- Return type
bool
- _find_completed_reductions(mindesign=19, combinatorics=False, on_disk=False)[source]
Search the DRP path for reduced CUBE files.
Function allows for one or both of
platelist
andifudesignlist
to be None. The behavior is:If both are None, all available CUBE files are used to create
platelist
andifudesignlist
.If one is None, all available CUBE files within the provided list of the one that is not None are used to create
platelist
andifudesignlist
. E.g., ifplatelist
=[7443] andifudesignlist
is None, all CUBE files with plate=7443 are chosen.If both are not None and they have different lengths, or the same length and combinatorics is True, all available CUBE files within the constraints of both lists are selected. E.g., if
platelist
=[7443,7495] andifudesignlist
=[12704], the CUBE files with (plate,ifudesign)=[(7443,12704),(7459,12704)] are chosen.If both are not None and they have the same length and combinatorics is False, all available CUBE files in the matched lists are chosen. E.g. if
platelist
=[7443,7495] andifudesignlist
=[12704,12703], the CUBE files with (plate,ifudesign)=[(7443,12704),(7459,12703) are chosen
- Parameters
mindesign (
int
, optional) – Minimum bundle design to consider. For example, to ignore all the 7-fiber bundles, set mindesign=19 (default) to only select the bundles with 19 or more fibers.combinatorics (
bool
, optional) – Createplatelist
andifudesignlist
by determining all possible combinations of the input values. See above.on_disk (
bool
, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file.
- Returns
Two lists with the available plates and ifudesigns for which to collect data.
- Return type
list
- _find_modes()[source]
Using the provided list of CUBE DRP files, find if the RSS mode is also available.
Warning
This assumes everything in
platelist
andifudesignlist
has a ‘CUBE’ file.
Currently only two modes are possible:
only ‘CUBE’ file are available
both ‘CUBE’ and ‘RSS’ files are available
- Returns
Array of modes for each input DRP file.
- Return type
numpy.ndarray
- _match_drpall()[source]
Find the data needed for the DRP complete databas in the DRPall file.
If a plate-ifudesign combination is not found in the DRPall file, the other parameter values are set to -9999 and the MaNGA ID is set to NULL.
If the plate-ifudesign combination is found, the columns in the plateTargets files that the code expects to find are ‘plate’, ‘ifudsgn’, ‘mangaid’, ‘objra’, ‘objdec’; the values that will be replaced with ‘NULL’ (str) or -9999 (int or float) if they do not exist are ‘nsa_version’, ‘nsa_nsaid’, ‘mngtarg1’, ‘mngtarg3’, ‘z’, ‘nsa_elpetro_ba’, ‘nsa_elpetro_phi’, ‘nsa_elpetro_th50_r’, ‘nsa_vdisp’.
- Returns
14 arrays with: MaNGA ID, object right ascension, object declination, catalog ID, index of the entry in the catalog, catalog version, ID of object in the catalog, main MaNGA survey target bitmask, ancillary MaNGA survey target bitmask, velocity, velocity dispersion, ellipticity, position angle, and effective radius.
- Return type
numpy.array
- _match_platetargets(quiet=True)[source]
Read the platetargets files and match the data therein to the completed DRP files based on the plate and ifudesign.
If a plate-ifudesign combination is not found in the plateTargets file, the other parameter values are set to -9999 and the MaNGA ID is set to NULL.
If the plate-ifudesign combination is found, the columns in the plateTargets files that the code expects to find are ‘plateid’, ‘ifudesign’, ‘mangaid’, ‘object_ra’, and ‘object_dec’; the values that will be replaced with ‘NULL’ (str) or -9999 (int or float) if they do not exist are ‘nsa_version’, ‘nsa_nsaid’, ‘manga_target1’, ‘manga_target3’, ‘z’ or ‘nsa_z’, ‘nsa_elpetro_ba’, ‘nsa_elpetro_phi’, ‘nsa_elpetro_th50_R’, ‘nsa_vdisp’.
Todo
Instead of searching through all files for the correct plate-ifudesign, use the MaNGA ID to get the correct catalog? Requires getting the MaNGA ID from somewhere, but it takes a long time to read from the fits files…
Is there some more concise method of performing the same thing as what is done below with many try/except blocks?
- Parameters
quiet (
bool
, optional) – Suppress terminal output- Returns
14 arrays with: MaNGA ID, object right ascension, object declination, catalog ID, index of the entry in the catalog, catalog version, ID of object in the catalog, main MaNGA survey target bitmask, ancillary MaNGA survey target bitmask, velocity, velocity dispersion, ellipticity, position angle, and effective radius.
- Return type
numpy.array
- _read()[source]
Read the data in the existing file at
file_path()
.
- _read_fix_data(fix_file)[source]
Read fix data from the provided file.
- Parameters
fix_file (
str
) – SDSS parameter file with the fix data.- Returns
A yanny structure with the data fixes. Returns None and raises a warning if the file does not exist.
- Return type
yanny
- _read_photometry_fix()[source]
Wrapper for
_read_fix_data()
andphotometry_fix_file()
that returns the photometry fix data.
- _read_platetargets()[source]
Read all the platetargets files using pydl.pydlutils.yanny and return a list of yanny structures.
- Returns
A list of yanny structures, one per platetargets file.
- Return type
list
- Raises
FileNotFoundError – Raised if cannot open one or more platetargets files.
Todo
This should be made more efficient by collating the required plateTargets data into a single record array!
- _read_redshift_fix()[source]
Wrapper for
_read_fix_data()
andredshift_fix_file()
that returns the redshift fix data.
- _write_parameter_list(ostream, index, mode)[source]
Write a DAPPAR entry to the SDSS-style parameter file.
- Parameters
ostream (_io.TextIOWrapper) – File stream for output
index (int) – Index of the entry in
data
to writemode (int) – Mode of the entry; see
_find_modes()
- _write_parameter_struct(ostream)[source]
Write the structure of the DAP SDSS-style parameter set to a file.
- Parameters
ostream (_io.TextIOWrapper) – File stream for output
- can_analyze(row=None)[source]
Determine if the DAP can analyze a plate-ifu entry in the database.
The selection is currently:
MaNGAID != ‘NULL’
MANGA_TARGET1 > 0 or MANGA_TARGET3 > 0
VEL > -500
- Parameters
row (
int
, optional) – The specific row to test. By default, return a boolean vector for all the database rows.- Returns
Either a single boolean or boolean numpy.ndarray flagging that DAP can (True) or cannot (False) analyze the data associated with the database entry (or entries).
- entry_index(plate, ifudesign, reread=False)[source]
Find the index of the row with the parameter data for the specified plate and ifudesign.
Warning
This is very inefficient if you’re looking for multiple entries…
- Parameters
plate (
int
) – Plate numberifudesign (
int
) – IFU designreread (
bool
, optional) – Force the database to be re-read
- Returns
Index of the row in
data
with the data for the given plate and ifudesign- Return type
int
- Raises
ValueError – Raised if the given plate and ifudesign were not found.
- update(platelist=None, ifudesignlist=None, combinatorics=False, force=False, alldrp=False, use_platetargets=False, on_disk=False, quiet=False)[source]
Update the DRP complete file.
If platelist and/or ifudesignlist are provided, the existing
platelist
andifudesignlist
attributes are replaced.If platelist and ifudesignlist do not have the same length or combinatorics is True, the lists are expanded to include all combinations of their elements.
If platelist and ifudesignlist are None or alldrp is True, all available plates/ifudesigns are collected from within the DRP directory structure.
If the result of
file_path()
does not exist, it is created.If the result of
file_path()
does exist, the available plates and ifudesigns in the file are compared against the list (provided or collected) to update. If the lists differ, the drpcomplete fits file is re-created from scratch. If all the plates and ifudesigns are available, nothing is done, unless force=True.- Parameters
platelist (
str
,list
, optional) – List of plates to include in the drpcomplete fits file.ifudesignlist (
str
,list
, optional) – List of ifudesigns to include in the drpcomplete fits file.combinatorics (
bool
, optional) – Determine all combinations of the entered plates and ifudesigns.force (
bool
, optional) – Overwrite any existing drpcomplete fits file with a new one built from scratch.alldrp (
bool
, optional) – Find the full list of available DRP files.use_platetargets (
bool
, optional) – Generate the data using the platetargets files instead of the DRPall file.on_disk (
bool
, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file.quiet (
bool
, optional) – Suppress terminal output
- Raises
ValueError – Raised if drpcomplete fits file was opened in read-only mode.
- write(platelist, ifudesignlist, modes, mangaid, objra, objdec, catid, catindx, trg_version, trg_id, manga_trg1, manga_trg3, vel, veldisp, ell, pa, Reff, drpver=None, redux_path=None, dapver=None, analysis_path=None, overwrite=True)[source]
Write the drpcomplete fits binary table.
- Header keywords are:
VERSDRP: DRP version
RDXPTH: DRP reduction path
VERSDAP: DAP version
SISPTH: DAP analysis path
PLTTRG N: plateTargets file N
CATID N: ID Number of target catalog N
AUTHOR: Set to ‘K.B. Westfall <kbwestfall@gmail.com>’
- Binary table columns:
PLATE (1J): Plate number
IFUDESIGN (1J): IFU design
- MODES (1B): Modes available
MODES=1: only ‘CUBE’ file is available
MODES=2: both ‘CUBE’ and ‘RSS’ files are available
MANGAID (n A): MaNGA ID (same as ‘CATID-CATINDX’)
OBJRA (1D): Object right ascension
OBJDEC (1D): Object declination
CATID (1J): Catalog ID used for target selection
CATINDX (1J): Index in catalog with target data
- TRG_VERSION (n A): Version of the catalog (e.g., NSA)
used in targetting (def=’NULL’ if not available)
TRG_ID (1J): Target ID in catalog (def=-9999 if not available)
MANGA_TARGET1: Main MaNGA survey target bitmask
MANGA_TARGET3: Ancillary MaNGA survey target bitmask
VEL (1D): Systemic velocity from catalog (def=-9999.0 if not available)
VDISP (1D): Velocity dispersion from catalog (def=-9999.0 if not available)
ELL (1D): Ellipticity (1-b/a) from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
PA (1D): Position angle from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
REFF (1D): Effective radius from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
- Parameters
platelist (list) – List of plates
ifudesignlist (list) – List of IFU designs
modes (numpy.array) – Mode values, see above
mangaid (numpy.array) – MaNGA IDs
objra (numpy.array) – Object right ascensions
objdec (numpy.array) – Object declinations
catid (numpy.array) – Catalog ID used for target selection
catindx (numpy.array) – Index in catalog with target data
trg_version (numpy.array) – Version of the catalog (e.g., NSA) used in targetting
trg_id (numpy.array) – Target ID in catalog
manga_trg1 (numpy.array) – Main MaNGA survey target bitmask
manga_trg3 (numpy.array) – Ancillary MaNGA survey target bitmask
vel (numpy.array) – Velocity from catalog, if available
veldisp (numpy.array) – Velocity dispersion from catalog, if available
ell (numpy.array) – Ellipticity (1-b/a) from catalog, if available
pa (numpy.array) – Position angle from catalog, if available
Reff (numpy.array) – Effective radius from catalog, if available
drpver (str) – (Optional) DRP version, see above.
redux_path (str) – (Optional) Path to the top level of directory for the DRP output, see above.
dapver (str) – (Optional) DAP version, see above.
analysis_path (str) – (Optional) Path to the top level directory for the DAP output files, see above.
overwrite (bool) – (Optional) Overwrite any existing file.
- Raises
AttributeError – Raised if drpcomplete fits file was opened in read-only mode.
FileExistsError – Raised if the drpcomplete file exists and overwrite=False.
- write_config(ofile, plate=None, ifudesign=None, index=None, sres_ext=None, sres_fill=None, covar_ext=None, reread=False, overwrite=True)[source]
Write a config file with the data used to instantiate a
mangadap.datacube.manga.MaNGADataCube
datacube for analysis.- Parameters
ofile (
str
) – Output file name.plate (
int
, optional) – Plate number.ifudesign (
int
, optional) – IFU design.index (
int
, optional) – Index of the row indata
with the data to return.sres_ext (
str
, optional) – The extension to use when constructing the spectral resolution vectors for the MaNGA datacubes. Seemangadap.datacube.manga.MaNGADataCube.spectral_resolution()
.sres_fill (
bool
, optional) – Fill masked spectral-resolution data by simple linear interpolation.covar_ext (
str
, optional) – Extension in the MaNGA DRP CUBE file to use as the single spatial correlation matrix for all wavelength channels.reread (
bool
, optional) – Force the database to be re-readoverwrite (
bool
, optional) – Overwrite any existing parameter file
- Raises
IOError – Raised if the parameter file already exists and overwrite is False.
ValueError – Raised if - the row with the data is unknown because either
index
is not defined or one or both ofplate
andifudesign
is not defined. -index
does not exist in the data array.
- write_par(ofile, mode, plate=None, ifudesign=None, index=None, reread=False, overwrite=True)[source]
Write the SDSS-style parameter (Yanny) file for use with the MaNGA DAP.
- Parameters
ofile (str) – Output file name
mode (str) – Mode of the DRP file to analyze; must be either ‘RSS’ or ‘CUBE’
plate (int) – (Optional) Plate number
ifudesign (int) – (Optional) IFU design
index (int) – (Optional) Index of the row in
data
with the data to returnreread (bool) – (Optional) Force the database to be re-read
overwrite (bool) – (Optional) Overwrite any existing parameter file
- Raises
IOError – Raised if the parameter file already exists and overwrite is False.
ValueError – Raised if - the row with the data is unknown because either index is not defined or one or both of plate and ifudesign is not defined. - index does not exist in the data array. - the ‘RSS’ mode is selected but unavailable