mangadap.survey.drpcomplete module¶
Defines a class used to produce and provide an interface for data required to create the input SDSS parameter files for the DAP.
The drpcomplete file/data is primarily required for the survey-level execution of the DAP at Utah. However, it does provide useful information regarding the completed DRP files, and it can be used to create DAP par files.
Preferrably, however, one should get this information from the DRPall, or forthcoming DAPall, file for a given MPL.
Until further documented here, the description of the DAP par file is available in the SDSS-IV/MaNGA Technical Reference Manual.
- License:
- Copyright (c) 2015, SDSS-IV/MaNGA Pipeline Group
- Licensed under BSD 3-clause license - see LICENSE.rst
- Class usage examples:
- Add some usage comments here!
- Revision history:
- 20 Nov 2014: Started implementation by K. Westfall (KBW)01 Dec 2014: (KBW) Committed to SVN12 Jan 2015: (KBW) Allow for use of plateTargets file using yanny.py19 Mar 2015: (KBW) Adjustments for changes to drpfile and drpfile_list. Now always sets the names of the NSA catalogs and the plateTargets files; added the nsa_catid. Changed drppath to redux_path. Added catid and catindx to output file. Major change to matching when using NSA catalog(s). Imports full
mangadap.drpfits
andmangadap.dapfile
classes, require a change to the function calls. Changed parameter order inwrite()
and allowed for use of attributes as default.20 May 2015: (KBW) Documentation and Sphinx tests. Prep for conversion from NSA to TRG nomenclatature, but still need to make the full conversion.21 Aug 2015: (KBW) Major revisions: File moved from main module to survey submodule; now only reads plateTargets files, which have the NSA IDs in them; changed the number of arrays returned byDRPComplete._match_platetargets()
and the default values; removed def_veldisp and use_trg fromDRPComplete.update()
; added target-catalog version, and NSAID v1_0_0 to the output fits file; removed options fromDRPComplete.write()
related to the target catalogs and platetargets files (these should be defined by the object);27 Aug 2015: (KBW) Include MANGA_TARGET1 and MANGA_TARGET3 in data structure, pulled from plateTargets files; NSAID v1_0_0 removed; changed from returning Sersic parameters to elliptical Petrosian parameters.06 Oct 2015: (KBW) Changed to reading ‘object_ra’ and ‘object_dec’, instead of target counter parts due to changes in MaNGA core17 Feb 2016: (KBW) Converted the name of the class to DRPComplete29 Feb 2016: (KBW) Import drpfits, not drpfile13 May 2016: (KBW) Switch to using pydl.pydlutils.yanny instead of internal yanny reader. Incorporated changes to plateTargets column names defined for DR13.02 Aug 2016: (KBW) Added directory_path input parameter toDRPComplete
01 Mar 2018: (KBW) Added ability to read the redshift fix file.15 Jan 2019: (KBW) Allow the object to be defined using the DRPall file.
-
class
mangadap.survey.drpcomplete.
DRPComplete
(platelist=None, ifudesignlist=None, drpall=None, platetargets=None, catid=None, drpver=None, redux_path=None, dapver=None, analysis_path=None, directory_path=None, readonly=False)[source]¶ Bases:
object
Database with information needed by the DAP to analyze the completed DRP files.
This class searches the defined paths for files resulting from the 3D phase of the MaNGA DRP, and then collates the information necessary to run those files through the MaNGA DAP. The necessary parameters are pulled from the provided platetargets files; see
update()
.Parameters: - platelist (
str
,list
, optional) – List of plates to search for. Default is to search the full DRP path. - ifudesignlist (
str
,list
, optional) – List of ifudesign to search for. Default is to search the full DRP path. - drpall (
str
, optional) – The full path to the DRPall fits file. Default is set bymangadap.config.defaults.default_drpall_file()
. - platetargets (
str
,list
, optional) – List of platetargets files to search through to find any given plate ifudesign combination. Default is returned as the first element inmangadap.config.defaults.default_plate_target_files()
. - catid (
str
,list
, optional) – List of target catalog ID numbers. Default is returned as the second element inmangadap.config.defaults.default_plate_target_files()
. - drpver (
str
, optional) –- DRP version, which is:
- used to define the default DRP redux path
- used when declaring a drpfits instance
- used in the name of the drpcomplete fits file
- included as a header keyword in the output file
Default is defined by
mangadap.config.defaults.default_drp_version()
. - redux_path (
str
, optional) – The path to the top level directory containing the DRP output files; this is the same as the redux_path in themangadap.drpfits.DRPFits
class. Default is defined bymangadap.config.defaults.default_redux_path()
. - dapver (
str
, optional) –- DAP version, which is:
- used to define the default DAP analysis path
- included as a header keyword in the output drpcomplete fits file
Default is defined by
mangadap.config.defaults.default_dap_version()
- analysis_path (
str
, optional) – A The path to the top level directory for the DAP output files; this is different from the directory_path in themangadap.dapfile
class. Default is defined by :func:`mangadap.config.defaults.default_analysis_path`A - directory_path (
str
, optional) – Direct path to the output file produced usingmangadap.config.defaults.default_dap_common_path()
- readonly (
bool
, optional) – Flag that the drpcomplete fits file is only opened for reading, not for updating. If True, many of the attributes will be set to None.
-
platelist
¶ List of plates to search for, see above.
Type: list
-
ifudesignlist
¶ List of IFU designs to search for, see above.
Type: list
-
drpall
¶ The DRPall file, see above.
Type: str
-
platetargets
¶ List of platetargets files to search through, see above.
Type: numpy.ndarray
-
catid
¶ List of target catalog IDs, see above
Type: numpy.ndarray
-
drpver
¶ DRP version, see above.
Type: str
-
redux_path
¶ Path to the top level of directory for the DRP output, see above.
Type: str
-
dapver
¶ DAP version, see above.
Type: str
-
analysis_path
¶ Path to the top level directory for the DAP output files, see above.
Type: str
-
directory_path
¶ Direct path to the output file produced using
mangadap.config.defaults.default_dap_common_path()
Type: str
-
readonly
¶ Flag that the drpcomplete fits file is only opened for reading, not for updating.
Type: bool
-
hdu
¶ Fits data with binary table data.
Type: astropy.io.fits.hdu.hdulist.HDUList
-
nobs
¶ Number of observations in the file
Type: int
-
_all_data_exists
()[source]¶ Determines if the data for all the plates/ifudesigns selected in the current compilation is already present in the current drpcomplete fits file.
Returns: Flag that all data has already been collated for the requested plate/ifudesign list. Return type: bool
-
_confirm_access
(reread=False)[source]¶ Check the drpcomplete fits file at
file_path()
is accessible, and read the data if not yet read.Parameters: reread ( bool
, optional) – Force the file to be re-readReturns: Flag that hdu
is read and valid.Return type: bool
-
_find_completed_reductions
(mindesign=19, combinatorics=False, on_disk=False)[source]¶ Search the DRP path for reduced CUBE files.
Function allows for one or both of
platelist
andifudesignlist
to be None. The behavior is:- If both are None, all available CUBE files are used to
create
platelist
andifudesignlist
. - If one is None, all available CUBE files within the
provided list of the one that is not None are used to
create
platelist
andifudesignlist
. E.g., ifplatelist
=[7443] andifudesignlist
is None, all CUBE files with plate=7443 are chosen. - If both are not None and they have different lengths, or
the same length and combinatorics is True, all available
CUBE files within the constraints of both lists are
selected. E.g., if
platelist
=[7443,7495] andifudesignlist
=[12704], the CUBE files with (plate,ifudesign)=[(7443,12704),(7459,12704)] are chosen. - If both are not None and they have the same length and
combinatorics is False, all available CUBE files in the
matched lists are chosen. E.g. if
platelist
=[7443,7495] andifudesignlist
=[12704,12703], the CUBE files with (plate,ifudesign)=[(7443,12704),(7459,12703) are chosen
Parameters: - mindesign (
int
, optional) – Minimum bundle design to consider. For example, to ignore all the 7-fiber bundles, set mindesign=19 (default) to only select the bundles with 19 or more fibers. - combinatorics (
bool
, optional) – Createplatelist
andifudesignlist
by determining all possible combinations of the input values. See above. - on_disk (
bool
, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file.
Returns: Two lists with the available plates and ifudesigns for which to collect data.
Return type: list
- If both are None, all available CUBE files are used to
create
-
_find_modes
()[source]¶ Using the provided list of CUBE DRP files, find if the RSS mode is also available.
Warning
- This assumes everything in
platelist
andifudesignlist
has a ‘CUBE’ file.
Currently only two modes are possible:
- only ‘CUBE’ file are available
- both ‘CUBE’ and ‘RSS’ files are available
Returns: Array of modes for each input DRP file. Return type: numpy.ndarray - This assumes everything in
-
_match_drpall
()[source]¶ Find the data needed for the DRP complete databas in the DRPall file.
If a plate-ifudesign combination is not found in the DRPall file, the other parameter values are set to -9999 and the MaNGA ID is set to NULL.
If the plate-ifudesign combination is found, the columns in the plateTargets files that the code expects to find are ‘plate’, ‘ifudsgn’, ‘mangaid’, ‘objra’, ‘objdec’; the values that will be replaced with ‘NULL’ (str) or -9999 (int or float) if they do not exist are ‘nsa_version’, ‘nsa_nsaid’, ‘mngtarg1’, ‘mngtarg3’, ‘z’, ‘nsa_elpetro_ba’, ‘nsa_elpetro_phi’, ‘nsa_elpetro_th50_r’, ‘nsa_vdisp’.
Returns: 14 arrays with: MaNGA ID, object right ascension, object declination, catalog ID, index of the entry in the catalog, catalog version, ID of object in the catalog, main MaNGA survey target bitmask, ancillary MaNGA survey target bitmask, velocity, velocity dispersion, ellipticity, position angle, and effective radius. Return type: numpy.array
-
_match_platetargets
(quiet=True)[source]¶ Read the platetargets files and match the data therein to the completed DRP files based on the plate and ifudesign.
If a plate-ifudesign combination is not found in the plateTargets file, the other parameter values are set to -9999 and the MaNGA ID is set to NULL.
If the plate-ifudesign combination is found, the columns in the plateTargets files that the code expects to find are ‘plateid’, ‘ifudesign’, ‘mangaid’, ‘object_ra’, and ‘object_dec’; the values that will be replaced with ‘NULL’ (str) or -9999 (int or float) if they do not exist are ‘nsa_version’, ‘nsa_nsaid’, ‘manga_target1’, ‘manga_target3’, ‘z’ or ‘nsa_z’, ‘nsa_elpetro_ba’, ‘nsa_elpetro_phi’, ‘nsa_elpetro_th50_R’, ‘nsa_vdisp’.
Todo
- Instead of searching through all files for the correct plate-ifudesign, use the MaNGA ID to get the correct catalog? Requires getting the MaNGA ID from somewhere, but it takes a long time to read from the fits files…
- Is there some more concise method of performing the same thing as what is done below with many try/except blocks?
Parameters: quiet ( bool
, optional) – Suppress terminal outputReturns: 14 arrays with: MaNGA ID, object right ascension, object declination, catalog ID, index of the entry in the catalog, catalog version, ID of object in the catalog, main MaNGA survey target bitmask, ancillary MaNGA survey target bitmask, velocity, velocity dispersion, ellipticity, position angle, and effective radius. Return type: numpy.array
-
_read
()[source]¶ Read the data in the existing file at
file_path()
.
-
_read_platetargets
()[source]¶ Read all the platetargets files using pydl.pydlutils.yanny and return a list of yanny structures.
Returns: A list of yanny structures, one per platetargets file. Return type: list Raises: FileNotFoundError
– Raised if cannot open one or more platetargets files.Todo
This should be made more efficient by collating the required plateTargets data into a single record array!
-
_read_redshift_fix
()[source]¶ Read data from the $MANGADAP_DIR/data/fix/redshift_fix.par file, if it exists.
Returns: A yanny structure with the redshift fixes. Returns None and raises a warning if the file does not exist. Return type: yanny
-
_write_parameter_list
(ostream, index, mode)[source]¶ Write a DAPPAR entry to the SDSS-style parameter file.
Parameters: - ostream (_io.TextIOWrapper) – File stream for output
- index (int) – Index of the entry in
data
to write - mode (int) – Mode of the entry; see
_find_modes()
-
_write_parameter_struct
(ostream)[source]¶ Write the structure of the DAP SDSS-style parameter set to a file.
Parameters: ostream (_io.TextIOWrapper) – File stream for output
-
can_analyze
(row=None)[source]¶ Determine the DAP can analyze a plate-ifu entry in the database.
Parameters: row ( int
, optional) – The specific row to test. By default, return a boolean vector for all the database rows.Returns: Either a single boolean or boolean numpy.ndarray flagging that DAP can (True) or cannot (False) analyze the data associated with the database entry (or entries).
-
entry_index
(plate, ifudesign, reread=False)[source]¶ Find the index of the row with the parameter data for the specified plate and ifudesign.
Warning
- This is very inefficient if you’re looking for multiple entries…
Parameters: - plate (
int
) – Plate number - ifudesign (
int
) – IFU design - reread (
bool
, optional) – Force the database to be re-read
Returns: Index of the row in
data
with the data for the given plate and ifudesignReturn type: int
Raises: ValueError
– Raised if the given plate and ifudesign were not found.
-
update
(platelist=None, ifudesignlist=None, combinatorics=False, force=False, alldrp=False, use_platetargets=False, on_disk=False, quiet=False)[source]¶ Update the DRP complete file.
If platelist and/or ifudesignlist are provided, the existing
platelist
andifudesignlist
attributes are replaced.If platelist and ifudesignlist do not have the same length or combinatorics is True, the lists are expanded to include all combinations of their elements.
If platelist and ifudesignlist are None or alldrp is True, all available plates/ifudesigns are collected from within the DRP directory structure.
If the result of
file_path()
does not exist, it is created.If the result of
file_path()
does exist, the available plates and ifudesigns in the file are compared against the list (provided or collected) to update. If the lists differ, the drpcomplete fits file is re-created from scratch. If all the plates and ifudesigns are available, nothing is done, unless force=True.Parameters: - platelist (
str
,list
, optional) – List of plates to include in the drpcomplete fits file. - ifudesignlist (
str
,list
, optional) – List of ifudesigns to include in the drpcomplete fits file. - combinatorics (
bool
, optional) – Determine all combinations of the entered plates and ifudesigns. - force (
bool
, optional) – Overwrite any existing drpcomplete fits file with a new one built from scratch. - alldrp (
bool
, optional) – Find the full list of available DRP files. - use_platetargets (
bool
, optional) – Generate the data using the platetargets files instead of the DRPall file. - on_disk (
bool
, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file. - quiet (
bool
, optional) – Suppress terminal output
Raises: ValueError
– Raised if drpcomplete fits file was opened in read-only mode.- platelist (
-
write
(platelist, ifudesignlist, modes, mangaid, objra, objdec, catid, catindx, trg_version, trg_id, manga_trg1, manga_trg3, vel, veldisp, ell, pa, Reff, drpver=None, redux_path=None, dapver=None, analysis_path=None, clobber=True)[source]¶ Write the drpcomplete fits binary table.
- Header keywords are:
- VERSDRP: DRP version
- RDXPTH: DRP reduction path
- VERSDAP: DAP version
- SISPTH: DAP analysis path
- PLTTRG N: plateTargets file N
- CATID N: ID Number of target catalog N
- AUTHOR: Set to ‘K.B. Westfall <kbwestfall@gmail.com>’
- Binary table columns:
- PLATE (1J): Plate number
- IFUDESIGN (1J): IFU design
- MODES (1B): Modes available
- MODES=1: only ‘CUBE’ file is available
- MODES=2: both ‘CUBE’ and ‘RSS’ files are available
- MANGAID (n A): MaNGA ID (same as ‘CATID-CATINDX’)
- OBJRA (1D): Object right ascension
- OBJDEC (1D): Object declination
- CATID (1J): Catalog ID used for target selection
- CATINDX (1J): Index in catalog with target data
- TRG_VERSION (n A): Version of the catalog (e.g., NSA)
- used in targetting (def=’NULL’ if not available)
- TRG_ID (1J): Target ID in catalog (def=-9999 if not available)
- MANGA_TARGET1: Main MaNGA survey target bitmask
- MANGA_TARGET3: Ancillary MaNGA survey target bitmask
- VEL (1D): Systemic velocity from catalog (def=-9999.0 if not available)
- VDISP (1D): Velocity dispersion from catalog (def=-9999.0 if not available)
- ELL (1D): Ellipticity (1-b/a) from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
- PA (1D): Position angle from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
- REFF (1D): Effective radius from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
Parameters: - platelist (list) – List of plates
- ifudesignlist (list) – List of IFU designs
- modes (numpy.array) – Mode values, see above
- mangaid (numpy.array) – MaNGA IDs
- objra (numpy.array) – Object right ascensions
- objdec (numpy.array) – Object declinations
- catid (numpy.array) – Catalog ID used for target selection
- catindx (numpy.array) – Index in catalog with target data
- trg_version (numpy.array) – Version of the catalog (e.g., NSA) used in targetting
- trg_id (numpy.array) – Target ID in catalog
- manga_trg1 (numpy.array) – Main MaNGA survey target bitmask
- manga_trg3 (numpy.array) – Ancillary MaNGA survey target bitmask
- vel (numpy.array) – Velocity from catalog, if available
- veldisp (numpy.array) – Velocity dispersion from catalog, if available
- ell (numpy.array) – Ellipticity (1-b/a) from catalog, if available
- pa (numpy.array) – Position angle from catalog, if available
- Reff (numpy.array) – Effective radius from catalog, if available
- drpver (str) – (Optional) DRP version, see above.
- redux_path (str) – (Optional) Path to the top level of directory for the DRP output, see above.
- dapver (str) – (Optional) DAP version, see above.
- analysis_path (str) – (Optional) Path to the top level directory for the DAP output files, see above.
- clobber (bool) – (Optional) Overwrite any existing file.
Raises: AttributeError
– Raised if drpcomplete fits file was opened in read-only mode.FileExistsError
– Raised if the drpcomplete file exists and clobber=False.
-
write_par
(ofile, mode, plate=None, ifudesign=None, index=None, reread=False, clobber=True)[source]¶ Write the SDSS-style parameter (Yanny) file for use with the MaNGA DAP.
Parameters: - ofile (str) – Output file name
- mode (str) – Mode of the DRP file to analyze; must be either ‘RSS’ or ‘CUBE’
- plate (int) – (Optional) Plate number
- ifudesign (int) – (Optional) IFU design
- index (int) – (Optional) Index of the row in
data
with the data to return - reread (bool) – (Optional) Force the database to be re-read
- clobber (bool) – (Optional) Overwrite any existing parameter file
Raises: IOError
– Raised if the parameter file already exists and clobber is False.ValueError
– Raised if- the row with the data is unknown because either index is not defined or one or both of plate and ifudesign is not defined.
- index does not exist in the data array.
- the ‘RSS’ mode is selected but unavailable
- platelist (