mangadap.survey.drpcomplete module

Defines a class used to produce and provide an interface for data required to create the input SDSS parameter files for the DAP.

The drpcomplete file/data is primarily required for the survey-level execution of the DAP at Utah. However, it does provide useful information regarding the completed DRP files, and it can be used to create DAP par files.

Preferrably, however, one should get this information from the DRPall, or forthcoming DAPall, file for a given MPL.

Until further documented here, the description of the DAP par file is available in the SDSS-IV/MaNGA Technical Reference Manual.

License:
Copyright (c) 2015, SDSS-IV/MaNGA Pipeline Group
Licensed under BSD 3-clause license - see LICENSE.rst
Class usage examples:
Add some usage comments here!
Revision history:
20 Nov 2014: Started implementation by K. Westfall (KBW)
01 Dec 2014: (KBW) Committed to SVN
12 Jan 2015: (KBW) Allow for use of plateTargets file using yanny.py
19 Mar 2015: (KBW) Adjustments for changes to drpfile and drpfile_list. Now always sets the names of the NSA catalogs and the plateTargets files; added the nsa_catid. Changed drppath to redux_path. Added catid and catindx to output file. Major change to matching when using NSA catalog(s). Imports full mangadap.drpfits and mangadap.dapfile classes, require a change to the function calls. Changed parameter order in write() and allowed for use of attributes as default.
20 May 2015: (KBW) Documentation and Sphinx tests. Prep for conversion from NSA to TRG nomenclatature, but still need to make the full conversion.
21 Aug 2015: (KBW) Major revisions: File moved from main module to survey submodule; now only reads plateTargets files, which have the NSA IDs in them; changed the number of arrays returned by DRPComplete._match_platetargets() and the default values; removed def_veldisp and use_trg from DRPComplete.update(); added target-catalog version, and NSAID v1_0_0 to the output fits file; removed options from DRPComplete.write() related to the target catalogs and platetargets files (these should be defined by the object);
27 Aug 2015: (KBW) Include MANGA_TARGET1 and MANGA_TARGET3 in data structure, pulled from plateTargets files; NSAID v1_0_0 removed; changed from returning Sersic parameters to elliptical Petrosian parameters.
06 Oct 2015: (KBW) Changed to reading ‘object_ra’ and ‘object_dec’, instead of target counter parts due to changes in MaNGA core
17 Feb 2016: (KBW) Converted the name of the class to DRPComplete
29 Feb 2016: (KBW) Import drpfits, not drpfile
13 May 2016: (KBW) Switch to using pydl.pydlutils.yanny instead of internal yanny reader. Incorporated changes to plateTargets column names defined for DR13.
02 Aug 2016: (KBW) Added directory_path input parameter to DRPComplete
01 Mar 2018: (KBW) Added ability to read the redshift fix file.
15 Jan 2019: (KBW) Allow the object to be defined using the DRPall file.
class mangadap.survey.drpcomplete.DRPComplete(platelist=None, ifudesignlist=None, drpall=None, platetargets=None, catid=None, drpver=None, redux_path=None, dapver=None, analysis_path=None, directory_path=None, readonly=False)[source]

Bases: object

Database with information needed by the DAP to analyze the completed DRP files.

This class searches the defined paths for files resulting from the 3D phase of the MaNGA DRP, and then collates the information necessary to run those files through the MaNGA DAP. The necessary parameters are pulled from the provided platetargets files; see update().

Parameters:
  • platelist (str, list, optional) – List of plates to search for. Default is to search the full DRP path.
  • ifudesignlist (str, list, optional) – List of ifudesign to search for. Default is to search the full DRP path.
  • drpall (str, optional) – The full path to the DRPall fits file. Default is set by mangadap.config.defaults.default_drpall_file().
  • platetargets (str, list, optional) – List of platetargets files to search through to find any given plate ifudesign combination. Default is returned as the first element in mangadap.config.defaults.default_plate_target_files().
  • catid (str, list, optional) – List of target catalog ID numbers. Default is returned as the second element in mangadap.config.defaults.default_plate_target_files().
  • drpver (str, optional) –
    DRP version, which is:
    • used to define the default DRP redux path
    • used when declaring a drpfits instance
    • used in the name of the drpcomplete fits file
    • included as a header keyword in the output file

    Default is defined by mangadap.config.defaults.default_drp_version().

  • redux_path (str, optional) – The path to the top level directory containing the DRP output files; this is the same as the redux_path in the mangadap.drpfits.DRPFits class. Default is defined by mangadap.config.defaults.default_redux_path().
  • dapver (str, optional) –
    DAP version, which is:
    • used to define the default DAP analysis path
    • included as a header keyword in the output drpcomplete fits file

    Default is defined by mangadap.config.defaults.default_dap_version()

  • analysis_path (str, optional) – A The path to the top level directory for the DAP output files; this is different from the directory_path in the mangadap.dapfile class. Default is defined by :func:`mangadap.config.defaults.default_analysis_path`A
  • directory_path (str, optional) – Direct path to the output file produced using mangadap.config.defaults.default_dap_common_path()
  • readonly (bool, optional) – Flag that the drpcomplete fits file is only opened for reading, not for updating. If True, many of the attributes will be set to None.
platelist

List of plates to search for, see above.

Type:list
ifudesignlist

List of IFU designs to search for, see above.

Type:list
drpall

The DRPall file, see above.

Type:str
platetargets

List of platetargets files to search through, see above.

Type:numpy.ndarray
catid

List of target catalog IDs, see above

Type:numpy.ndarray
drpver

DRP version, see above.

Type:str
redux_path

Path to the top level of directory for the DRP output, see above.

Type:str
dapver

DAP version, see above.

Type:str
analysis_path

Path to the top level directory for the DAP output files, see above.

Type:str
directory_path

Direct path to the output file produced using mangadap.config.defaults.default_dap_common_path()

Type:str
readonly

Flag that the drpcomplete fits file is only opened for reading, not for updating.

Type:bool
hdu

Fits data with binary table data.

Type:astropy.io.fits.hdu.hdulist.HDUList
nobs

Number of observations in the file

Type:int
_all_data_exists()[source]

Determines if the data for all the plates/ifudesigns selected in the current compilation is already present in the current drpcomplete fits file.

Returns:Flag that all data has already been collated for the requested plate/ifudesign list.
Return type:bool
_confirm_access(reread=False)[source]

Check the drpcomplete fits file at file_path() is accessible, and read the data if not yet read.

Parameters:reread (bool, optional) – Force the file to be re-read
Returns:Flag that hdu is read and valid.
Return type:bool
_find_completed_reductions(mindesign=19, combinatorics=False, on_disk=False)[source]

Search the DRP path for reduced CUBE files.

Function allows for one or both of platelist and ifudesignlist to be None. The behavior is:

  • If both are None, all available CUBE files are used to create platelist and ifudesignlist.
  • If one is None, all available CUBE files within the provided list of the one that is not None are used to create platelist and ifudesignlist. E.g., if platelist =[7443] and ifudesignlist is None, all CUBE files with plate=7443 are chosen.
  • If both are not None and they have different lengths, or the same length and combinatorics is True, all available CUBE files within the constraints of both lists are selected. E.g., if platelist =[7443,7495] and ifudesignlist =[12704], the CUBE files with (plate,ifudesign)=[(7443,12704),(7459,12704)] are chosen.
  • If both are not None and they have the same length and combinatorics is False, all available CUBE files in the matched lists are chosen. E.g. if platelist =[7443,7495] and ifudesignlist =[12704,12703], the CUBE files with (plate,ifudesign)=[(7443,12704),(7459,12703) are chosen
Parameters:
  • mindesign (int, optional) – Minimum bundle design to consider. For example, to ignore all the 7-fiber bundles, set mindesign=19 (default) to only select the bundles with 19 or more fibers.
  • combinatorics (bool, optional) – Create platelist and ifudesignlist by determining all possible combinations of the input values. See above.
  • on_disk (bool, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file.
Returns:

Two lists with the available plates and ifudesigns for which to collect data.
Return type:

list
_find_modes()[source]

Using the provided list of CUBE DRP files, find if the RSS mode is also available.

Warning

Currently only two modes are possible:

    1. only ‘CUBE’ file are available
    1. both ‘CUBE’ and ‘RSS’ files are available
Returns:Array of modes for each input DRP file.
Return type:numpy.ndarray
_match_drpall()[source]

Find the data needed for the DRP complete databas in the DRPall file.

If a plate-ifudesign combination is not found in the DRPall file, the other parameter values are set to -9999 and the MaNGA ID is set to NULL.

If the plate-ifudesign combination is found, the columns in the plateTargets files that the code expects to find are ‘plate’, ‘ifudsgn’, ‘mangaid’, ‘objra’, ‘objdec’; the values that will be replaced with ‘NULL’ (str) or -9999 (int or float) if they do not exist are ‘nsa_version’, ‘nsa_nsaid’, ‘mngtarg1’, ‘mngtarg3’, ‘z’, ‘nsa_elpetro_ba’, ‘nsa_elpetro_phi’, ‘nsa_elpetro_th50_r’, ‘nsa_vdisp’.

Returns:14 arrays with: MaNGA ID, object right ascension, object declination, catalog ID, index of the entry in the catalog, catalog version, ID of object in the catalog, main MaNGA survey target bitmask, ancillary MaNGA survey target bitmask, velocity, velocity dispersion, ellipticity, position angle, and effective radius.
Return type:numpy.array
_match_platetargets(quiet=True)[source]

Read the platetargets files and match the data therein to the completed DRP files based on the plate and ifudesign.

If a plate-ifudesign combination is not found in the plateTargets file, the other parameter values are set to -9999 and the MaNGA ID is set to NULL.

If the plate-ifudesign combination is found, the columns in the plateTargets files that the code expects to find are ‘plateid’, ‘ifudesign’, ‘mangaid’, ‘object_ra’, and ‘object_dec’; the values that will be replaced with ‘NULL’ (str) or -9999 (int or float) if they do not exist are ‘nsa_version’, ‘nsa_nsaid’, ‘manga_target1’, ‘manga_target3’, ‘z’ or ‘nsa_z’, ‘nsa_elpetro_ba’, ‘nsa_elpetro_phi’, ‘nsa_elpetro_th50_R’, ‘nsa_vdisp’.

Todo

  • Instead of searching through all files for the correct plate-ifudesign, use the MaNGA ID to get the correct catalog? Requires getting the MaNGA ID from somewhere, but it takes a long time to read from the fits files…
  • Is there some more concise method of performing the same thing as what is done below with many try/except blocks?
Parameters:quiet (bool, optional) – Suppress terminal output
Returns:14 arrays with: MaNGA ID, object right ascension, object declination, catalog ID, index of the entry in the catalog, catalog version, ID of object in the catalog, main MaNGA survey target bitmask, ancillary MaNGA survey target bitmask, velocity, velocity dispersion, ellipticity, position angle, and effective radius.
Return type:numpy.array
_read()[source]

Read the data in the existing file at file_path().

_read_platetargets()[source]

Read all the platetargets files using pydl.pydlutils.yanny and return a list of yanny structures.

Returns:A list of yanny structures, one per platetargets file.
Return type:list
Raises:FileNotFoundError – Raised if cannot open one or more platetargets files.

Todo

This should be made more efficient by collating the required plateTargets data into a single record array!

_read_redshift_fix()[source]

Read data from the $MANGADAP_DIR/data/fix/redshift_fix.par file, if it exists.

Returns:A yanny structure with the redshift fixes. Returns None and raises a warning if the file does not exist.
Return type:yanny
_write_parameter_list(ostream, index, mode)[source]

Write a DAPPAR entry to the SDSS-style parameter file.

Parameters:
  • ostream (_io.TextIOWrapper) – File stream for output
  • index (int) – Index of the entry in data to write
  • mode (int) – Mode of the entry; see _find_modes()
_write_parameter_struct(ostream)[source]

Write the structure of the DAP SDSS-style parameter set to a file.

Parameters:ostream (_io.TextIOWrapper) – File stream for output
can_analyze(row=None)[source]

Determine the DAP can analyze a plate-ifu entry in the database.

Parameters:row (int, optional) – The specific row to test. By default, return a boolean vector for all the database rows.
Returns:Either a single boolean or boolean numpy.ndarray flagging that DAP can (True) or cannot (False) analyze the data associated with the database entry (or entries).
entry_index(plate, ifudesign, reread=False)[source]

Find the index of the row with the parameter data for the specified plate and ifudesign.

Warning

  • This is very inefficient if you’re looking for multiple entries…
Parameters:
  • plate (int) – Plate number
  • ifudesign (int) – IFU design
  • reread (bool, optional) – Force the database to be re-read
Returns:

Index of the row in data with the data for the given plate and ifudesign
Return type:

int
Raises:

ValueError – Raised if the given plate and ifudesign were not found.
file_name()[source]

Return the name of the DRP complete database.

file_path()[source]

Return the full pat to the DRP complete database.

update(platelist=None, ifudesignlist=None, combinatorics=False, force=False, alldrp=False, use_platetargets=False, on_disk=False, quiet=False)[source]

Update the DRP complete file.

If platelist and/or ifudesignlist are provided, the existing platelist and ifudesignlist attributes are replaced.

If platelist and ifudesignlist do not have the same length or combinatorics is True, the lists are expanded to include all combinations of their elements.

If platelist and ifudesignlist are None or alldrp is True, all available plates/ifudesigns are collected from within the DRP directory structure.

If the result of file_path() does not exist, it is created.

If the result of file_path() does exist, the available plates and ifudesigns in the file are compared against the list (provided or collected) to update. If the lists differ, the drpcomplete fits file is re-created from scratch. If all the plates and ifudesigns are available, nothing is done, unless force=True.

Parameters:
  • platelist (str, list, optional) – List of plates to include in the drpcomplete fits file.
  • ifudesignlist (str, list, optional) – List of ifudesigns to include in the drpcomplete fits file.
  • combinatorics (bool, optional) – Determine all combinations of the entered plates and ifudesigns.
  • force (bool, optional) – Overwrite any existing drpcomplete fits file with a new one built from scratch.
  • alldrp (bool, optional) – Find the full list of available DRP files.
  • use_platetargets (bool, optional) – Generate the data using the platetargets files instead of the DRPall file.
  • on_disk (bool, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file.
  • quiet (bool, optional) – Suppress terminal output
Raises:

ValueError – Raised if drpcomplete fits file was opened in read-only mode.
write(platelist, ifudesignlist, modes, mangaid, objra, objdec, catid, catindx, trg_version, trg_id, manga_trg1, manga_trg3, vel, veldisp, ell, pa, Reff, drpver=None, redux_path=None, dapver=None, analysis_path=None, clobber=True)[source]

Write the drpcomplete fits binary table.

Header keywords are:
  • VERSDRP: DRP version
  • RDXPTH: DRP reduction path
  • VERSDAP: DAP version
  • SISPTH: DAP analysis path
  • PLTTRG N: plateTargets file N
  • CATID N: ID Number of target catalog N
  • AUTHOR: Set to ‘K.B. Westfall <kbwestfall@gmail.com>’
Binary table columns:
  1. PLATE (1J): Plate number
  2. IFUDESIGN (1J): IFU design
  3. MODES (1B): Modes available
    • MODES=1: only ‘CUBE’ file is available
    • MODES=2: both ‘CUBE’ and ‘RSS’ files are available
  4. MANGAID (n A): MaNGA ID (same as ‘CATID-CATINDX’)
  5. OBJRA (1D): Object right ascension
  6. OBJDEC (1D): Object declination
  7. CATID (1J): Catalog ID used for target selection
  8. CATINDX (1J): Index in catalog with target data
  9. TRG_VERSION (n A): Version of the catalog (e.g., NSA)
    used in targetting (def=’NULL’ if not available)
  10. TRG_ID (1J): Target ID in catalog (def=-9999 if not available)
  11. MANGA_TARGET1: Main MaNGA survey target bitmask
  12. MANGA_TARGET3: Ancillary MaNGA survey target bitmask
  13. VEL (1D): Systemic velocity from catalog (def=-9999.0 if not available)
  14. VDISP (1D): Velocity dispersion from catalog (def=-9999.0 if not available)
  15. ELL (1D): Ellipticity (1-b/a) from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
  16. PA (1D): Position angle from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
  17. REFF (1D): Effective radius from catalog (def=-9999.0 if not available); Elliptical Petrosian value from NSA
Parameters:
  • platelist (list) – List of plates
  • ifudesignlist (list) – List of IFU designs
  • modes (numpy.array) – Mode values, see above
  • mangaid (numpy.array) – MaNGA IDs
  • objra (numpy.array) – Object right ascensions
  • objdec (numpy.array) – Object declinations
  • catid (numpy.array) – Catalog ID used for target selection
  • catindx (numpy.array) – Index in catalog with target data
  • trg_version (numpy.array) – Version of the catalog (e.g., NSA) used in targetting
  • trg_id (numpy.array) – Target ID in catalog
  • manga_trg1 (numpy.array) – Main MaNGA survey target bitmask
  • manga_trg3 (numpy.array) – Ancillary MaNGA survey target bitmask
  • vel (numpy.array) – Velocity from catalog, if available
  • veldisp (numpy.array) – Velocity dispersion from catalog, if available
  • ell (numpy.array) – Ellipticity (1-b/a) from catalog, if available
  • pa (numpy.array) – Position angle from catalog, if available
  • Reff (numpy.array) – Effective radius from catalog, if available
  • drpver (str) – (Optional) DRP version, see above.
  • redux_path (str) – (Optional) Path to the top level of directory for the DRP output, see above.
  • dapver (str) – (Optional) DAP version, see above.
  • analysis_path (str) – (Optional) Path to the top level directory for the DAP output files, see above.
  • clobber (bool) – (Optional) Overwrite any existing file.
Raises:
  • AttributeError – Raised if drpcomplete fits file was opened in read-only mode.
  • FileExistsError – Raised if the drpcomplete file exists and clobber=False.
write_par(ofile, mode, plate=None, ifudesign=None, index=None, reread=False, clobber=True)[source]

Write the SDSS-style parameter (Yanny) file for use with the MaNGA DAP.

Parameters:
  • ofile (str) – Output file name
  • mode (str) – Mode of the DRP file to analyze; must be either ‘RSS’ or ‘CUBE’
  • plate (int) – (Optional) Plate number
  • ifudesign (int) – (Optional) IFU design
  • index (int) – (Optional) Index of the row in data with the data to return
  • reread (bool) – (Optional) Force the database to be re-read
  • clobber (bool) – (Optional) Overwrite any existing parameter file
Raises:
  • IOError – Raised if the parameter file already exists and clobber is False.
  • ValueError – Raised if
    • the row with the data is unknown because either index is not defined or one or both of plate and ifudesign is not defined.
    • index does not exist in the data array.
    • the ‘RSS’ mode is selected but unavailable