mangadap.survey.rundap module¶
Defines the class used to automate the execution of the MaNGA DAP at Utah.
- License:
- Copyright (c) 2015, SDSS-IV/MaNGA Pipeline Group
- Licensed under BSD 3-clause license - see LICENSE.rst
- Class usage examples:
- Add some usage comments here!
- Revision history:
- ?? Nov 2014: Architecture setup by J. Brownstein <joelbrownstein@astro.utah.edu>18 Nov 2014: Handover to K. Westfall (KBW) for development. (Attempt to) conform style to PEP 8, except that lines are 99 characters long (docstrings still limited to 72 characters). Also (attempt to) conform to docstrings style according to PEP 257.01 Dec 2014: (KBW) Committed to SVN02 Dec 2014: (KBW) Changed drpver and idlutilsver to mangaver13 Jan 2015: (KBW) Changed it back, added platetargets05 Feb 2015: (KBW) Change to load module based on MPL19 Mar 2015: (KBW) Major changes to allow for more control over the directory structure containing the DRP data and for the DAP results; following changed in the drpfile and drpcomplete classes.27 Aug 2015: (KBW) Sphinx documentation; prep for MPL-4; accommodate changes in drpcomplete; removed arg as an attribute because it was only used in
mangadap.survey.rundap._read_arg()
; despite not doing anything before, commented out _write_module_commands for the time-being; removed file_root() and parameter_file() functions in favor of adding/using functions frommangadap.config.defaults
; version number set to 1_1_0.06 Oct 2015: (KBW) Added functionality to select the types of additional output. Minor changes to allow for DR13 QA plots.22 Oct 2015: (KBW) Added check that the selected MPL versions match the current environmental versions. Includes edit tomangadap.survey.mangampl.MaNGAMPL
to include MANGACORE_VER. Changed default number of nodes to 9. Added definition of idl command for use on sciama cluster at the ICG, Portsmouth.04 Nov 2015: (KBW) Added capability to turn off main DAP processing call so thatrundap
can be used to, e.g., just create the plots for existing data.17 Feb 2016: (KBW) Added try/except block for importing pbs.queue; converted old drpfile to newmangadap.drpfits.DRPFits
, and drpcomplete to newmangadap.survey.drpcomplete.DRPComplete
.13 May 2016: (KBW) Substantial changes to accommodate conversion of DAP from IDL to python19 May 2016: (KBW) Added command-line options for including log file11 Oct 2016: (KBW) Version checking moved tomangadap.survey.mangampl.MaNGAMPL
.23 Aug 2017: (KBW) Include ppxffit QA plots.29 Aug 2017: (KBW) Include DAPall post-processing.16 Jan 2019: (KBW) Major changes: Only one operation mode (daily, all, redo removed; code always run in “redo” mode). Remove old “prior” options. Setup done using DRPall by default, but option to use mangacore/platetargets. ‘CUBE’ vs. ‘RSS’ no longer an option, only analyzes CUBEs.23 Jan 2019: (KBW) Add fit QA plots
-
class
mangadap.survey.rundap.
rundap
(clobber=None, console=None, quiet=False, print_version=False, strictver=True, mplver=None, redux_path=None, dapver=None, analysis_path=None, plan_file=None, platelist=None, ifudesignlist=None, combinatorics=False, list_file=None, use_platetargets=False, platetargets=None, on_disk=False, log=False, dapproc=True, pltifu_plots=True, post_process=False, post_plots=False, report_progress=False, verbose=0, label='mangadap', nodes=1, cpus=None, qos=None, umask='0027', walltime='240:00:00', hard=True, create=False, submit=False, queue=None)[source]¶ Bases:
object
Class used for automated execution of the MaNGA DAP via submission of scripts to a cluster.
[ More complete explanation of the class here. ]
Note
This is purely a survey-level utility for running the DAP at Utah, including submitting jobs to the cluster. No user-level usage of the DAP should be considered here, apart from not hindering such usage of the primary python programs!
Todo
- Have
_read_arg()
return the boolean forversion
instead of keeping it as an attribute.
Parameters: - clobber (
bool
, optional) – Clobber and redo any existing analysis. - console (
bool
, optional) – The class has been executed from the terminal with command-line arguments that should be parsed. - quiet (
bool
, optional) – Suppress output - print_version (
bool
, optional) – Print the class version and return. - strictver (
bool
, optional) – Strictly check the version requirements of the dependencies defined in :mangadap.survey.mangampl.MaNGAMPL` for this version of the DAP. Default is True, meaning the code will raise an exception if the expected version are not the same as the environment. - mplver (
str
, optional) – MPL version to analyze. Used withmangadap.survey.mangampl.MaNGAMPL
to set the DAP dependencies. Ideally, this controls the modules that are loaded before execution of the DAP. The default version (selected if mplver=None) is defined bymangadap.survey.mangampl.MaNGAMPL
. - redux_path (
str
, optional) – The top-level path with the DRP files used to override the default defined bymangadap.config.defaults.default_redux_path()
. - dapver (
str
, optional) – The DAP version to use for the analysis, used to override the default defined bymangadap.config.defaults.default_dap_version()
. Only used to define the file paths, not used to switch to the specified code version! - analysis_path (
str
, optional) – The top-level path for the DAP output files, used to override the default defined bymangadap.config.defaults.default_analysis_path()
. - plan_file (
str
, optional) – Name of the plan file to use for all DRP data in this run, used to override the default defined bymangadap.config.defaults.default_dap_plan_file()
. - platelist (
int
,str
, optional) – List of plates to analyze; default is to search theredux_path
for any DRP files; seemangadap.survey.drpcomplete.DRPComplete
. - ifudesignlist (
int
,str
, optional) – List of ifudesigns to analyze: default is to search theredux_path
for any DRP files; seemangadap.survey.drpcomplete.DRPComplete
. - combinatorics (
bool
, optional) – Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze. Seemangadap.drpfits.drpfits_list()
. - use_platetargets (
bool
, optional) – Flag to use the plateTargets files, instead of the DRPall file, as the source of the input data need by the DAP. - platetargets (
str
,list
, optional) – List of platetargets files to search through to find any given plate-ifudesign combination. Default is returned as the first element inmangadap.config.defaults.default_plate_target_files()
. - on_disk (
bool
, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file. This is implicitly true when using the plateTargets files to collate the data used for the DAP input instead of the DRPall file (i.e., when use_platetargets=True). - log (
bool
, optional) – Flag to create the DAP log files. - dapproc (
bool
, optional) – Flag to execute the main DAP processing. - pltifu_plots (
bool
, optional) – Create the QA plate-ifu specific plots. - post_process (
bool
, optional) – Prepare (and possibly perform) the post processing steps: create the DAPall file and its associated QA plots, and the fit QA plots for each plate. - post_plots (
bool
, optional) – Create the post-processing QA plots. - report_progress (
bool
, optional) – Report the progress of the analysis. - vebose (
int
, optional) – Verbosity level between 0 and 2. - label (
str
, optional) – Label to use in cluster queue. - nodes (
int
, optional) – Number of cluster nodes to use. - qos (
str
, optional) – Select a specific processor (e.g. sdss-fast when the manga user). This option is ignored ifq
is set. - umask (
str
, optional) – umask to set for output. - walltime (
str
, optional) – Maximum wall time for cluster job. - hard (
bool
, optional) – Same as hard keyword in cluster submission; seepbs.queue.commit()
. This keyword is in the opposite sense of the “–toughness” command-line options; i.e. including –toughness on the command line sets hard=False. - create (
bool
, optional) – Use the pbs package to create the scripts. - submit (
bool
, optional) – Submit all jobs to the cluster. - queue (
str
, optional) – Name of the destination queue. When submitting jobs at Utah, this should not be set (leaving it at the default of None). When submitting jobs at Portsmouth, this can be used to select either sciama1.q, cluster.q (default), or sciama3.q.
-
clobber
¶ Only used with :attr:`all`. Clobber and redo any existing analysis.
Type: bool
-
quiet
¶ Suppress output
Type: bool
-
print_version
¶ Print the version and then return
Type: bool
-
strictver
¶ Strictly check the version requirements for this version of the dap. If True, an exception is raised if the expected version are not the same as the environment.
Type: bool
-
mpl
¶ MPL version; see above.
Type: mangadap.survey.mangampl.MaNGAMPL
-
redux_path
¶ The top-level path with the DRP files.
Type: str
-
dapver
¶ The DAP version.
Type: str
-
analysis_path
¶ The top-level path for the DAP output files.
Type: str
-
plan_file
¶ Name of the plan file to use for ALL DRP data in this run.
Type: str
-
plan
¶ The set of analysis plans
Type: AnalysisPlanSet
-
daptypes
¶ This list of daptypes, one per plan
Type: list
-
platelist
¶ List of plates to analyze.
Type: list
-
ifudesignlist
¶ List of ifudesigns to analyze.
Type: list
-
combinatorics
¶ Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze.
Type: bool
-
use_platetargets
¶ See above
Type: bool
-
platetargets
¶ List of platetargets files to search through to find any given plate-ifudesign combination.
Type: list
-
on_disk
¶ See above
Type: bool
-
log
¶ See above
Type: str
-
dapproc
¶ Flag to execute the main DAP processing
Type: bool
-
pltifu_plots
¶ Create the QA plate-ifu specific plots
Type: bool
-
post_process
¶ See above
Type: bool
-
post_plots
¶ See above
Type: bool
-
report_progress
¶ See abve
Type: bool
-
label
¶ Label to use in cluster queue.
Type: str
-
nodes
¶ Number of cluster nodes to use.
Type: int
-
qos
¶ Specific processor to use (Utah specific)
Type: str
-
umask
¶ umask to set for output.
Type: str
-
walltime
¶ Maximum wall time for cluster job.
Type: str
-
hard
¶ Same as hard keyword in cluster submission; see
pbs.queue.commit()
.Type: bool
-
create
¶ Use the pbs package to create the submission scripts.
Type: bool
-
submit
¶ Submit all jobs to the cluster.
Type: bool
-
q
¶ Queue destination (Portsmouth specific)
Type: str
-
drpc
¶ Database of the available DRP files and the parameters necessary to write the DAP par files.
Type: mangadap.survey.drpcomplete.DRPComplete
Raises: FileNotFoundError
– Raised if a plate-ifu list file is provided but it does not exist, or if the plan file does not exist.ValueError
– Raised if all processing steps are turned off, if the selected MPL is undefined; if the fast node is selected with nodes > 1; if multiple run modes are selected; or if the plate list, ifudesign list, or mode list are not provided during a redo mode or if they are provided with any other mode.
-
_build_proc_queue
(drpc_rows)[source]¶ Create a queue instance, write the script files for all CPUs using
prepare_for_analysis()
and append the job to the queue, and then submit the jobs to the cluster queue. Ifsubmit
is False, this is essentially a wrapper forprepare_for_analysis()
.This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth. The available queues are:
Queue Nodes PPN GB/core sciama1.q 80 12 2 sciama2.q 96 16 4 sciama3.q 48 20 3.2 One should not submit to sciama3.q without permission from Will Percival. The default queue is cluster.q (sciama2.q); so basically only use
q
to select sciama1.q.Submissions to Utah should have their queue destination set to None. However, the fast node can be selected using
qos
.Todo
- This algorithm is slow because it has to search through the drpcomplete fits file each time to find the plate/ifudesign combo. Should make this more efficient.
Parameters: drpc_rows (numpy.ndarray) – A vector with the rows in the mangadap.survey.drpcomplete.DRPComplete
database with the DRP files to analyze.
-
_check_paths
(plate, ifudesign)[source]¶ Check if the output paths exists for a given plate and ifudesign. If not, create them. The necessary output paths are:
- The calling path that contains a copy of the plan, the script input and output files, and the touch files
- The main common path defined by
mangadap.config.defaults.default_dap_common_path()
. - The plan-based subdirectories based on the plan file
Parameters: - plate (
int
) – Plate number - ifudesign (
int
) – ifudesign number
-
_create_queue
()[source]¶ Create a queue instance.
This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth. The available queues are:
Queue Nodes PPN GB/core sciama1.q 80 12 2 sciama2.q 96 16 4 sciama3.q 48 20 3.2 One should not submit to sciama3.q without permission from Will Percival. The default queue is cluster.q (sciama2.q); so basically only use
q
to select sciama1.q.Submissions to Utah should have their queue destination set to None. However, the fast node can be selected using
qos
.
-
_read_arg
()[source]¶ Read and interpret the terminal command-line arguments.
Function uses the argparse module, which provide a –help option to list all available arguments. The arguments mimic those required by the class constructor.
-
_selected_drpfile_rows
()[source]¶ Find the rows in the
mangadap.survey.drpcomplete.DRPComplete
database with the selected plates and IFUs to analyze.Returns: A vector with the rows in the DRPComplete object with the relevant DRP file data. Return type: numpy.ndarray
-
set_pltifu_status
(plate, ifudesign, mode=None, status='queued')[source]¶ Generate a touch file that signifies the status for a given plate/ifudesign/mode. The root of the touch file is set by
mangadap.config.defaults.default_dap_file_root()
.Parameters: - plate (int) – Plate number
- ifudesign (int) – IFU design number
- mode (str) – (Optional) DRP 3D mode, ‘RSS’ or ‘CUBE’
- status (str) – (Optional) That status signifier for the touch file.
-
set_status
(root, status='queued')[source]¶ Generate a general touch file.
Parameters: - root (str) – Full path and root name of the touch file
- status (str) – (Optional) That status signifier for the touch file.
-
write_compute_script
(index, dapproc=True, plots=True, clobber=False, relative_symlink=True)[source]¶ Write the MaNGA DAP script file that is sent to a single CPU to analyze a single DRP file with a given plate, ifudesign, and mode.
Parameters: - index (
int
, optional) – Row in themangadap.survey.drpcomplete.DRPComplete
database with the DRP file to analysis. - dapproc (
bool
, optional) – Flag to execute the main DAP processing. - plots (
bool
, optional) – Create the QA plots. - clobber (
bool
, optional) – Flag to clobber any existing files. - relative_symlink (
bool
, optional) – Set the symlink to the par file to be a relative path instead of the absolute path.
Returns: Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.
Return type: str
Raises: Exception
– Raised if DAP version is not correctly defined.- index (
-
write_dapall_script
(plots=True, clobber=False)[source]¶ Write the script used to construct the DAPall file and its associated quality assessment plots.
Parameters: - plots (bool) – (Optional) Create the QA plots. Default is True.
- clobber (bool) – (Optional) Flag to clobber any existing files.
Returns: Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.
Return type: str
Raises: ValueError
– Raised if DAP version is not correctly defined.
-
write_plate_qa_script
(plate, clobber=False)[source]¶ Write the script used to create the plate fit QA plot.
Parameters: - plate (
int
) – The plate number for the plot. - clobber (
bool
, optional) – Flag to clobber any existing files.
Returns: Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.
Return type: str
- plate (
- Have