mangadap.survey.rundap module¶

Defines the class used to automate the execution of the MaNGA DAP at Utah.

Revision history¶

?? Nov 2014: Architecture setup by J. Brownstein <joelbrownstein@astro.utah.edu>

18 Nov 2014: Handover to K. Westfall (KBW) for development. (Attempt to) conform style to PEP 8, except that lines are 99 characters long (docstrings still limited to 72 characters). Also (attempt to) conform to docstrings style according to PEP 257.

01 Dec 2014: (KBW) Committed to SVN

02 Dec 2014: (KBW) Changed drpver and idlutilsver to mangaver

13 Jan 2015: (KBW) Changed it back, added platetargets

05 Feb 2015: (KBW) Change to load module based on MPL

19 Mar 2015: (KBW) Major changes to allow for more control over the directory structure containing the DRP data and for the DAP results; following changed in the drpfile and drpcomplete classes.

27 Aug 2015: (KBW) Sphinx documentation; prep for MPL-4; accommodate changes in drpcomplete; removed arg as an attribute because it was only used in mangadap.survey.rundap._read_arg(); despite not doing anything before, commented out _write_module_commands for the time-being; removed file_root() and parameter_file() functions in favor of adding/using functions from mangadap.config.defaults; version number set to 1_1_0.

06 Oct 2015: (KBW) Added functionality to select the types of additional output. Minor changes to allow for DR13 QA plots.

22 Oct 2015: (KBW) Added check that the selected MPL versions match the current environmental versions. Includes edit to mangadap.survey.mangampl.MaNGAMPL to include MANGACORE_VER. Changed default number of nodes to 9. Added definition of idl command for use on sciama cluster at the ICG, Portsmouth.

04 Nov 2015: (KBW) Added capability to turn off main DAP processing call so that rundap can be used to, e.g., just create the plots for existing data.

17 Feb 2016: (KBW) Added try/except block for importing pbs.queue; converted old drpfile to new mangadap.drpfits.DRPFits, and drpcomplete to new mangadap.survey.drpcomplete.DRPComplete.

13 May 2016: (KBW) Substantial changes to accommodate conversion of DAP from IDL to python

19 May 2016: (KBW) Added command-line options for including log file

11 Oct 2016: (KBW) Version checking moved to mangadap.survey.mangampl.MaNGAMPL.

23 Aug 2017: (KBW) Include ppxffit QA plots.

29 Aug 2017: (KBW) Include DAPall post-processing.

16 Jan 2019: (KBW) Major changes: Only one operation mode (daily, all, redo removed; code always run in “redo” mode). Remove old “prior” options. Setup done using DRPall by default, but option to use mangacore/platetargets. ‘CUBE’ vs. ‘RSS’ no longer an option, only analyzes CUBEs.

23 Jan 2019: (KBW) Add fit QA plots

class mangadap.survey.rundap.rundap(clobber=None, console=None, quiet=False, print_version=False, strictver=True, mplver=None, redux_path=None, dapver=None, analysis_path=None, plan_file=None, platelist=None, ifudesignlist=None, combinatorics=False, list_file=None, use_platetargets=False, platetargets=None, on_disk=False, log=False, dapproc=True, pltifu_plots=True, post_process=False, post_plots=False, report_progress=False, verbose=0, label='mangadap', nodes=1, cpus=None, qos=None, umask='0027', walltime='240:00:00', hard=True, create=False, submit=False, queue=None)[source]¶

Bases: object

Class used for automated execution of the MaNGA DAP via submission of scripts to a cluster.

[ More complete explanation of the class here. ]

Note

This is purely a survey-level utility for running the DAP at Utah, including submitting jobs to the cluster. No user-level usage of the DAP should be considered here, apart from not hindering such usage of the primary python programs!

Todo

Have _read_arg() return the boolean for version instead of keeping it as an attribute.

Parameters:

clobber (bool, optional) – Clobber and redo any existing analysis.
console (bool, optional) – The class has been executed from the terminal with command-line arguments that should be parsed.
quiet (bool, optional) – Suppress output
print_version (bool, optional) – Print the class version and return.
strictver (bool, optional) – Strictly check the version requirements of the dependencies defined in :mangadap.survey.mangampl.MaNGAMPL` for this version of the DAP. Default is True, meaning the code will raise an exception if the expected version are not the same as the environment.
mplver (str, optional) – MPL version to analyze. Used with mangadap.survey.mangampl.MaNGAMPL to set the DAP dependencies. Ideally, this controls the modules that are loaded before execution of the DAP. The default version (selected if mplver=None) is defined by mangadap.survey.mangampl.MaNGAMPL.
redux_path (str, optional) – The top-level path with the DRP files used to override the default defined by mangadap.config.defaults.default_redux_path().
dapver (str, optional) – The DAP version to use for the analysis, used to override the default defined by mangadap.config.defaults.default_dap_version(). Only used to define the file paths, not used to switch to the specified code version!
analysis_path (str, optional) – The top-level path for the DAP output files, used to override the default defined by mangadap.config.defaults.default_analysis_path().
plan_file (str, optional) – Name of the plan file to use for all DRP data in this run, used to override the default defined by mangadap.config.defaults.default_dap_plan_file().
platelist (int, str, optional) – List of plates to analyze; default is to search the redux_path for any DRP files; see mangadap.survey.drpcomplete.DRPComplete.
ifudesignlist (int, str, optional) – List of ifudesigns to analyze: default is to search the redux_path for any DRP files; see mangadap.survey.drpcomplete.DRPComplete.
combinatorics (bool, optional) – Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze. See mangadap.drpfits.drpfits_list().
use_platetargets (bool, optional) – Flag to use the plateTargets files, instead of the DRPall file, as the source of the input data need by the DAP.
platetargets (str, list, optional) – List of platetargets files to search through to find any given plate-ifudesign combination. Default is returned as the first element in mangadap.config.defaults.default_plate_target_files().
on_disk (bool, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file. This is implicitly true when using the plateTargets files to collate the data used for the DAP input instead of the DRPall file (i.e., when use_platetargets=True).
log (bool, optional) – Flag to create the DAP log files.
dapproc (bool, optional) – Flag to execute the main DAP processing.
pltifu_plots (bool, optional) – Create the QA plate-ifu specific plots.
post_process (bool, optional) – Prepare (and possibly perform) the post processing steps: create the DAPall file and its associated QA plots, and the fit QA plots for each plate.
post_plots (bool, optional) – Create the post-processing QA plots.
report_progress (bool, optional) – Report the progress of the analysis.
vebose (int, optional) – Verbosity level between 0 and 2.
label (str, optional) – Label to use in cluster queue.
nodes (int, optional) – Number of cluster nodes to use.
qos (str, optional) – Select a specific processor (e.g. sdss-fast when the manga user). This option is ignored if q is set.
umask (str, optional) – umask to set for output.
walltime (str, optional) – Maximum wall time for cluster job.
hard (bool, optional) – Same as hard keyword in cluster submission; see pbs.queue.commit(). This keyword is in the opposite sense of the “–toughness” command-line options; i.e. including –toughness on the command line sets hard=False.
create (bool, optional) – Use the pbs package to create the scripts.
submit (bool, optional) – Submit all jobs to the cluster.
queue (str, optional) – Name of the destination queue. When submitting jobs at Utah, this should not be set (leaving it at the default of None). When submitting jobs at Portsmouth, this can be used to select either sciama1.q, cluster.q (default), or sciama3.q.

clobber¶

Only used with :attr:`all`. Clobber and redo any existing analysis.

Type:	bool

quiet¶

Suppress output

Type:	bool

print_version¶

Print the version and then return

Type:	bool

strictver¶

Strictly check the version requirements for this version of the dap. If True, an exception is raised if the expected version are not the same as the environment.

Type:	bool

mpl¶

MPL version; see above.

Type:	`mangadap.survey.mangampl.MaNGAMPL`

redux_path¶

The top-level path with the DRP files.

Type:	str

dapver¶

The DAP version.

Type:	str

analysis_path¶

The top-level path for the DAP output files.

Type:	str

plan_file¶

Name of the plan file to use for ALL DRP data in this run.

Type:	str

plan¶

The set of analysis plans

Type:	`AnalysisPlanSet`

daptypes¶

This list of daptypes, one per plan

Type:	`list`

platelist¶

List of plates to analyze.

Type:	list

ifudesignlist¶

List of ifudesigns to analyze.

Type:	list

combinatorics¶

Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze.

Type:	bool

use_platetargets¶

See above

Type:	bool

platetargets¶

List of platetargets files to search through to find any given plate-ifudesign combination.

Type:	list

on_disk¶

See above

Type:	bool

log¶

See above

Type:	str

dapproc¶

Flag to execute the main DAP processing

Type:	bool

pltifu_plots¶

Create the QA plate-ifu specific plots

Type:	bool

post_process¶

See above

Type:	`bool`

post_plots¶

See above

Type:	`bool`

report_progress¶

See abve

Type:	`bool`

label¶

Label to use in cluster queue.

Type:	str

nodes¶

Number of cluster nodes to use.

Type:	int

qos¶

Specific processor to use (Utah specific)

Type:	str

umask¶

umask to set for output.

Type:	str

walltime¶

Maximum wall time for cluster job.

Type:	str

hard¶

Same as hard keyword in cluster submission; see pbs.queue.commit().

Type:	bool

create¶

Use the pbs package to create the submission scripts.

Type:	bool

submit¶

Submit all jobs to the cluster.

Type:	bool

q¶

Queue destination (Portsmouth specific)

Type:	str

drpc¶

Database of the available DRP files and the parameters necessary to write the DAP par files.

Type:	`mangadap.survey.drpcomplete.DRPComplete`

Raises:

FileNotFoundError – Raised if a plate-ifu list file is provided but it does not exist, or if the plan file does not exist.
ValueError – Raised if all processing steps are turned off, if the selected MPL is undefined; if the fast node is selected with nodes > 1; if multiple run modes are selected; or if the plate list, ifudesign list, or mode list are not provided during a redo mode or if they are provided with any other mode.

_build_post_queue(drpc_rows)[source]¶: Build the queue that performs the post-processing steps.

_build_proc_queue(drpc_rows)[source]¶

Create a queue instance, write the script files for all CPUs using prepare_for_analysis() and append the job to the queue, and then submit the jobs to the cluster queue. If submit is False, this is essentially a wrapper for prepare_for_analysis().

This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth. The available queues are:

Queue	Nodes	PPN	GB/core
sciama1.q	80	12	2
sciama2.q	96	16	4
sciama3.q	48	20	3.2

One should not submit to sciama3.q without permission from Will Percival. The default queue is cluster.q (sciama2.q); so basically only use q to select sciama1.q.

Submissions to Utah should have their queue destination set to None. However, the fast node can be selected using qos.

Todo

This algorithm is slow because it has to search through the drpcomplete fits file each time to find the plate/ifudesign combo. Should make this more efficient.

Parameters:	drpc_rows (numpy.ndarray) – A vector with the rows in the `mangadap.survey.drpcomplete.DRPComplete` database with the DRP files to analyze.

_check_paths(plate, ifudesign)[source]¶

Check if the output paths exists for a given plate and ifudesign. If not, create them. The necessary output paths are:

The calling path that contains a copy of the plan, the script input and output files, and the touch files

The main common path defined by mangadap.config.defaults.default_dap_common_path().

The plan-based subdirectories based on the plan file

Parameters:	plate (`int`) – Plate number ifudesign (`int`) – ifudesign number

_create_queue()[source]¶

Create a queue instance.

This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth. The available queues are:

Queue	Nodes	PPN	GB/core
sciama1.q	80	12	2
sciama2.q	96	16	4
sciama3.q	48	20	3.2

One should not submit to sciama3.q without permission from Will Percival. The default queue is cluster.q (sciama2.q); so basically only use q to select sciama1.q.

Submissions to Utah should have their queue destination set to None. However, the fast node can be selected using qos.

_read_arg()[source]¶

Read and interpret the terminal command-line arguments.

Function uses the argparse module, which provide a –help option to list all available arguments. The arguments mimic those required by the class constructor.

_read_file_list(list_file)[source]¶

_selected_drpfile_rows()[source]¶

Find the rows in the mangadap.survey.drpcomplete.DRPComplete database with the selected plates and IFUs to analyze.

Returns:	A vector with the rows in the DRPComplete object with the relevant DRP file data.
Return type:	numpy.ndarray

set_pltifu_status(plate, ifudesign, mode=None, status='queued')[source]¶

Generate a touch file that signifies the status for a given plate/ifudesign/mode. The root of the touch file is set by mangadap.config.defaults.default_dap_file_root().

Parameters:	plate (int) – Plate number ifudesign (int) – IFU design number mode (str) – (Optional) DRP 3D mode, ‘RSS’ or ‘CUBE’ status (str) – (Optional) That status signifier for the touch file.

set_status(root, status='queued')[source]¶

Generate a general touch file.

Parameters:	root (str) – Full path and root name of the touch file status (str) – (Optional) That status signifier for the touch file.

write_compute_script(index, dapproc=True, plots=True, clobber=False, relative_symlink=True)[source]¶

Write the MaNGA DAP script file that is sent to a single CPU to analyze a single DRP file with a given plate, ifudesign, and mode.

Parameters:	index (`int`, optional) – Row in the `mangadap.survey.drpcomplete.DRPComplete` database with the DRP file to analysis. dapproc (`bool`, optional) – Flag to execute the main DAP processing. plots (`bool`, optional) – Create the QA plots. clobber (`bool`, optional) – Flag to clobber any existing files. relative_symlink (`bool`, optional) – Set the symlink to the par file to be a relative path instead of the absolute path.
Returns:	Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.
Return type:	str
Raises:	`Exception` – Raised if DAP version is not correctly defined.

write_dapall_script(plots=True, clobber=False)[source]¶

Write the script used to construct the DAPall file and its associated quality assessment plots.

Parameters:	plots (bool) – (Optional) Create the QA plots. Default is True. clobber (bool) – (Optional) Flag to clobber any existing files.
Returns:	Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.
Return type:	str
Raises:	`ValueError` – Raised if DAP version is not correctly defined.

write_plate_qa_script(plate, clobber=False)[source]¶

Write the script used to create the plate fit QA plot.

Parameters:	plate (`int`) – The plate number for the plot. clobber (`bool`, optional) – Flag to clobber any existing files.
Returns:	Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.
Return type:	str