mangadap.survey.rundap module

Defines the class used to automate the batch execution of the MaNGA DAP.

License

Copyright © 2019, SDSS-IV/MaNGA Pipeline Group

class mangadap.survey.rundap.rundap(overwrite=None, quiet=False, drpver=None, redux_path=None, dapver=None, analysis_path=None, plan_file=None, platelist=None, ifudesignlist=None, combinatorics=False, list_file=None, sres_ext=None, sres_fill=None, covar_ext=None, on_disk=False, can_analyze=False, log=False, dapproc=True, pltifu_plots=True, post_process=False, post_plots=False, report_progress=False, verbose=0, label='mangadap', nodes=1, cpus=None, qos=None, umask='0027', walltime='240:00:00', hard=True, create=False, submit=False, queue=None)[source]

Bases: object

Class used for automated execution of the MaNGA DAP via submission of scripts to a cluster.

[ More complete explanation of the class here. ]

Note

This is purely a survey-level utility for running the DAP at Utah, including submitting jobs to the cluster. No user-level usage of the DAP should be considered here, apart from not hindering such usage of the primary python programs!

Todo

  • Have _read_arg() return the boolean for version instead of keeping it as an attribute.

  • Ditch console and _read_arg() in favor of the more typical parser and main pairing.

Parameters:

  • overwrite (bool, optional) – Overwrite and redo any existing analysis.

  • console (bool, optional) – The class has been executed from the terminal with command-line arguments that should be parsed.

  • quiet (bool, optional) – Suppress output

  • print_version (bool, optional) – Print the class version and return.

  • drpver (str, optional) – DRP version to analyze. Default is defined by mangadap.config.manga.drp_version()

  • redux_path (str, optional) – The top-level path with the DRP files used to override the default defined by mangadap.config.manga.drp_redux_path().

  • dapver (str, optional) – The DAP version to use for the analysis, used to override the default defined by mangadap.config.manga.dap_version(). Only used to define the file paths, not used to switch to the specified code version!

  • analysis_path (str, optional) – The top-level path for the DAP output files, used to override the default defined by mangadap.config.manga.dap_analysis_path().

  • plan_file (str, optional) – Name of the plan file to use for all DRP data in this run, used to override the default defined by mangadap.par.analysisplan.AnalysisPlanSet.default().

  • platelist (int, str, optional) – List of plates to analyze; default is to search the redux_path for any DRP files; see mangadap.survey.drpcomplete.DRPComplete.

  • ifudesignlist (int, str, optional) – List of ifudesigns to analyze: default is to search the redux_path for any DRP files; see mangadap.survey.drpcomplete.DRPComplete.

  • combinatorics (bool, optional) – Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze.

  • list_file (str, optional) – File with a list of plates and ifudesigns to analyze. The file must have two columns, with one plate-ifu combination per file row.

  • sres_ext (str, optional) – The extension to use when constructing the spectral resolution vectors for the MaNGA datacubes. See mangadap.datacube.manga.MaNGADataCube.spectral_resolution().

  • sres_fill (bool, optional) – Fill masked spectral-resolution data by simple linear interpolation.

  • covar_ext (str, optional) – Extension in the MaNGA DRP CUBE file to use as the single spatial correlation matrix for all wavelength channels.

  • on_disk (bool, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file.

  • can_analyze (bool, optional) – Only construct script files for datacubes that can/should be analyzed by the DAP. See can_analyze().

  • log (bool, optional) – Flag to create the DAP log files.

  • dapproc (bool, optional) – Flag to execute the main DAP processing.

  • pltifu_plots (bool, optional) – Create the QA plate-ifu specific plots.

  • post_process (bool, optional) – Prepare (and possibly perform) the post processing steps: create the DAPall file and its associated QA plots, and the fit QA plots for each plate.

  • post_plots (bool, optional) – Create the post-processing QA plots.

  • report_progress (bool, optional) – Report the progress of the analysis.

  • vebose (int, optional) – Verbosity level between 0 and 2.

  • label (str, optional) – Label to use in cluster queue.

  • nodes (int, optional) – Number of cluster nodes to use.

  • qos (str, optional) – Select a specific processor (e.g. sdss-fast when the manga user). This option is ignored if q is set.

  • umask (str, optional) – umask to set for output.

  • walltime (str, optional) – Maximum wall time for cluster job.

  • hard (bool, optional) – Same as hard keyword in cluster submission; see pbs.queue.commit(). This keyword is in the opposite sense of the “–toughness” command-line options; i.e. including –toughness on the command line sets hard=False.

  • create (bool, optional) – Use the pbs package to create the scripts.

  • submit (bool, optional) – Submit all jobs to the cluster.

  • queue (str, optional) – Name of the destination queue. When submitting jobs at Utah, this should not be set (leaving it at the default of None). When submitting jobs at Portsmouth, this can be used to select either sciama1.q, cluster.q (default), or sciama3.q.

overwrite

Only used with :attr:`all`. Overwrite and redo any existing analysis.

Type:

bool

quiet

Suppress output

Type:

bool

print_version

Print the version and then return

Type:

bool

mpl

MPL version; see above.

Type:

mangadap.survey.mangampl.MaNGAMPL

redux_path

The top-level path with the DRP files.

Type:

str

dapver

The DAP version.

Type:

str

analysis_path

The top-level path for the DAP output files.

Type:

str

plan_file

Name of the plan file to use for ALL DRP data in this run. Will be None if using the default plan.

Type:

str

plan

The set of analysis plans

Type:

AnalysisPlanSet

daptypes

This list of daptypes, one per plan

Type:

list

platelist

List of plates to analyze.

Type:

list

ifudesignlist

List of ifudesigns to analyze.

Type:

list

combinatorics

Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze.

Type:

bool

on_disk

See above

Type:

bool

can_analyze

See above

Type:

bool

log

See above

Type:

str

dapproc

Flag to execute the main DAP processing

Type:

bool

pltifu_plots

Create the QA plate-ifu specific plots

Type:

bool

post_process

See above

Type:

bool

post_plots

See above

Type:

bool

report_progress

See abve

Type:

bool

label

Label to use in cluster queue.

Type:

str

nodes

Number of cluster nodes to use.

Type:

int

qos

Specific processor to use (Utah specific)

Type:

str

umask

umask to set for output.

Type:

str

walltime

Maximum wall time for cluster job.

Type:

str

hard

Same as hard keyword in cluster submission; see pbs.queue.commit().

Type:

bool

create

Use the pbs package to create the submission scripts.

Type:

bool

submit

Submit all jobs to the cluster.

Type:

bool

q

Queue destination (Portsmouth specific)

Type:

str

drpc

Database of the available DRP files and the parameters necessary to write the DAP par files.

Type:

mangadap.survey.drpcomplete.DRPComplete

Raises:

  • FileNotFoundError – Raised if a plate-ifu list file is provided but it does not exist, or if the plan file does not exist.

  • ValueError – Raised if all processing steps are turned off, if the selected MPL is undefined; if the fast node is selected with nodes > 1; if multiple run modes are selected; or if the plate list, ifudesign list, or mode list are not provided during a redo mode or if they are provided with any other mode.

_build_post_queue(drpc_rows)[source]

Build the queue that performs the post-processing steps.

_build_proc_queue(drpc_rows)[source]

Create a queue instance, write the script files for all CPUs using prepare_for_analysis() and append the job to the queue, and then submit the jobs to the cluster queue. If submit is False, this is essentially a wrapper for prepare_for_analysis().

This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth; see _create_queue().

Todo

  • This algorithm is slow because it has to search through the drpcomplete fits file each time to find the plate/ifudesign combo. Should make this more efficient.

Parameters:

drpc_rows (numpy.ndarray) – A vector with the rows in the mangadap.survey.drpcomplete.DRPComplete database with the DRP files to analyze.

_create_queue()[source]

Create a queue instance.

This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth. The available queues are:

Queue

Nodes

PPN

GB/core

ember

??

12

??

sciama1.q

80

12

2

sciama2.q

96

16

4

sciama3.q

48

20

3.2

One should not submit to sciama3.q without permission from Will Percival. The default queue is cluster.q (sciama2.q).

Submissions to Utah should have their queue destination set to None. However, the fast node can be selected using qos.

_read_file_list(list_file)[source]
_selected_drpfile_rows(can_analyze=False)[source]

Find the rows in the mangadap.survey.drpcomplete.DRPComplete database with the selected plates and IFUs to analyze.

Parameters:

can_analyze (bool, optional) – In addition to returning the selected rows, only include those rows with DRP datacubes that can be analyzed. See mangadap.survey.drpcomplete.DRPComplete.can_analyze().

Returns:

A vector with the rows in the DRPComplete object with the relevant DRP file data.

Return type:

numpy.ndarray

set_pltifu_status(plate, ifudesign, status='queued')[source]

Generate a touch file that signifies the status for a given plate/ifudesign/mode.

Parameters:

  • plate (int) – Plate number

  • ifudesign (int) – IFU design number

  • status (str, optional) – The status signifier for the touch file.

set_status(root, status='queued')[source]

Generate a general touch file.

Parameters:

  • root (str) – Full path and root name of the touch file

  • status (str, optional) – The status signifier for the touch file.

write_compute_script(index, dapproc=True, plots=True, overwrite=False, relative_symlink=True)[source]

Write the MaNGA DAP script file that is sent to a single CPU to analyze a single DRP file with a given plate and ifudesign.

Parameters:

  • index (int, optional) – Row in the mangadap.survey.drpcomplete.DRPComplete database with the DRP file to analysis.

  • dapproc (bool, optional) – Flag to execute the main DAP processing.

  • plots (bool, optional) – Create the QA plots.

  • overwrite (bool, optional) – Flag to overwrite any existing files.

  • relative_symlink (bool, optional) – Set the symlink to the par file to be a relative path instead of the absolute path.

Returns:

Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.

Return type:

str

write_dapall_script(plots=True, overwrite=False)[source]

Write the script used to construct the DAPall file and its associated quality assessment plots.

Parameters:

  • plots (bool, optional) – Create the QA plots. Default is True.

  • overwrite (bool, optional) – Flag to overwrite any existing files.

Returns:

Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.

Return type:

str

write_plate_qa_script(plate, overwrite=False)[source]

Write the script used to create the plate fit QA plot.

Parameters:

  • plate (int) – The plate number for the plot.

  • overwrite (bool, optional) – Flag to overwrite any existing files.

Returns:

Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.

Return type:

str