mangadap.survey.rundap module

Defines the class used to automate the batch execution of the MaNGA DAP.

License

Copyright © 2019, SDSS-IV/MaNGA Pipeline Group

class mangadap.survey.rundap.rundap(overwrite=None, quiet=False, drpver=None, redux_path=None, dapver=None, analysis_path=None, plan_file=None, platelist=None, ifudesignlist=None, combinatorics=False, list_file=None, sres_ext=None, sres_fill=None, covar_ext=None, on_disk=False, can_analyze=False, log=False, dapproc=True, pltifu_plots=True, post_process=False, post_plots=False, report_progress=False, verbose=0, label='mangadap', nodes=1, cpus=None, qos=None, umask='0027', walltime='240:00:00', hard=True, create=False, submit=False, queue=None)[source]

Bases: object

Class used for automated execution of the MaNGA DAP via submission of scripts to a cluster.

[ More complete explanation of the class here. ]

Note

This is purely a survey-level utility for running the DAP at Utah, including submitting jobs to the cluster. No user-level usage of the DAP should be considered here, apart from not hindering such usage of the primary python programs!

Todo

  • Have _read_arg() return the boolean for version instead of keeping it as an attribute.

  • Ditch console and _read_arg() in favor of the more typical parser and main pairing.

Parameters

  • overwrite (bool, optional) – Overwrite and redo any existing analysis.

  • console (bool, optional) – The class has been executed from the terminal with command-line arguments that should be parsed.

  • quiet (bool, optional) – Suppress output

  • print_version (bool, optional) – Print the class version and return.

  • drpver (str, optional) – DRP version to analyze. Default is defined by mangadap.config.manga.drp_version()

  • redux_path (str, optional) – The top-level path with the DRP files used to override the default defined by mangadap.config.manga.drp_redux_path().

  • dapver (str, optional) – The DAP version to use for the analysis, used to override the default defined by mangadap.config.manga.dap_version(). Only used to define the file paths, not used to switch to the specified code version!

  • analysis_path (str, optional) – The top-level path for the DAP output files, used to override the default defined by mangadap.config.manga.dap_analysis_path().

  • plan_file (str, optional) – Name of the plan file to use for all DRP data in this run, used to override the default defined by mangadap.par.analysisplan.AnalysisPlanSet.default().

  • platelist (int, str, optional) – List of plates to analyze; default is to search the redux_path for any DRP files; see mangadap.survey.drpcomplete.DRPComplete.

  • ifudesignlist (int, str, optional) – List of ifudesigns to analyze: default is to search the redux_path for any DRP files; see mangadap.survey.drpcomplete.DRPComplete.

  • combinatorics (bool, optional) – Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze.

  • list_file (str, optional) – File with a list of plates and ifudesigns to analyze. The file must have two columns, with one plate-ifu combination per file row.

  • sres_ext (str, optional) – The extension to use when constructing the spectral resolution vectors for the MaNGA datacubes. See mangadap.datacube.manga.MaNGADataCube.spectral_resolution().

  • sres_fill (bool, optional) – Fill masked spectral-resolution data by simple linear interpolation.

  • covar_ext (str, optional) – Extension in the MaNGA DRP CUBE file to use as the single spatial correlation matrix for all wavelength channels.

  • on_disk (bool, optional) – When searching for available files to analyze, search the DRP directory path instead of using the data in the DRPall file.

  • can_analyze (bool, optional) – Only construct script files for datacubes that can/should be analyzed by the DAP. See can_analyze().

  • log (bool, optional) – Flag to create the DAP log files.

  • dapproc (bool, optional) – Flag to execute the main DAP processing.

  • pltifu_plots (bool, optional) – Create the QA plate-ifu specific plots.

  • post_process (bool, optional) – Prepare (and possibly perform) the post processing steps: create the DAPall file and its associated QA plots, and the fit QA plots for each plate.

  • post_plots (bool, optional) – Create the post-processing QA plots.

  • report_progress (bool, optional) – Report the progress of the analysis.

  • vebose (int, optional) – Verbosity level between 0 and 2.

  • label (str, optional) – Label to use in cluster queue.

  • nodes (int, optional) – Number of cluster nodes to use.

  • qos (str, optional) – Select a specific processor (e.g. sdss-fast when the manga user). This option is ignored if q is set.

  • umask (str, optional) – umask to set for output.

  • walltime (str, optional) – Maximum wall time for cluster job.

  • hard (bool, optional) – Same as hard keyword in cluster submission; see pbs.queue.commit(). This keyword is in the opposite sense of the “–toughness” command-line options; i.e. including –toughness on the command line sets hard=False.

  • create (bool, optional) – Use the pbs package to create the scripts.

  • submit (bool, optional) – Submit all jobs to the cluster.

  • queue (str, optional) – Name of the destination queue. When submitting jobs at Utah, this should not be set (leaving it at the default of None). When submitting jobs at Portsmouth, this can be used to select either sciama1.q, cluster.q (default), or sciama3.q.

overwrite

Only used with :attr:`all`. Overwrite and redo any existing analysis.

Type

bool

quiet

Suppress output

Type

bool

print_version

Print the version and then return

Type

bool

mpl

MPL version; see above.

Type

mangadap.survey.mangampl.MaNGAMPL

redux_path

The top-level path with the DRP files.

Type

str

dapver

The DAP version.

Type

str

analysis_path

The top-level path for the DAP output files.

Type

str

plan_file

Name of the plan file to use for ALL DRP data in this run. Will be None if using the default plan.

Type

str

plan

The set of analysis plans

Type

AnalysisPlanSet

daptypes

This list of daptypes, one per plan

Type

list

platelist

List of plates to analyze.

Type

list

ifudesignlist

List of ifudesigns to analyze.

Type

list

combinatorics

Use all unique combinations of the entered plate/ifudesign/mode lists to create the full list of DRP files to analyze.

Type

bool

on_disk

See above

Type

bool

can_analyze

See above

Type

bool

log

See above

Type

str

dapproc

Flag to execute the main DAP processing

Type

bool

pltifu_plots

Create the QA plate-ifu specific plots

Type

bool

post_process

See above

Type

bool

post_plots

See above

Type

bool

report_progress

See abve

Type

bool

label

Label to use in cluster queue.

Type

str

nodes

Number of cluster nodes to use.

Type

int

qos

Specific processor to use (Utah specific)

Type

str

umask

umask to set for output.

Type

str

walltime

Maximum wall time for cluster job.

Type

str

hard

Same as hard keyword in cluster submission; see pbs.queue.commit().

Type

bool

create

Use the pbs package to create the submission scripts.

Type

bool

submit

Submit all jobs to the cluster.

Type

bool

q

Queue destination (Portsmouth specific)

Type

str

drpc

Database of the available DRP files and the parameters necessary to write the DAP par files.

Type

mangadap.survey.drpcomplete.DRPComplete

Raises

  • FileNotFoundError – Raised if a plate-ifu list file is provided but it does not exist, or if the plan file does not exist.

  • ValueError – Raised if all processing steps are turned off, if the selected MPL is undefined; if the fast node is selected with nodes > 1; if multiple run modes are selected; or if the plate list, ifudesign list, or mode list are not provided during a redo mode or if they are provided with any other mode.

_build_post_queue(drpc_rows)[source]

Build the queue that performs the post-processing steps.

_build_proc_queue(drpc_rows)[source]

Create a queue instance, write the script files for all CPUs using prepare_for_analysis() and append the job to the queue, and then submit the jobs to the cluster queue. If submit is False, this is essentially a wrapper for prepare_for_analysis().

This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth; see _create_queue().

Todo

  • This algorithm is slow because it has to search through the drpcomplete fits file each time to find the plate/ifudesign combo. Should make this more efficient.

Parameters

drpc_rows (numpy.ndarray) – A vector with the rows in the mangadap.survey.drpcomplete.DRPComplete database with the DRP files to analyze.

_create_queue()[source]

Create a queue instance.

This function is written to be functional for submission to the cluster queues both at Utah and using the sciama cluster at Portsmouth. The available queues are:

Queue

Nodes

PPN

GB/core

ember

??

12

??

sciama1.q

80

12

2

sciama2.q

96

16

4

sciama3.q

48

20

3.2

One should not submit to sciama3.q without permission from Will Percival. The default queue is cluster.q (sciama2.q).

Submissions to Utah should have their queue destination set to None. However, the fast node can be selected using qos.

_read_file_list(list_file)[source]
_selected_drpfile_rows(can_analyze=False)[source]

Find the rows in the mangadap.survey.drpcomplete.DRPComplete database with the selected plates and IFUs to analyze.

Parameters

can_analyze (bool, optional) – In addition to returning the selected rows, only include those rows with DRP datacubes that can be analyzed. See mangadap.survey.drpcomplete.DRPComplete.can_analyze().

Returns

A vector with the rows in the DRPComplete object with the relevant DRP file data.

Return type

numpy.ndarray

set_pltifu_status(plate, ifudesign, status='queued')[source]

Generate a touch file that signifies the status for a given plate/ifudesign/mode.

Parameters

  • plate (int) – Plate number

  • ifudesign (int) – IFU design number

  • status (str, optional) – The status signifier for the touch file.

set_status(root, status='queued')[source]

Generate a general touch file.

Parameters

  • root (str) – Full path and root name of the touch file

  • status (str, optional) – The status signifier for the touch file.

write_compute_script(index, dapproc=True, plots=True, overwrite=False, relative_symlink=True)[source]

Write the MaNGA DAP script file that is sent to a single CPU to analyze a single DRP file with a given plate and ifudesign.

Parameters

  • index (int, optional) – Row in the mangadap.survey.drpcomplete.DRPComplete database with the DRP file to analysis.

  • dapproc (bool, optional) – Flag to execute the main DAP processing.

  • plots (bool, optional) – Create the QA plots.

  • overwrite (bool, optional) – Flag to overwrite any existing files.

  • relative_symlink (bool, optional) – Set the symlink to the par file to be a relative path instead of the absolute path.

Returns

Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.

Return type

str

write_dapall_script(plots=True, overwrite=False)[source]

Write the script used to construct the DAPall file and its associated quality assessment plots.

Parameters

  • plots (bool, optional) – Create the QA plots. Default is True.

  • overwrite (bool, optional) – Flag to overwrite any existing files.

Returns

Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.

Return type

str

write_plate_qa_script(plate, overwrite=False)[source]

Write the script used to create the plate fit QA plot.

Parameters

  • plate (int) – The plate number for the plot.

  • overwrite (bool, optional) – Flag to overwrite any existing files.

Returns

Three strings with the name of the written script file, the file for the output sent to STDOUT, and the file for the output sent to STDERR.

Return type

str