.. include:: include/links.rst ********************* What's New in the DAP ********************* Version 4.2.0 ============= - Deprecate support for python 3.10 - Change TOML dependency from ``tomli`` to ``tomllib``. - Verify support for numpy>=2.0 and bump dependency to pydl>=1.0 - Adds "MASTARHC2-NOISM" and "MASTARSSP-NOISM" templates Version 4.1.2 ============= - Fixed a bug when using the covariance calibration method, instead of a direct calculation of the covariance matrix. Thanks to reporter from issue #115, @iandroberts. - Minor package updates: bumps python support to 3.10-3.12, minor doc updates, and removes use of deprecated ``pkg_resources``. Version 4.1.1 ============= - Fixed bug in initial guess used for emission-line kinematics. Now ensures that initial guess satisfies any imposed inequalities. Version 4.1.0 ============= User-level changes/bug fixes ---------------------------- * Changes made to format of emission-line database files to allow for more flexible line tying. See updates to :ref:`emissionlines`. * Included a script to construct synthetic MaNGA datacubes for testing purposes. Version 4.0.0 ============= User-level changes/bug fixes ---------------------------- * Major overhaul of how paths and output files are defined. * Input paths and file names now defined as part of the :class:`~mangadap.datacube.datacube.DataCube` subclass. For MaNGA, this is done by subclassing the :class:`~mangadap.datacube.manga.MaNGADataCube` from the :class:`~mangadap.config.manga.MaNGAConfig` class. For user-defined :class:`~mangadap.datacube.datacube.DataCube` subclasses, these paths can be defined directly. * Output paths for primary output files now defined by the :class:`~mangadap.config.analysisplan.AnalysisPlan` object or a relevant subclass. For MaNGA, a subclass is necessary to put all the files in a subdirectory using the `plate` and `ifudesign` numbers, but likely most other implementations can use the :class:`~mangadap.config.analysisplan.AnalysisPlan` object directly. * Update to ingestion of custom :class:`~mangadap.datacube.datacube.DataCube` class in `manga_dap` script; new code allows for imports from a local user file. * Update `manga_dap_inspector` script to use a cursor on the map selector and a range slider for the vmin,vmax in the map. Some bugs persist in the map z range. * Refactor to simplify installation (inspired by work on `pypeit` package) * Improved/Added documentation for how to fit one spectrum and to fit a (non-MaNGA) datacube. * Major refactor of how module parameters are input and changed; analysis plan is now provided/changed via a toml file. Under-the-hood changes/bug fixes -------------------------------- * Deal with cubes where all the spaxels have valid data. * Remove "filtered" fitting from pPXF module * Remove all instances of `numpy.float`, `numpy.int`, `numpy.bool`, given impending numpy deprecation of these types. MPL-11 (3.1.0) ============== High-level changes ------------------ * Most analysis approaches (``DAPTYPE``) now use a set of MaStar Simple Stellar Population models as the continuum for the emission-line fitting module. The models have been published by `Maraston et al. (2020, MNRAS, 496, 2962)`_; see also https://www.icg.port.ac.uk/MaStar/. The models used are a subset of the full model library, which spans the full range of parameter space of the full library but enables a faster execution time. The weighting of the different models is **not** constrained to be smooth or to match any parametric star-formation/chemical-enrichment history. The models included in the fit have the following grid of physical parameters:: Age [Gyr]: 0.00316228, 0.01, 0.03162278, 0.100, 0.300, 1.000, 3.000, 9.000, 14.000 log(Z/Z_sun): -1.35, -1., -0.7, -0.33, 0, 0.35 IMF Slope: 2.35 (Salpeter) * The MPL-11 results include the usual 3 binning schemes used with the MaStar SSP templates, as well as a fourth ``DAPTYPE`` that uses the same hierarchically clustered MaStar templates from the MPL-10 release. This fourth analysis approach provides a useful baseline for assessing the effects of the change in the continuum on the emission-line results; i.e., all differences in the analysis results between the ``HYB10-MILESHC-MASTARSSP`` and ``HYB10-MILESHC-MASTARHC2`` results should be driven by the difference in the continuum templates used in the emission-line fitting module. User-level changes/bug fixes ---------------------------- * A bug in the reconstruction of the model spectra, **after** the model had been fit, meant that some regions that were masked were incorrectly reconstructed. This did not affect the parameters in the MAPS files or the emission-line-only or stellar-continuum-only spectra. This has now been fixed. * Significant revision to the format of the input files used to provide the list of emission-lines to fit. This change primarily enables the DAP to automatically construct the matrices used by `ppxf`_ to constrain the kinematics of different kinematic components. This functionality was used to test the results of tying the dispersion of emission lines to, say, all be within a factor of 2 of the H-alpha velocity dispersion. In the end, these constraints **were not** implemented in this MPL; the tying strategy for MPL-11 is identical to MPL-10. * The rest wavelengths of a number of emission lines have been updated to better account for the affects of fine-structure blending. For example, the rest wavelength (in vacuum) of the H-alpha line was 6564.632 angstrom in MPL-10, but it is 6564.608 in MPL-11. This is representative of the wavelength changes; i.e., the differences are typically a few hundredths of an angstrom. This change should have a very limited effect on the emission-line measurements. Under-the-hood changes/bug fixes -------------------------------- * Added the :class:`~mangadap.util.datatable.DataTable` class to handle the binary tables in the reference files, mostly to enable a uniform data model and enable automated documentation. * Enabled fitting a restricted wavelength range in the emission-line module. * Add parameter to change the required fraction of spectral coverage to perform a fit. * Detailed handling of masks for binning, MW extinction correction, and emission-line continuum construction. * Enable the output grid for :class:`~mangadap.util.sampling.Resample` to be defined directly. * Include covariance calculation in :class:`~mangadap.util.sampling.Resample`. * Add a class that reads trace sets produced by IDLUTILS, :class:`~mangadap.util.trace.TraceSet`, but not well tested yet. * Fix untested velocity registration and stacking code, and added registration test. * Upgraded package requirements * Line profile in :class:`~mangadap.proc.emissionlinetempltes.EmissionLineTemplates` now hardwired to be :class:`~mangadap.util.lineprofiles.FFTGaussianLSF`. MPL-10 (3.0.1) ============== High-level changes ------------------ * Incorporates the new LSF measurements from the DRP. * The emission-line module now uses a new set of hierarchically cluster MaStar spectra to model the stellar continuum; the library is called ``MASTARHC2`` (for now) to distinguish it from the templates used in MPL-9. * Significant additions to the spectral index calculations. See :class:`~mangadap.proc.spectralindices.AbsorptionLineIndices`, :class:`~mangadap.proc.spectralindices.BandheadIndices`, and changes to the :ref:`datamodel-maps`. Specifically, with respect to the change in the data model, the following channels were **removed**: ``SPECINDEX_BCEN``, ``SPECINDEX_BCNT``, ``SPECINDEX_RCEN``, and ``SPECINDEX_RCNT``. The following channels were **added**: * ``SPECINDEX_BF``: A calculation of the index with a modified definition from Burstein et al. (1984) and Faber et al. (1985), instead of the more typical definitions of Worthey et al. (1994) and Trager et al. (1998). See :class:`~mangadap.proc.spectralindices.AbsorptionLineIndices`. The bandhead or color indices in this extension are identical to the values in the ``SPECINDEX`` extension. * ``SPECINDEX_BF_IVAR``, ``SPECINDEX_BF_MASK``: Inverse variance and mask for the "BF" spectral indices. * ``SPECINDEX_BF_CORR``: Stellar velocity dispersion corrections for the "BF" indices. * ``SPECINDEX_BF_MODEL``: BF indices measured using the best-fitting model spectrum. * ``SPECINDEX_WGT``: Weights to use when calculating an index by aggregating index measurements from many spaxels/bins. See :class:`~mangadap.proc.spectralindices.AbsorptionLineIndices` and :class:`~mangadap.proc.spectralindices.BandheadIndices`. * ``SPECINDEX_WGT_IVAR``, ``SPECINDEX_WGT_MASK``: Inverse variance and mask for index weights. * ``SPECINDEX_WGT_CORR``: Stellar velocity dispersion corrections for the index weights. * ``SPECINDEX_WGT_MODEL``: Index weight based on the best-fitting model spectrum. * The estimated dispersion in the line-spread function of the binned spectra is now included in the :ref:`datamodel-cube` in the new ``LSF`` extension. In MPL-10, the ``LSFPRE`` extension from the DRP LOGCUBE is used for the spectral resolution of each spaxel. * DAPall file is now divided into one extension for each analysis method; see the desciption of the :ref:`metadatamodel-dapall`. User-level changes/bug fixes ---------------------------- * Major revision of the front-end to ease incorporation of non-MaNGA datacubes. * Major restructuring of directories to ease `pip`_ distribution and provide a more formal structure of the executables in ``bin`` and their associated scripts. * Added ``bin/dap_status`` to check the status of a batch DAP run. * For instantiating a :class:`~mangadap.datacube.manga.MaNGADataCube`, changed from using a yanny par file to a configuration (ini) file. Included code that can write these files using data from the DRPall or DRPComplete files. * Spectral resolution is no longer chosen by the spectral binning module; instead the spectral resolution is selected when reading the datacube (makes much more sense!). Led to some clean-up of the binning config files. To select different spectral resolution extensions, use command-line arguments in ``rundap`` or ``write_dap_config``. * Include a default analysis plan, so that executions of ``manga_dap`` don't require an analysis plan yanny parameter file. Under-the-hood algorithmic changes ---------------------------------- * The effective wavelength used for the two sidebands when calculating absorption-line indices now uses the center of the band, instead of the flux-weighted center of the band. Under-the-hood changes/bug fixes -------------------------------- * Remove "default" prefix from methods in :mod:`mangadap.config.defaults`. * Remove obsolete ``dapsrc`` keyword arguments * Some general clean-up of commented code and docs. * Import clean-up, including removal of any ``from __future__`` imports. * ``__credits__`` now include all authors of the DAP papers * Added new :class:`~mangadap.datacube.datacube.DataCube` and :class:`~mangadap.spectra.rowstackedspectra.RowStackedSpectra` classes, beginning the generalization of the interface with the input data for use with non-MaNGA data. * Integrated use of :class:`~mangadap.datacube.manga.MaNGADataCube` instead of :class:`~mangadap.util.drpfits.DRPFits` when executing the pipeline. Old :class:`~mangadap.util.drpfits.DRPFits` is now deprecated; :class:`~mangadap.util.drpfits.DRPFits` now repurposed to provide functionality common to both :class:`~mangadap.datacube.manga.MaNGADataCube` and :class:`~mangadap.spectra.manga.MaNGARSS`. * Included a script that will download data into a new ``mangadap/data/remote`` directory for testing. The directory is not included in the repo and has been added to ``.gitignore`` to prevent it from being accidentally added. * Added a number of tests that use the new remote data, including a nominal run of ``manga_dap`` using the ``ALL`` binning scheme. These will be skipped if the remote data is not available. * Significant improvements and testing of :mod:`mangadap.util.covariance`. Ensuring that the correlation matrices provided by the DRP can be read by :class:`~mangadap.datacube.manga.MaNGADataCube` and are effectively identical to the calculation performed by :class:`~mangadap.spectra.manga.MaNGARSS`. * Moved all core script code from ``bin`` to ``mangadap/scripts``. Code in ``bin`` now make simple calls to these scripts. Moved ``rundap.py`` from :mod:`mangadap.survey` to :mod:`mangadap.scripts`. * Usage of :class:`ObsInputPar` is now obsolete and deprecated. * Docstring updates for modules up through :class:`~mangadap.proc.stellarcontinuummodel.StellarContinuumModel`, but still lots to do. * Fixed a :math:`(1+z)` bug in the calculation of the spectral index errors. * Package version requirements were updated in general. * Now uses ``ppxf`` version 7.2.0, but still uses the default ``linear_method``. Use of ``linear_method='lsqlin'`` requires more testing. MPL-9 (2.4.1) ============= High-level changes ------------------ * Extensions ``SPX_ELLCOO`` and ``BIN_LWELLCOO`` have a new channel. The 4 channels are now radius in arcsec, radius in :math:`R_{\rm eff}`, radius in :math:`h^{-1}{\rm kpc}`, and azimuth; the third channel is the new one. So the channels are:: C1 = 'Elliptical radius' / Data in channel 1 U1 = 'arcsec ' / Units of data in channel 1 C2 = 'R/Re ' / Data in channel 2 U2 = '' / Units of data in channel 2 C3 = 'R h/kpc ' / Data in channel 3 U3 = 'kpc/h ' / Units of data in channel 3 C4 = 'Elliptical azimuth' / Data in channel 4 U4 = 'degrees ' / Units of data in channel 4 * The following extensions were added to the `MAPS` files: * ``EMLINE_SEW_CNT``: Continuum used for summed-flux equivalent-width measurement * ``EMLINE_GEW_CNT``: Continuum used for Gaussian-fit equivalent-width measurement * ``SPECINDEX_BCEN``: Flux-weighted center of the blue sideband in the spectral-index measurement * ``SPECINDEX_BCNT``: Continuum level in the blue sideband used in the spectral-index measurement * ``SPECINDEX_RCEN``: Flux-weighted center of the red sideband in the spectral-index measurement * ``SPECINDEX_RCNT``: Continuum level in the red sideband used in the spectral-index measurement * ``SPECINDEX_MODEL``: Index measurement made using the best-fitting stellar continuum model. * All emission-line extensions now contain 35 lines, and channel ordering of the line list has changed compared to MPL-8. Current ordering is:: C01 = 'OII-3727' / Data in channel 1 C02 = 'OII-3729' / Data in channel 2 C03 = 'H12-3751' / Data in channel 3 C04 = 'H11-3771' / Data in channel 4 C05 = 'Hthe-3798' / Data in channel 5 C06 = 'Heta-3836' / Data in channel 6 C07 = 'NeIII-3869' / Data in channel 7 C08 = 'HeI-3889' / Data in channel 8 C09 = 'Hzet-3890' / Data in channel 9 C10 = 'NeIII-3968' / Data in channel 10 C11 = 'Heps-3971' / Data in channel 11 C12 = 'Hdel-4102' / Data in channel 12 C13 = 'Hgam-4341' / Data in channel 13 C14 = 'HeII-4687' / Data in channel 14 C15 = 'Hb-4862 ' / Data in channel 15 C16 = 'OIII-4960' / Data in channel 16 C17 = 'OIII-5008' / Data in channel 17 C18 = 'NI-5199 ' / Data in channel 18 C19 = 'NI-5201 ' / Data in channel 19 C20 = 'HeI-5877' / Data in channel 20 C21 = 'OI-6302 ' / Data in channel 21 C22 = 'OI-6365 ' / Data in channel 22 C23 = 'NII-6549' / Data in channel 23 C24 = 'Ha-6564 ' / Data in channel 24 C25 = 'NII-6585' / Data in channel 25 C26 = 'SII-6718' / Data in channel 26 C27 = 'SII-6732' / Data in channel 27 C28 = 'HeI-7067' / Data in channel 28 C29 = 'ArIII-7137' / Data in channel 29 C30 = 'ArIII-7753' / Data in channel 30 C31 = 'Peta-9017' / Data in channel 31 C32 = 'SIII-9071' / Data in channel 32 C33 = 'Pzet-9231' / Data in channel 33 C34 = 'SIII-9533' / Data in channel 34 C35 = 'Peps-9548' / Data in channel 35 * Fixed flux ratios of doublets were improved using the following calculation: :math:`\lambda_2/\lambda_1\ \cdot\ (M1_1+E2_1)/(M1_2+E2_2)`, where :math:`M1` and :math:`E2` are the magnetic dipole and electric quadrapole Einstein :math:`A` coefficients and :math:`\lambda` is the line wavelength. The Einstein :math:`A` coefficients are all taken from NIST. In all but the [O II] 7320,7330 complex (not included in MPL-9 fit), the magnetic dipole term dominates such that the electric quadrapole terms are effectively irrelevant. The fixed flux ratios are: * NeIII-3968/NeIII-3968 = 0.3 * OIII-4960/OIII-5008 = 0.35 * OI-6365/OI-6302 = 0.32 * NII-6549/NII-6585 = 0.34 * SIII-9071/SIII-9533 = 0.41 * Lines with tied velocity dispersions (some because their fluxes are also tied) are: * OII-3727 + OII-3729 * HeI-3889 + Hzet-3890 * NeIII-3968 + NeIII-3968 * OIII-4960 + OIII-5008 * NI-5199 + NI-5201 * OI-6302 + OI-6365 * NII-6549 + NII-6585 * SIII-9071 + SIII-9533 User-level changes/bug fixes ---------------------------- * Allow emission-line moment, modeling, and spectral-index keys to be ``None``, meaning that the DAP only fits the stellar kinematics but still produces the MAPS and LOGCUBE files. * Added beta version of hierarchically clustered MaStar templates * Added a script that computes the weights used during the ``ppxffit_qa`` plot construction. Minor changes/bug fixes ----------------------- * Hotfix to accommodate change in padding computation in ``ppxf>=6.7.15`` * Fixed units bug in :func:`mangadap.proc.util.flux_to_fnu`. * Fixed bug in templates when using ``iteration_mode = 'global_template'`` * Change from ``time.clock()`` to ``time.perf_counter()`` * Bug fix in record array dimensionality when writing to binary table * Minor plotting changes for Overview paper plots * Fixed masking of emission-line chi-square measurements * Remove ``MASKNAME`` header keyword inherited from DRP files MPL-8 (2.3.0) ============= High-level changes ------------------ * Change to ``DAPTYPE`` construction. ``DAPTYPE`` is now ``binning`` - ``stellar templates`` - ``emission-line templates``. This was done because in future releases we plan to switch the templates used for the stellar kinematics (likely to remain ``MILESHC``) to a different template set for the emission-line modeling with a longer spectral range to fit the full MaNGA spectral range. * Three additional emission-lines are fit: He I at 3889 angstroms, and the [N I] doublet at 5200 angstroms. The He I line has its dispersion tied to :math:`{\rm H}\zeta` at 3890, and the dispersions of the [N I] doublet are tied. * The ``MAPS`` file extensions were modified: * ``STELLAR_SIGMACORR`` now has two channels. The first provides the correction constructed using the same methodology as in MPL-7/DR15; the second provides a correction that we are currently testing for robustness as a replacement correction. * The data in the ``STELLAR_CONT_FRESID`` and ``STELLAR_CONT_RCHI2`` extensions has been consolidated into a new single extension, ``STELLAR_FOM`` (FOM=figure-of-merit). Channel 3 of ``STELLAR_FOM`` is the same as ``STELLAR_CONT_RCHI2``, and channels 4 and 5 are the same as the two channels in ``STELLAR_CONT_FRESID``. See the data model for the full description of the additional channels in this extension. * 4 additional channels are provided related to the emission lines: * ``EMLINE_GA``: The amplitude of the fitted Gaussians * ``EMLINE_GANR``: The amplitude over noise of the fitted Gaussians * ``EMLINE_FOM``: Full-spectrum figures-of-merit for the emission-line module; this now has exactly the same format as the STELLAR_FOM extension * ``EMLINE_LFOM``: The reduced chi-square in 15 pixel windows around each fitted emission line. * To make the files easier to use, the ``LOGCUBE`` extensions were modified: * The ``EMLINE_BASE`` and ``EMLINE_MASK`` extensions have been removed. * The following extensions have been added: * ``MODEL_MASK`` is the mask to use with the ``MODEL`` extension (the result of the emission-line+continuum fit) * ``STELLAR`` is best-fitting stellar continuum from the stellar continuum fit * ``STELLAR_MASK`` is the mask for the stellar-continuum fit. * Numerous QA plots have been added; see the data model description. Under-the-hood algorithmic changes ---------------------------------- * Allow the emission-line fitter to use the bin ID numbers directly instead of matching the spaxels to bins by coordinate proximity * Construction of the parameter tying object in the emission-line fitter is now done just before each spectrum is fit by ppxf (not globally in Sasuke) to better handle when components are omitted. * When deconstructing bins into spaxels for the emission-line modeling (hybrid binning scheme), the second fit iteration only fits spaxels that are components of binned spectra and does not refit spectra that constitute an entire bin themselves. I.e. this removes some largely redundant fitting. * :class:`~mangadap.proc.ppxffit.PPXFFit` and :class:`~mangadap.proc.sasuke.Sasuke` include calculations of the chi-square growth; and changed the names of the growth columns in the reference files. * Changed definitions of :math:`A` to be the model amplitude; :math:`A/N` is the model amplitude divided by the median noise in the two sidebands defined for the emission-line EW calculation. * Major changes to survey-level execution of the DAP, including that the input data is now pulled from the ``DRPall`` file instead of the ``plateTargets`` files. * Ignore minimum :math:`S/N` limitation in emission-line moments and spectral indices for hybrid scheme as a stop-gap to minimize differences in moments, models, and indices ``BINID``. User-level changes/bug fixes ---------------------------- * Fixed an error in setting up the tying structure passed to pPXF that led to the [O II] dispersion maps being fully masked. * Fixed an error in the propagation of the error in the passband integral calculation, which affects the non-parameteric emission-line measurement errors (summed data) and the spectral index errors. The calculation is now formally correct, but these propagated errors are still underestimated with respect to more robust calculations via an MC. * Fixed a bug that omitted the ``FORESTAR`` flag from getting propagated in the hybrid binning case. * Fixed :math:`\chi^2` calculations reported in ``MAPS`` files (does not affect chi-square used during fit optimization) for both the stellar-continuum fit and the emission-line modeling. * Velocity-dispersion corrections are now applied to the spectral indices summary data provided in the ``DAPall`` file. * Fixed minor issue in propagating masks from the reference files to the maps files; primarily an issue for the hybrid binning scheme. Minor changes/bug fixes ----------------------- * Fixed the bug that led to the error in the sigma corrections for MPL-7, what were replaced before distributing these data via DR15. * Fixed bug that was causing multiple instances of "Warning: converting a masked element to nan" during the emission-line moment measurements. * Significant changes to the pixel resampling code, but has a minor effect on the results.