mangadap.util.bitmask module¶
Base class for handling bit masks.
Class usage examples¶
Bitmasks allow you to define a set of bit values signified by strings, and then toggle and interpret bits held by a numpy.ndarray. For example, say you’re processing an image and you want to set up a set of bits that indicate that the pixel is part of a bad-pixel mask, has a cosmic ray, or is saturated. You can define the following:
from collections import OrderedDict
from mangadap.util.bitmask import BitMask
bits = OrderedDict([('BPM', 'Pixel is part of a bad-pixel mask'),
('COSMIC', 'Pixel is contaminated by a cosmic ray'),
('SATURATED', 'Pixel is saturated.')])
image_bm = BitMask(list(bits.keys()), descr=list(bits.values()))
Or, better yet, define a derived class:
from collections import OrderedDict
from mangadap.util.bitmask import BitMask
class ImageBitMask(BitMask):
def __init__(self):
bits = OrderedDict([('BPM', 'Pixel is part of a bad-pixel mask'),
('COSMIC', 'Pixel is contaminated by a cosmic ray'),
('SATURATED', 'Pixel is saturated.')])
super(ImageBitMask, self).__init__(list(bits.keys()), descr=list(bits.values()))
image_bm = ImageBitMask()
In either case, you can see the list of bits and their bit numbers by running:
>>> image_bm.info()
Bit: BPM = 0
Description: Pixel is part of a bad-pixel mask
Bit: COSMIC = 1
Description: Pixel is contaminated by a cosmic ray
Bit: SATURATED = 2
Description: Pixel is saturated.
>>> image_bm.bits
{'BPM': 0, 'COSMIC': 1, 'SATURATED': 2}
>>> image_bm.keys()
['BPM', 'COSMIC', 'SATURATED']
Now you can define a numpy.ndarray to hold the mask value for each
image pixel; the minimum_dtype()
returns the the smallest data type required to represent the list of
defined bits. The maximum number of bits that can be defined is 64.
Assuming you have an image img
:
import numpy
mask = numpy.zeros(img.shape, dtype=image_bm.minimum_dtype())
Assuming you have boolean or integer arrays that identify pixels to mask, you can turn on the mask bits as follows:
mask[cosmics_indx] = image_bm.turn_on(mask[cosmics_indx], 'COSMIC')
mask[saturated_indx] = image_bm.turn_on(mask[saturated_indx], 'SATURATED')
or make sure certain bits are off:
mask[not_a_cosmic] = image_bm.turn_off(mask[not_a_cosmic], 'COSMIC')
The form of these methods is such that the array passed to the method
are not altered. Instead the altered bits are returned, which is why
the lines above have the form m = bm.turn_on(m, flag)
.
Some other short usage examples:
To find which flags are set for a single value:
image_bm.flagged_bits(mask[0,10])To find the list of unique flags set for any pixel:
unique_flags = numpy.sort(numpy.unique(numpy.concatenate( [image_bm.flagged_bits(b) for b in numpy.unique(mask)]))).tolist()To get a boolean array that selects pixels with one or more mask bits:
cosmics_indx = image_bm.flagged(mask, flag='COSMIC') all_but_bpm_indx = image_bm.flagged(mask, flag=['COSMIC', 'SATURATED']) any_flagged = image_bm.flagged(mask)To construct masked arrays, following from the examples above:
masked_img = numpy.ma.MaskedArray(img, mask=image_bm.flagged(mask))
BitMask
objects can be defined
programmatically, as shown above for the ImageBitMask
derived class,
but they can also be defined by reading formatted files. The current
options are:
SDSS-style parameter file: This is largely used to make
BitMask
compatible with the SDSS maskbits file (see$MANGADAP_DIR/mangadap/data/sdss/sdssMaskbits.par
). For theImageBitMask
example, the par file would look like this:typedef struct { char flag[20]; # Flag name short datatype; # Data type {8, 16, 32, 64} char description[100]; # text description } masktype; typedef struct { char flag[20]; # Flag name short bit; # Bit number, 0-indexed char label[30]; # Bit label char description[100]; # text description } maskbits; masktype IMAGEMASK 16 "Mask bits for image flagging" maskbits IMAGEMASK 0 BPM "Pixel is part of a bad-pixel mask" maskbits IMAGEMASK 1 COSMIC "Pixel is contaminated by a cosmic ray" maskbits IMAGEMASK 2 SATURATED "Pixel is saturated"Assuming this is written to
imagebitmask.par
, you can instantiate theBitMask
like so:from mangadap.util.bitmask import BitMask bm = BitMask.from_par_file('imagebitmask.par', 'IMAGEMASK')Configuration (ini) file: This is how the DAP defines most of its internal bitmasks. For the
ImageBitMask
example, the ini file would look like this:[BPM] value = 0 descr = Pixel is part of a bad-pixel mask [COSMIC] value = 1 descr = Pixel is contaminated by a cosmic ray [SATURATED] value = 2 descr = Pixel is saturatedAssuming this is written to
imagebitmask.ini
, you can instantiate theBitMask
like so:from mangadap.util.bitmask import BitMask bm = BitMask.from_ini_file('imagebitmask.ini')Fits headers: There are both reading and writing methods for bitmask I/O using astropy.io.fits.Header objects. Using the
ImageBitMask
class as an example:>>> from astropy.io import fits >>> hdr = fits.Header() >>> image_bm = ImageBitMask() >>> image_bm.to_header(hdr) >>> hdr BIT0 = 'BPM ' / Pixel is part of a bad-pixel mask BIT1 = 'COSMIC ' / Pixel is contaminated by a cosmic ray BIT2 = 'SATURATED' / Pixel is saturated. >>> copy_bm = BitMask.from_header(hdr)
Copyright © 2019, SDSS-IV/MaNGA Pipeline Group
- class mangadap.util.bitmask.BitMask(keys, descr=None)[source]¶
Bases:
object
Generic class to handle and manipulate bitmasks. The input list of bit names (keys) must be unique, except that values of ‘NULL’ are ignored. The index in the input keys determines the bit value; ‘NULL’ keys are included in the count. For example:
>>> from mangadap.util.bitmask import BitMask >>> keys = [ 'key1', 'key2', 'NULL', 'NULL', 'key3' ] >>> bm = BitMask(keys) >>> bm.info() Bit: key1 = 0 Bit: key2 = 1 Bit: key3 = 4
Todo
Have the class keep the mask values internally instead of having it only operate on the mask array…
- Parameters
keys (
str
,list
) – List of keys (or single key) to use as the bit name. Each key is given a bit number ranging from 0..N-1.descr (
str
,list
, optional) – List of descriptions (or single description) provided byinfo()
for each bit. No descriptions by default.
- Raises
ValueError – Raised if more than 64 bits are provided.
TypeError – Raised if the provided keys do not have the correct type.
- nbits¶
Number of bits
- Type
int
- bits¶
A dictionary with the bit name and value
- Type
dict
- descr¶
List of bit descriptions
- Type
numpy.ndarray
- max_value¶
The maximum valid bitmask value given the number of bits.
- Type
int
- static _fill_sequence(keys, vals, descr)[source]¶
Fill bit sequence with NULL keys if bit values are not sequential.
The instantiation of
BitMask
does not include the value of the bit, it just assumes that the bits should be in sequence such that the first key has a value of 0, and the last key has a value of N-1. This is a convenience function that finds the range of the bits and then slots in NULL keywords and empty descriptions where necessary to fill in the full complement of bits. NULL keywords are ignored by theBitMask
object.- Parameters
keys (
list
,str
) – Bit namesvals (
list
,int
) – Bit valuesdescr (
list
,str
, optional) – The description of each bit. If None, no bit descriptions are defined.
- Returns
Three arrays with the filled keys, values, and descriptions.
- Return type
- Raises
ValueError – Raised if a bit value is less than 0.
- _init_objs()[source]¶
Return the objects needed to instantate another BitMask object that’s identical to self.
- consolidate(value, flag_set, consolidated_flag)[source]¶
Consolidate a set of flags into a single flag.
- flagged(value, flag=None)[source]¶
Determine if a bit is on in the provided bitmask value. The function can be used to determine if any individual bit is on or any one of many bits is on.
- Parameters
value (int, array-like) – Bitmask value. It should be less than or equal to
max_value
; however, that is not checked.flag (str, array-like, optional) – One or more bit names to check. If None, then it checks if any bit is on.
- Returns
Boolean flags that the provided flags (or any flag) is on for the provided bitmask value. Shape is the same as value.
- Return type
bool
- Raises
KeyError – Raised by the dict data type if the input flag is not one of the valid
flags
.TypeError – Raised if the provided flag does not contain one or more strings.
- flagged_bits(value)[source]¶
Return the list of flagged bit names for a single bit value.
- Parameters
value (int) – Bitmask value. It should be less than or equal to
max_value
; however, that is not checked.- Returns
List of flagged bit value keywords.
- Return type
list
- Raises
KeyError – Raised by the dict data type if the input flag is not one of the valid
flags
.TypeError – Raised if the provided flag does not contain one or more strings.
- classmethod from_header(hdr, prefix=None)[source]¶
Instantiate the BitMask using data parsed from a fits header.
Todo
This is very similar to the function in ParSet. Abstract to a general routine?
If comments are truncated by the comment line length, they’ll be different than a direct instantiation.
- Parameters
hdr (astropy.io.fits.Header) – Header object with the bits.
prefix (
str
, optional) – Prefix of the relevant header keywords, which overwrites the string defined for the class. If None, uses the default for the class.
- classmethod from_ini_file(f)[source]¶
Define the object using a ini configuration file. The sections of the configuration file define the keys, and each section is expected to have value and descr components that define the bit value and provided a description of the bit. An example ini file might look like this:
[NO_DATA] value = 0 descr = Pixel has no data [INVALID_DATA] value = 1 descr = Pixel value is invalid
See
mangadap.proc.templatelibrary.TemplateLibraryBitMask
for an example that uses this function.
- classmethod from_par_file(f, name)[source]¶
Define the object using an SDSS-style parameter file. This has been tailored to work with the sdssMaskbits.par file in IDLUTILS; however, it can work with similar files.
See
mangadap.util.drpfits.DRPFitsBitMask
for an example that uses this function.- Parameters
f (
str
) – File name to use for defining theBitMask
.name (
str
) – The designation of the bits to assign. For example, inmangadap.util.drpfits.DRPFitsBitMask
this is ‘MANGA_DRP3PIXMASK’.
- Returns
Object with bitmasks defined by the parameter file.
- Return type
- Raises
FileNotFoundError – Raised if the input file does not exist.
- keys()[source]¶
Return a list of the bit keywords.
Keywords are sorted by their bit value and ‘NULL’ keywords are ignored.
- Returns
List of bit keywords.
- Return type
list
- minimum_dtype(asuint=False)[source]¶
Return the smallest int datatype that is needed to contain all the bits in the mask. Output as an unsigned int if requested.
- Parameters
asuint (
bool
, optional) – Return an unsigned integer type. Signed types are returned by default.
Warning
uses int16 if the number of bits is less than 8 and asuint=False because of issue astropy.io.fits has writing int8 values.
- static parse_bits_from_hdr(hdr, prefix)[source]¶
Parse bit names, values, and descriptions from a fits header.
Todo
This is very similar to the function in ParSet. Abstract to a general routine?
- Parameters
hdr (astropy.io.fits.Header) – Header object with the bits.
prefix (
str
) – The prefix used for the header keywords.
- Returns
Three lists are returned providing the bit names, values, and descriptions.
- prefix = 'BIT'¶
- to_header(hdr, prefix=None, quiet=False)[source]¶
Write the bits to a fits header.
Todo
This is very similar to the function in ParSet. Abstract to a general routine?
The comment might have a limited length and be truncated.
- Parameters
hdr (astropy.io.fits.Header) – Header object for the parameters. Modified in-place.
prefix (
str
, optional) – Prefix to use for the header keywords, which overwrites the string defined for the class. If None, uses the default for the class.quiet (
bool
, optional) – Suppress print statements.
- to_rst_table(header=True, class_link=True)[source]¶
Construct a reStructuredText table describing the bitmask.
- Parameters
header (
bool
, optional) – Include a section headerclass_link (
bool
, optional) – Include an rst-style link to the class instantiation documentation.
- Returns
Returns a list of lines that can be written to an
*.rst
file.- Return type
list
- toggle(value, flag)[source]¶
Toggle a bit in the provided bitmask value.
- Parameters
value (int, array-like) – Bitmask value. It should be less than or equal to
max_value
; however, that is not checked.flag (str, array-like) – Bit name(s) to toggle.
- Returns
New bitmask value after toggling the selected bit.
- Return type
array-like
- Raises
ValueError – Raised if the provided flag is None.
- turn_off(value, flag)[source]¶
Ensure that a bit is turned off in the provided bitmask value.
- Parameters
value (int, array-like) – Bitmask value. It should be less than or equal to
max_value
; however, that is not checked.flag (str, array-like) – Bit name(s) to turn off.
- Returns
New bitmask value after turning off the selected bit.
- Return type
uint
- Raises
ValueError – Raised if the provided flag is None.
- turn_on(value, flag)[source]¶
Ensure that a bit is turned on in the provided bitmask value.
- Parameters
value (int, array-like) – Bitmask value. It should be less than or equal to
max_value
; however, that is not checked.flag (str, array-like) – Bit name(s) to turn on.
- Returns
New bitmask value after turning on the selected bit.
- Return type
uint
- Raises
ValueError – Raised if the provided flag is None.
- unpack(value, flag=None)[source]¶
Construct boolean arrays with the selected bits flagged.
- Parameters
value (numpy.ndarray) – The bitmask values to unpack.
flag (
str
,list
, optional) – The specific bits to unpack. If None, all values are unpacked.
- Returns
A tuple of boolean numpy.ndarrays flagged according to each bit.
- Return type
tuple