mangadap.util.bitmask module

Base class for handling bit masks.

Class usage examples

Bitmasks allow you to define a set of bit values signified by strings, and then toggle and interpret bits held by a numpy.ndarray. For example, say you’re processing an image and you want to set up a set of bits that indicate that the pixel is part of a bad-pixel mask, has a cosmic ray, or is saturated. You can define the following:

from collections import OrderedDict
from mangadap.util.bitmask import BitMask

bits = OrderedDict([('BPM', 'Pixel is part of a bad-pixel mask'),
                    ('COSMIC', 'Pixel is contaminated by a cosmic ray'),
                    ('SATURATED', 'Pixel is saturated.')])
image_bm = BitMask(list(bits.keys()), descr=list(bits.values()))

Or, better yet, define a derived class:

from collections import OrderedDict
from mangadap.util.bitmask import BitMask

class ImageBitMask(BitMask):
    def __init__(self):
        bits = OrderedDict([('BPM', 'Pixel is part of a bad-pixel mask'),
                            ('COSMIC', 'Pixel is contaminated by a cosmic ray'),
                            ('SATURATED', 'Pixel is saturated.')])
        super(ImageBitMask, self).__init__(list(bits.keys()), descr=list(bits.values()))

image_bm = ImageBitMask()

In either case, you can see the list of bits and their bit numbers by running:

>>> image_bm.info()
         Bit: BPM = 0
 Description: Pixel is part of a bad-pixel mask

         Bit: COSMIC = 1
 Description: Pixel is contaminated by a cosmic ray

         Bit: SATURATED = 2
 Description: Pixel is saturated.
>>> image_bm.bits
{'BPM': 0, 'COSMIC': 1, 'SATURATED': 2}
>>> image_bm.keys()
['BPM', 'COSMIC', 'SATURATED']

Now you can define a numpy.ndarray to hold the mask value for each image pixel; the minimum_dtype() returns the the smallest data type required to represent the list of defined bits. The maximum number of bits that can be defined is 64. Assuming you have an image img:

import numpy
mask = numpy.zeros(img.shape, dtype=image_bm.minimum_dtype())

Assuming you have boolean or integer arrays that identify pixels to mask, you can turn on the mask bits as follows:

mask[cosmics_indx] = image_bm.turn_on(mask[cosmics_indx], 'COSMIC')
mask[saturated_indx] = image_bm.turn_on(mask[saturated_indx], 'SATURATED')

or make sure certain bits are off:

mask[not_a_cosmic] = image_bm.turn_off(mask[not_a_cosmic], 'COSMIC')

The form of these methods is such that the array passed to the method are not altered. Instead the altered bits are returned, which is why the lines above have the form m = bm.turn_on(m, flag).

Some other short usage examples:

To find which flags are set for a single value:
image_bm.flagged_bits(mask[0,10])
To find the list of unique flags set for any pixel:
unique_flags = numpy.sort(numpy.unique(numpy.concatenate(
                    [image_bm.flagged_bits(b) for b in numpy.unique(mask)]))).tolist()
To get a boolean array that selects pixels with one or more mask bits:
cosmics_indx = image_bm.flagged(mask, flag='COSMIC')
all_but_bpm_indx = image_bm.flagged(mask, flag=['COSMIC', 'SATURATED'])
any_flagged = image_bm.flagged(mask)
To construct masked arrays, following from the examples above:
masked_img = numpy.ma.MaskedArray(img, mask=image_bm.flagged(mask))

BitMask objects can be defined programmatically, as shown above for the ImageBitMask derived class, but they can also be defined by reading formatted files. The current options are:

SDSS-style parameter file: This is largely used to make BitMask compatible with the SDSS maskbits file (see $MANGADAP_DIR/mangadap/data/sdss/sdssMaskbits.par). For the ImageBitMask example, the par file would look like this:

typedef struct {
    char flag[20]; # Flag name
    short datatype; # Data type {8, 16, 32, 64}
    char description[100]; # text description
} masktype;

typedef struct {
    char flag[20]; # Flag name
    short bit; # Bit number, 0-indexed
    char label[30]; # Bit label
    char description[100]; # text description
} maskbits;

masktype IMAGEMASK 16           "Mask bits for image flagging"
maskbits IMAGEMASK  0 BPM       "Pixel is part of a bad-pixel mask"
maskbits IMAGEMASK  1 COSMIC    "Pixel is contaminated by a cosmic ray"
maskbits IMAGEMASK  2 SATURATED "Pixel is saturated"

Assuming this is written to imagebitmask.par, you can instantiate the BitMask like so:

from mangadap.util.bitmask import BitMask
bm = BitMask.from_par_file('imagebitmask.par', 'IMAGEMASK')

Configuration (ini) file: This is how the DAP defines most of its internal bitmasks. For the ImageBitMask example, the ini file would look like this:

[BPM]
value = 0
descr = Pixel is part of a bad-pixel mask

[COSMIC]
value = 1
descr = Pixel is contaminated by a cosmic ray

[SATURATED]
value = 2
descr = Pixel is saturated

Assuming this is written to imagebitmask.ini, you can instantiate the BitMask like so:

from mangadap.util.bitmask import BitMask
bm = BitMask.from_ini_file('imagebitmask.ini')

Fits headers: There are both reading and writing methods for bitmask I/O using astropy.io.fits.Header objects. Using the ImageBitMask class as an example:

>>> from astropy.io import fits
>>> hdr = fits.Header()
>>> image_bm = ImageBitMask()
>>> image_bm.to_header(hdr)
>>> hdr
BIT0    = 'BPM     '           / Pixel is part of a bad-pixel mask
BIT1    = 'COSMIC  '           / Pixel is contaminated by a cosmic ray
BIT2    = 'SATURATED'          / Pixel is saturated.
>>> copy_bm = BitMask.from_header(hdr)

class mangadap.util.bitmask.BitMask(keys, descr=None)[source]

Bases: object

Generic class to handle and manipulate bitmasks. The input list of bit names (keys) must be unique, except that values of ‘NULL’ are ignored. The index in the input keys determines the bit value; ‘NULL’ keys are included in the count. For example:

>>> from mangadap.util.bitmask import BitMask
>>> keys = [ 'key1', 'key2', 'NULL', 'NULL', 'key3' ]
>>> bm = BitMask(keys)
>>> bm.info()
         Bit: key1 = 0

         Bit: key2 = 1

         Bit: key3 = 4

Todo

Have the class keep the mask values internally instead of having it only operate on the mask array…

Parameters:

keys (str, list) – List of keys (or single key) to use as the bit name. Each key is given a bit number ranging from 0..N-1.
descr (str, list, optional) – List of descriptions (or single description) provided by info() for each bit. No descriptions by default.

Raises:

ValueError – Raised if more than 64 bits are provided.
TypeError – Raised if the provided keys do not have the correct type.

nbits

Number of bits

Type:: int

bits

A dictionary with the bit name and value

Type:: dict

descr

List of bit descriptions

Type:: numpy.ndarray

max_value

The maximum valid bitmask value given the number of bits.

Type:: int

static _fill_sequence(keys, vals, descr)[source]

Fill bit sequence with NULL keys if bit values are not sequential.

The instantiation of BitMask does not include the value of the bit, it just assumes that the bits should be in sequence such that the first key has a value of 0, and the last key has a value of N-1. This is a convenience function that finds the range of the bits and then slots in NULL keywords and empty descriptions where necessary to fill in the full complement of bits. NULL keywords are ignored by the BitMask object.

Parameters:

keys (list, str) – Bit names
vals (list, int) – Bit values
descr (list, str, optional) – The description of each bit. If None, no bit descriptions are defined.

Returns:

Three arrays with the filled keys, values, and descriptions.

Return type:

numpy.ndarray

Raises:

ValueError – Raised if a bit value is less than 0.

_init_objs()[source]: Return the objects needed to instantate another BitMask object that’s identical to self.

_prep_flags(flag)[source]: Prep the flags for use.

consolidate(value, flag_set, consolidated_flag)[source]: Consolidate a set of flags into a single flag.

flagged(value, flag=None)[source]

Determine if a bit is on in the provided bitmask value. The function can be used to determine if any individual bit is on or any one of many bits is on.

Parameters:

value (int, array-like) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.
flag (str, array-like, optional) – One or more bit names to check. If None, then it checks if any bit is on.

Returns:

Boolean flags that the provided flags (or any flag) is on for the provided bitmask value. Shape is the same as value.

Return type:

bool

Raises:

KeyError – Raised by the dict data type if the input flag is not one of the valid flags.
TypeError – Raised if the provided flag does not contain one or more strings.

flagged_bits(value)[source]

Return the list of flagged bit names for a single bit value.

Parameters:

value (int) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.

Returns:

List of flagged bit value keywords.

Return type:

list

Raises:

KeyError – Raised by the dict data type if the input flag is not one of the valid flags.
TypeError – Raised if the provided flag does not contain one or more strings.

classmethod from_header(hdr, prefix=None)[source]

Instantiate the BitMask using data parsed from a fits header.

Todo

This is very similar to the function in ParSet. Abstract to a general routine?
If comments are truncated by the comment line length, they’ll be different than a direct instantiation.

Parameters:

hdr (astropy.io.fits.Header) – Header object with the bits.
prefix (str, optional) – Prefix of the relevant header keywords, which overwrites the string defined for the class. If None, uses the default for the class.

classmethod from_ini_file(f)[source]

Define the object using a ini configuration file. The sections of the configuration file define the keys, and each section is expected to have value and descr components that define the bit value and provided a description of the bit. An example ini file might look like this:

[NO_DATA]
 value = 0
 descr = Pixel has no data

[INVALID_DATA]
 value = 1
 descr = Pixel value is invalid

See mangadap.proc.templatelibrary.TemplateLibraryBitMask for an example that uses this function.

Parameters:: f (str) – File name to use for defining the BitMask.
Returns:: Object with bitmasks defined by the ini file.
Return type:: BitMask
Raises:: FileNotFoundError – Raised if the input file does not exist.

classmethod from_par_file(f, name)[source]

Define the object using an SDSS-style parameter file. This has been tailored to work with the sdssMaskbits.par file in IDLUTILS; however, it can work with similar files.

See mangadap.util.drpbitmask.DRPFitsBitMask for an example that uses this function.

Parameters:

f (str) – File name to use for defining the BitMask.
name (str) – The designation of the bits to assign. For example, in mangadap.util.drpbitmask.DRPFitsBitMask this is ‘MANGA_DRP3PIXMASK’.

Returns:

Object with bitmasks defined by the parameter file.

Return type:

BitMask

Raises:

FileNotFoundError – Raised if the input file does not exist.

info()[source]: Print the list of bits and, if available, their descriptions.

keys()[source]

Return a list of the bit keywords.

Keywords are sorted by their bit value and ‘NULL’ keywords are ignored.

Returns:: List of bit keywords.
Return type:: list

minimum_dtype(asuint=False)[source]

Return the smallest int datatype that is needed to contain all the bits in the mask. Output as an unsigned int if requested.

Parameters:: asuint (bool, optional) – Return an unsigned integer type. Signed types are returned by default.

Warning

uses int16 if the number of bits is less than 8 and asuint=False because of issue astropy.io.fits has writing int8 values.

static parse_bits_from_hdr(hdr, prefix)[source]

Parse bit names, values, and descriptions from a fits header.

Todo

This is very similar to the function in ParSet. Abstract to a general routine?

Parameters:

hdr (astropy.io.fits.Header) – Header object with the bits.
prefix (str) – The prefix used for the header keywords.

Returns:

Three lists are returned providing the bit names, values, and descriptions.

prefix = 'BIT'

to_header(hdr, prefix=None, quiet=False)[source]

Write the bits to a fits header.

Todo

This is very similar to the function in ParSet. Abstract to a general routine?
The comment might have a limited length and be truncated.

Parameters:

hdr (astropy.io.fits.Header) – Header object for the parameters. Modified in-place.
prefix (str, optional) – Prefix to use for the header keywords, which overwrites the string defined for the class. If None, uses the default for the class.
quiet (bool, optional) – Suppress print statements.

to_rst_table(header=True, class_link=True)[source]

Construct a reStructuredText table describing the bitmask.

Parameters:

header (bool, optional) – Include a section header
class_link (bool, optional) – Include an rst-style link to the class instantiation documentation.

Returns:

Returns a list of lines that can be written to an *.rst file.

Return type:

list

toggle(value, flag)[source]

Toggle a bit in the provided bitmask value.

Parameters:

value (int, array-like) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.
flag (str, array-like) – Bit name(s) to toggle.

Returns:

New bitmask value after toggling the selected bit.

Return type:

array-like

Raises:

ValueError – Raised if the provided flag is None.

turn_off(value, flag)[source]

Ensure that a bit is turned off in the provided bitmask value.

Parameters:

value (int, array-like) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.
flag (str, array-like) – Bit name(s) to turn off.

Returns:

New bitmask value after turning off the selected bit.

Return type:

uint

Raises:

ValueError – Raised if the provided flag is None.

turn_on(value, flag)[source]

Ensure that a bit is turned on in the provided bitmask value.

Parameters:

value (int, array-like) – Bitmask value. It should be less than or equal to max_value; however, that is not checked.
flag (str, array-like) – Bit name(s) to turn on.

Returns:

New bitmask value after turning on the selected bit.

Return type:

uint

Raises:

ValueError – Raised if the provided flag is None.

unpack(value, flag=None)[source]

Construct boolean arrays with the selected bits flagged.

Parameters:

value (numpy.ndarray) – The bitmask values to unpack.
flag (str, list, optional) – The specific bits to unpack. If None, all values are unpacked.

Returns:

A tuple of boolean numpy.ndarrays flagged according to each bit.

Return type:

tuple