CompressiveSpectralImaging

class deepinv.physics.CompressiveSpectralImaging(img_size: Tuple[int, int, int], mask: Tensor | float | None = None, mode: str = 'ss', shear_dir: str = 'h', device: device = 'cpu', rng: Generator | None = None, **kwargs)

Bases: LinearPhysics

Compressive Hyperspectral Imaging operator.

Coded-aperture snapshot spectral imaging (CASSI) operator, which is a popular approach for hyperspectral imaging.

The CASSI operator performs a combination of masking (“coded aperture”), shearing, and flattening in the channel dim. We provide two specific popular CASSI models: single-disperser (i.e. only spatial encoding) and spatial-spectral encoding:

\[\begin{split}y = \begin{cases} \Sigma_{c=1}^{C} S^{-1} MSx & \text{if mode='spatial-spectral'} \\ \Sigma_{c=1}^{C} SMx & \text{if mode='single-disperser'} \end{cases}\end{split}\]

where \(M\) is a binary mask (the “coded aperture”), \(S\) is a pixel shear in the 2D channel-height of channel-width plane and \(C\) is number of channels. Note that the output size of the single-disperser mode has the H or W dim extended by C-1 pixels.

For more details see e.g. the paper High-Quality Hyperspectral Reconstruction Using a Spectral Prior.

The implementation is a type of linear physics as it is not completely decomposable due to edge effects and different scaling.

---

Examples:

>>> from deepinv.physics import CompressiveSpectralImaging
>>> physics = CompressiveSpectralImaging(img_size=(7, 32, 32))
>>> x = torch.rand(1, 7, 32, 32) # 7-band image
>>> y = physics(x)
>>> y.shape
torch.Size([1, 1, 32, 32])

Parameters:

• img_size (tuple) – image size, must be of form (C,H,W) where C is number of bands.

• mask (Tensor, float) – coded-aperture mask. If None, generate random mask using deepinv.physics.generator.BernoulliSplittingMaskGenerator with masking ratio of 0.5, if mask is float, sets mask ratio to this. If Tensor, set mask to this, must be of shape (B,C,H,W).

• mode (str) – ‘sd’ for SD-CASSI i.e. single disperser (only spatial encoding) or ‘ss’ for SS-CASSI i.e. spatial-spectral encoding. Defaults to ‘ss’. See above for details.

• shear_dir (str) – “h” for shear in H-C plane or “w” for shear in W-C plane where C is channel dim, defaults to “h”

• device (torch.device) – torch device, only used if mask is None or float

• rng (torch.Generator) – torch random generator, only used if mask is None or float

A(x: Tensor, mask: Tensor | None = None, **kwargs) → Tensor

Applies the CASSI forward operator.

If a mask is provided, it updates the class attribute mask on the fly.

Parameters:

• x (Tensor) – input image

• mask (Tensor) – CASSI mask

Returns:

(torch.Tensor) output measurements

A_adjoint(y: Tensor, mask: Tensor | None = None, **kwargs) → Tensor

Applies the CASSI adjoint operator.

If a mask is provided, it updates the class attribute mask on the fly.

Parameters:

• x (Tensor) – input measurements

• mask (Tensor) – CASSI mask

Returns:

(torch.Tensor) output image

crop(x: Tensor) → Tensor

Crop image on bottom or on right.

Parameters:

x (Tensor) – input padded image

flatten(x: Tensor) → Tensor

Average over channel dimension

Parameters:

x (Tensor) – input image of shape B,C,H,W

pad(x: Tensor) → Tensor

Pad image on bottom or on right.

Parameters:

x (Tensor) – input image

shear(x: Tensor, un=False) → Tensor

Efficient pixel shear in channel-spatial plane

Parameters:

• x (Tensor) – input image of shape (B,C,H,W)

• un (bool) – if True, unshear in opposite direction.

unflatten(y: Tensor) → Tensor

Repeat over channel dimension

Parameters:

y (Tensor) – input image of shape B,C,H,W

Examples using CompressiveSpectralImaging:

Remote sensing with satellite images

Remote sensing with satellite images