Inpainting

class deepinv.physics.Inpainting(tensor_size, mask=None, pixelwise=True, device='cpu', rng: Generator | None = None, **kwargs)[source]

Bases: DecomposablePhysics

Inpainting forward operator, keeps a subset of entries.

The operator is described by the diagonal matrix

\[A = \text{diag}(m) \in \mathbb{R}^{n\times n}\]

where \(m\) is a binary mask with n entries.

This operator is linear and has a trivial SVD decomposition, which allows for fast computation of the pseudo-inverse and proximal operator.

An existing operator can be loaded from a saved .pth file via self.load_state_dict(save_path), in a similar fashion to torch.nn.Module.

Masks can also be created on-the-fly using mask generators such as deepinv.physics.generator.BernoulliSplittingMaskGenerator, see example below.

Parameters:
  • mask (torch.Tensor, float) – If the input is a float, the entries of the mask will be sampled from a bernoulli distribution with probability equal to mask. If the input is a torch.tensor matching tensor_size, the mask will be set to this tensor. If mask is torch.Tensor, it must be shape that is broadcastable to input shape and will be broadcast during forward call. If None, it must be set during forward pass or using update_parameters method.

  • tensor_size (tuple) – size of the input images without batch dimension e.g. of shape (C, H, W) or (C, M) or (M,)

  • device (torch.device) – gpu or cpu

  • pixelwise (bool) – Apply the mask in a pixelwise fashion, i.e., zero all channels in a given pixel simultaneously. If existing mask passed (i.e. mask is Tensor), this has no effect.

  • rng (torch.Generator) – a pseudorandom random number generator for the mask generation. Default to None.


Examples:

Inpainting operator using defined mask, removing the second column of a 3x3 image:

>>> from deepinv.physics import Inpainting
>>> seed = torch.manual_seed(0) # Random seed for reproducibility
>>> x = torch.randn(1, 1, 3, 3) # Define random 3x3 image
>>> mask = torch.zeros(1, 3, 3) # Define empty mask
>>> mask[:, 2, :] = 1 # Keeping last line only
>>> physics = Inpainting(mask=mask, tensor_size=x.shape[1:])
>>> physics(x)
tensor([[[[ 0.0000, -0.0000, -0.0000],
          [ 0.0000, -0.0000, -0.0000],
          [ 0.4033,  0.8380, -0.7193]]]])

Inpainting operator using random mask, keeping 70% of the entries of a 3x3 image:

>>> from deepinv.physics import Inpainting
>>> seed = torch.manual_seed(0) # Random seed for reproducibility
>>> x = torch.randn(1, 1, 3, 3) # Define random 3x3 image
>>> physics = Inpainting(mask=0.7, tensor_size=x.shape[1:])
>>> physics(x)
tensor([[[[ 1.5410, -0.0000, -2.1788],
          [ 0.5684, -0.0000, -1.3986],
          [ 0.4033,  0.0000, -0.0000]]]])

Generate random masks on-the-fly using mask generators:

>>> from deepinv.physics import Inpainting
>>> from deepinv.physics.generator import BernoulliSplittingMaskGenerator
>>> x = torch.randn(1, 1, 3, 3) # Define random 3x3 image
>>> physics = Inpainting(tensor_size=x.shape[1:])
>>> gen = BernoulliSplittingMaskGenerator(x.shape[1:], split_ratio=0.7)
>>> params = gen.step(batch_size=1, seed = 0) # Generate random mask
>>> physics(x, **params) # Set mask on-the-fly
tensor([[[[-0.4033, -0.0000,  0.1820],
          [-0.8567,  1.1006, -1.0712],
          [ 0.1227, -0.0000,  0.3731]]]])
>>> physics.update_parameters(**params) # Alternatively update mask before forward call
>>> physics(x)
tensor([[[[-0.4033, -0.0000,  0.1820],
          [-0.8567,  1.1006, -1.0712],
          [ 0.1227, -0.0000,  0.3731]]]])
__mul__(other)[source]

Concatenates two forward operators \(A = A_1\circ A_2\) via the mul operation

If the second operator is an Inpainting or MRI operator, the masks are multiplied elementwise, otherwise the default implementation of LinearPhysics is used (see deepinv.physics.LinearPhysics.__mul__()).

Parameters:

other (deepinv.physics.Physics) – Physics operator \(A_2\)

Returns:

(deepinv.physics.Physics) concantenated operator

noise(x, **kwargs)[source]

Incorporates noise into the measurements \(\tilde{y} = N(y)\)

Parameters:

x (torch.Tensor) – clean measurements

Return torch.Tensor:

noisy measurements

Examples using Inpainting:

Reconstructing an image using the deep image prior.

Reconstructing an image using the deep image prior.

Training a reconstruction network.

Training a reconstruction network.

A tour of forward sensing operators

A tour of forward sensing operators

Saving and loading models

Saving and loading models

Image inpainting with wavelet prior

Image inpainting with wavelet prior

Expected Patch Log Likelihood (EPLL) for Denoising and Inpainting

Expected Patch Log Likelihood (EPLL) for Denoising and Inpainting

Uncertainty quantification with PnP-ULA.

Uncertainty quantification with PnP-ULA.

Image reconstruction with a diffusion model

Image reconstruction with a diffusion model

Implementing DPS

Implementing DPS

Image transformations for Equivariant Imaging

Image transformations for Equivariant Imaging

Self-supervised learning from incomplete measurements of multiple operators.

Self-supervised learning from incomplete measurements of multiple operators.

Unfolded Chambolle-Pock for constrained image inpainting

Unfolded Chambolle-Pock for constrained image inpainting