Training Losses#

This package contains popular training losses for supervised and self-supervised learning, which are especially designed for inverse problems.

Introduction#

All losses inherit from the base class deepinv.loss.Loss, which is a torch.nn.Module.

>>> import torch
>>> import deepinv as dinv
>>> loss = dinv.loss.SureGaussianLoss(.1)
>>> physics = dinv.physics.Denoising()
>>> x = torch.ones(1, 3, 16, 16)
>>> y = physics(x)
>>> model = dinv.models.DnCNN()
>>> x_net = model(y)
>>> l = loss(x_net=x_net, y=y, physics=physics, model=model) # self-supervised loss, doesn't require ground truth x

Supervised Learning#

Use a dataset of pairs of signals and measurements (and possibly information about the forward operator), i.e., they can be written as \(\mathcal{L}(x,\inverse{y})\). The main loss function is deepinv.loss.SupLoss which can use any distortion metric.

Self-Supervised Learning#

Use a dataset of measurement data alone (and possibly information about the forward operator), i.e., they can be written as \(\mathcal{L}(y,\inverse{y})\) and take into account information about the forward measurement process.

Self-supervised losses can be roughly classified according to whether they are designed to take care of the noise in the measurements, or take care of the ill-posedness of the forward operator (e.g., incomplete operators with less measurements than pixels in the image)

Table 17 Denoising Losses#
Loss	Assumptions on Noise	Compatible with general forward operators
`deepinv.loss.MCLoss`	Small or no noise.	Yes
`deepinv.loss.Neighbor2Neighbor`	Independent noise across pixels.	No
`deepinv.loss.SplittingLoss`	Independent noise across measurements.	Yes
`deepinv.loss.SureGaussianLoss`	Gaussian noise	Yes
`deepinv.loss.SurePoissonLoss`	Poisson noise	Yes
`deepinv.loss.SurePGLoss`	Poisson-Gaussian noise	Yes
`deepinv.loss.R2RLoss`	Gaussian noise	Yes
`deepinv.loss.ScoreLoss`	Poisson, Gaussian or Gamma noise	No

In order to learn from incomplete data, you can either:

Use multiple operators (e.g., different masking patterns or performing further measurement splitting)
Use a single operator and leverage invariance to transformations (e.g., rotations, translations) using Equivariant Imaging.

Table 18 Other self-supervised losses#
Loss	Assumptions
`deepinv.loss.EILoss`	Assumes invariance of the signal distribution to transformations. (i.e. Equivariant Imaging)
`deepinv.loss.MOILoss`	Assumes measurements observed through multiple operators. (i.e. Multi-Operator Imaging)
`deepinv.loss.MOEILoss`	Assumes measurements observed through multiple operators and invariance of the signal distribution
`deepinv.loss.SplittingLoss`	Assumes masks observe whole distribution. (i.e. measurement splitting or data undersampling)
`deepinv.loss.Phase2PhaseLoss`	Splitting loss but across time dimension
`deepinv.loss.Artifact2ArtifactLoss`	Splitting loss but across time dimension
`deepinv.loss.TVLoss`	Assumes images have piecewise smooth regions; based on Total Variation regularization

Tip

Splitting losses such as SplittingLoss, Phase2PhaseLoss, and Artifact2ArtifactLoss can also be used to train the network from incomplete measurements of multiple forward operators.

Network Regularization#

These losses can be used to regularize the learned function, e.g., controlling its Lipschitz constant.

Table 19 Network Regularization Losses Overview#
Loss	Description
`deepinv.loss.JacobianSpectralNorm`	Controls spectral norm of the Jacobian matrix
`deepinv.loss.FNEJacobianSpectralNorm`	Promotes a firmly non-expansive network.

Adversarial Learning#

Adversarial losses train a generator network by jointly training with an additional discriminator network in a minimax game. These can be adapted to various flavours of GAN, e.g. WGAN, LSGAN. We implement various popular (supervised and unsupervised) adversarial training frameworks below. See Adversarial Networks for more details, and see Imaging inverse problems with adversarial networks for examples. The base class for generators is deepinv.loss.adversarial.GeneratorLoss and for discriminators is deepinv.loss.adversarial.DiscriminatorLoss.

Table 20 Adversarial Losses Overview#
Generator Loss	Discriminator Loss	Description
`SupAdversarialGeneratorLoss`	`SupAdversarialDiscriminatorLoss`	Supervised adversarial loss
`UnsupAdversarialGeneratorLoss`	`UnsupAdversarialDiscriminatorLoss`	Unsupervised adversarial loss
`UAIRGeneratorLoss`		Unsupervised reconstruction & adversarial loss.

Loss schedulers#

Loss schedulers can be used to control which losses are used when during more advanced training. The base class is deepinv.loss.BaseLossScheduler.

Table 21 Schedulers Overview#
Loss	Description
`deepinv.loss.RandomLossScheduler`	Schedule losses at random.
`deepinv.loss.InterleavedLossScheduler`	Schedule losses sequentially one-by-one.
`deepinv.loss.StepLossScheduler`	Activate losses at specified epoch.
`deepinv.loss.InterleavedEpochLossScheduler`	Schedule losses sequentially epoch-by-epoch.