5 minute quickstart tutorial#

Follow this example to get started with DeepInverse in under 5 minutes.

Contents

  1. Install

  2. Physics

  3. Models

  4. Datasets

  5. What’s next

1. Install#

First, install and import the latest stable release of deepinv:

pip install deepinv

We then get the device (CPU in the case of this example).

import deepinv as dinv
import torch

device = dinv.utils.get_freer_gpu() if torch.cuda.is_available() else "cpu"

2. Physics#

In DeepInverse, x are images:

x = dinv.utils.load_example("butterfly.png", device=device)

Images are tensors of shape B, C, ... where B is batch size, C are channels and ... are spatial dimensions:

print(x.shape)
torch.Size([1, 3, 256, 256])

Imaging forward operators are called physics and simulate measurements y from x.

DeepInverse implements many different types of physics across various imaging modalities. Physics also possess noise models such as Gaussian or Poisson noise.

physics.noise_model = dinv.physics.GaussianNoise(sigma=0.1)

y = physics(x)

dinv.utils.plot({"GT": x, "Noisy inpainting measurement": y})
GT, Noisy inpainting measurement

Many physics also take physics parameters such as mask, filter, sigma etc.:

# Blur with Gaussian filter parameter
filter = dinv.physics.blur.gaussian_blur((5, 5))

physics = dinv.physics.BlurFFT(x.shape[1:], filter=filter, device=device)

# Simulate measurements
y = physics(x)

You can easily use your own params by passing these into the physics, or you can use a generator to generate random params:

# Blur kernel random generator
physics_generator = dinv.physics.generator.MotionBlurGenerator(
    psf_size=(31, 31), num_channels=3, device=device
)

# Generate a dict of random params {"filter": ...}
params = physics_generator.step()

# Update physics during forward call
y2 = physics(x, **params)

dinv.utils.plot(
    {
        "GT": x,
        "Blurred...": y,
        "... with Gaussian kernel": filter,
        "Blurred ...": y2,
        "...with motion kernel": params["filter"],
    }
)
GT, Blurred..., ... with Gaussian kernel, Blurred ..., ...with motion kernel

Physics are powerful objects and have many methods, for example a pseudo-inverse:

# You can also update params like so
physics.update(filter=filter.to(device))

x_pinv = physics.A_dagger(y)

As it is well-known in the field of inverse problems, the pseudo-inverse can give good results if the problem is noiseless, but it completely fails in the presence of noise - this is why we need reconstructors!

physics.noise_model = dinv.physics.GaussianNoise(sigma=0.1)

y = physics(x)

x_pinv_noise = physics.A_dagger(y)

dinv.utils.plot({"Pseudoinv w/o noise": x_pinv, "Pseudoinv with noise": x_pinv_noise})
Pseudoinv w/o noise, Pseudoinv with noise

Tip

Want to use DeepInverse with your own physics operator? Check out Bring your own physics for a tutorial!

3. Models#

In DeepInverse, a model is a reconstruction algorithm that reconstructs images from y and knowledge of physics.

Tip

Many models, such as Reconstruct Anything Model, are pretrained reconstructors and can be used out of the box. See Use a pretrained model for a full example.

Downloading: "https://huggingface.co/mterris/ram/resolve/main/ram.pth.tar" to /home/runner/.cache/torch/hub/checkpoints/ram.pth.tar

  0%|          | 0.00/136M [00:00<?, ?B/s]
  2%|▏         | 2.75M/136M [00:00<00:05, 23.5MB/s]
  4%|▎         | 5.00M/136M [00:00<00:10, 13.2MB/s]
  5%|▍         | 6.50M/136M [00:00<00:12, 10.7MB/s]
  6%|▌         | 8.12M/136M [00:00<00:10, 12.3MB/s]
 27%|██▋       | 36.2M/136M [00:00<00:01, 86.1MB/s]
 44%|████▍     | 60.4M/136M [00:00<00:00, 131MB/s]
 67%|██████▋   | 91.9M/136M [00:00<00:00, 187MB/s]
 88%|████████▊ | 119M/136M [00:01<00:00, 214MB/s]
100%|██████████| 136M/136M [00:01<00:00, 125MB/s]

Plot the image x, the measurement y and the reconstructed image x_hat and compute metrics:

metric = dinv.metric.PSNR()

psnr_y = metric(y, x).item()
psnr_x_hat = metric(x_hat, x).item()

dinv.utils.plot(
    {
        f"Measurement\n {psnr_y:.2f} dB": y,
        f"Reconstruction\n {psnr_x_hat:.2f} dB": x_hat,
        "GT": x,
    }
)
Measurement  14.19 dB, Reconstruction  19.22 dB, GT

Some models are only denoisers that denoise images from y and sigma, which can be used to build many model-based reconstruction algorithms.

denoiser = dinv.models.DRUNet(device=device)

x_denoised = denoiser(y, sigma=0.1)

model = dinv.optim.DPIR(sigma=0.1, denoiser=denoiser, device=device)

x_hat = model(y, physics)

dinv.utils.plot(
    {"Measurement": y, "Denoised": x_denoised, "Reconstructed": x_hat, "GT": x}
)
Measurement, Denoised, Reconstructed, GT
Downloading: "https://huggingface.co/deepinv/drunet/resolve/main/drunet_deepinv_color_finetune_22k.pth?download=true" to /home/runner/.cache/torch/hub/checkpoints/drunet_deepinv_color_finetune_22k.pth

  0%|          | 0.00/125M [00:00<?, ?B/s]
 10%|▉         | 12.4M/125M [00:00<00:00, 129MB/s]
 32%|███▏      | 39.2M/125M [00:00<00:00, 218MB/s]
 53%|█████▎    | 66.1M/125M [00:00<00:00, 247MB/s]
 76%|███████▌  | 94.4M/125M [00:00<00:00, 266MB/s]
 99%|█████████▉| 124M/125M [00:00<00:00, 281MB/s]
100%|██████████| 125M/125M [00:00<00:00, 259MB/s]

DeepInverse covers many frameworks of reconstruction algorithms including deep model architectures, iterative algorithms, sampling algorithms (e.g. diffusion models), and unfolded models.

# Reconstruct Anything Model foundation model
model = dinv.models.RAM(pretrained=True, device=device)

Tip

Want to use DeepInverse with your own network? Just inherit from the reconstructor base class deepinv.models.Reconstructor!

4. Datasets#

You can use DeepInverse with dataset, for testing or training. First, define a ground-truth dataset. We implement wrappers for many popular imaging datasets across domains including natural images, medical imaging, satellite imaging, etc.

Tip

It’s easy to use your own dataset with DeepInverse. See Bring your own dataset for a tutorial.

dataset = dinv.datasets.SimpleFastMRISliceDataset(
    "data", anatomy="brain", download=True
)
  0%|          | 0/820534 [00:00<?, ?it/s]
100%|██████████| 801k/801k [00:00<00:00, 18.6MB/s]

Datasets return either x, tuples x, y or x, y, params of images, measurements, and optional physics parameters. Given a ground-truth dataset, you can simulate a dataset with random physics:

Dataset has been saved at data/dinv_dataset0.h5

You can use this dataset to test or train a model:

import torch

dinv.test(
    model,
    torch.utils.data.DataLoader(dataset),
    physics,
    plot_images=True,
    device=device,
)
Ground truth, Measurement, No learning, Reconstruction
  0%|                                                                                                                           | 0/2 [00:00<?, ?it/s]
Test:   0%|                                                                                                                     | 0/2 [00:00<?, ?it/s]
Test:   0%|                                                                                   | 0/2 [00:04<?, ?it/s, PSNR=41.4, PSNR no learning=31.4]
Test:  50%|█████████████████████████████████████▌                                     | 1/2 [00:04<00:04,  4.43s/it, PSNR=41.4, PSNR no learning=31.4]
Test:  50%|█████████████████████████████████████▌                                     | 1/2 [00:04<00:04,  4.43s/it, PSNR=41.4, PSNR no learning=31.4]
Test:  50%|█████████████████████████████████████▌                                     | 1/2 [00:08<00:04,  4.43s/it, PSNR=40.2, PSNR no learning=30.5]
Test: 100%|███████████████████████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.43s/it, PSNR=40.2, PSNR no learning=30.5]
Test: 100%|███████████████████████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.43s/it, PSNR=40.2, PSNR no learning=30.5]
Test results:
PSNR no learning: 30.459 +- 0.940
PSNR: 40.170 +- 1.241

{'PSNR no learning': np.float64(30.458812713623047), 'PSNR no learning_std': np.float64(0.939930729178376), 'PSNR': np.float64(40.16980743408203), 'PSNR_std': np.float64(1.2413168748421881)}

Tip

Want to use DeepInverse with your own dataset? Check out Bring your own dataset for a tutorial!

🎉 Well done, you now know how to use DeepInverse!

What’s next?#

Total running time of the script: (0 minutes 31.913 seconds)

Gallery generated by Sphinx-Gallery