Fully Unsupervised Probabilistic Noise2Void

Mangal Prakash, Manan Lalit, Pavel Tomancak, Alexander Krull, Florian Jug

Posted on: 11 January 2020

Preprint posted on 11 January 2020

Advancing image denoising: Improvements to Probabilistic Noise2Void (PN2V), a method to train convolutional neural networks for image denoising.

Selected by Mariana De Niz

Categories: bioinformatics, cell biology, microbiology, pathology, physiology, systems biology

Background

Image denoising is the first step in many biomedical image analysis pipelines, and it boils down to separating an image into its two components: the signal s, and the signal-degrading noise n. For this purpose, discriminative deep learning based methods are currently best performing. Previously, methods achieved denoising by training on pairs of noisy images, and corresponding clean target images. Later, a method was proposed which is capable of using independent pairs of noisy images: Noise2Noise (N2N). This approach has the advantage that it removes the need for strenuous collection of clean data. An improvement towards making this step more efficient, came in early 2019 with a method called Noise2Void (N2V) (1), which is a self-supervised training scheme, namely, it does not require noisy image pairs, nor clean target images. It allows training directly on the body of data to be denoised. This method was applied to various types of biomedical images, including fluorescence microscopy, and cryo-EM. A couple of limitations identified in N2V were a) the assumed predictability of signal s – (i.e. if a pixel’s signal is difficult to predict from its surroundings, more errors will appear in N2V predictions); and b) that N2V could not distinguish between the signal and structured noise that violates the assumption that for any given signal the noise is pixel-wise independent (1). Furthermore, the fact that self-supervised methods such as N2V are not competitive with models trained on image pairs raised the question how they could further be improved.

This is because self-supervised training assumes that the noise is pixel-wise independent given the ground-truth signal, and that the true intensity of the pixel can be predicted from local image context, excluding blind spots (single missing pixels) (1). To address this, Laine et al (2) proposed using a Gaussian noise model and predicting Gaussian intensity distributions per pixel. With a method called Probabilistic Noise 2 Void (PN2V), Laine et al (2) and Krug et al (3) proposed a way to leverage information to the network’s blind spots. PN2V puts a probabilistic model for each pixel in place, from which it is possible to infer better denoising results after choose a statistical estimator,i.e. MMSE. The required PN2V noise models are generated from a sequence of noisy calibration images, and characterize the distribution of noisy pixels around their respective ground truth signal value. The current preprint by Prakash et al. goes yet another step further, making PN2V fully unsupervised, hence not requiring to image calibration data any more (4). (Figure 1).

Figure 1. GMM bootstrapping method does not require paired training or calibration data.

Key findings and developments

General summary

The current work is an improvement over Noise2Void, Noise2Self, and Probabilistic Noise2Void. The previously published Probabilistic Noise2Void (PN2V) required additional noise models for which calibration data is required. The improvement to PN2V presented here replaces histogram based noise models by parametric noise models (using Gaussian Mixture Models (GMM)), and shows how suitable noise models can be created even in the absence of calibration data (i.e. Bootstrapped PN2V), hence rendering PN2V fully unsupervised.

The GMM-based variation of PN2V noise models can lead to higher reconstruction quality, even with imperfect calibration data. The bootstrapping scheme allows PN2V to be trained fully unsupervised, making use of only the data to be denoised. The denoising quality of bootstrapped PN2V was close to fully supervised CARE and outperformed N2V notably.

The authors applied the model to three image sets including a Convalaria dataset, a mouse skull nuclei dataset and a mouse actin dataset, all consisting of calibration and denoised sets. Consistent with open science, the authors make the calibration data and noisy image data publicly available together with the code.

The improvement to PN2V will help to make high quality deep learning-based denoising an easily applicable tool that does not require the acquisition of paired training data or calibration data. This is particularly relevant to experiments where photosensitivity or fast dynamics hinder the acquisition of image pairs.

What I like about this paper

I liked most about this paper the usefulness of the tool presented, and the continuity they give to the work they had previously presented. This is a tool useful to many labs across the biomedical science community. I like the fact that their development is a good example of open science, and of considering the needs of the scientific community- as the authors discuss the process since this tools’ predecessors. I also like that the full datasets are made available, and that the full basis of the improvements to PN2V are explained in detail.

Open questions

1. In your preprint you touch on the question why self-supervised methods are not competitive with models trained on image pairs. Can you expand further on your explanation?

2. You tested the improvements you made on images acquired from spinning disc and point scanning confocal microscopy with great success. Do you expect equal success on images obtained by MRI or EM (as you showed in your previous paper, where you first introduced PN2V)?

3. What are general limitations you would expect using different types of samples (eg. live vs. fixed; strong markers vs. weak), or different imaging methods (eg. MRI, EM, cryo-EM, SR, etc)?

4. Are there specific cases where your improved PN2V model is still unsuccessful and image pairs are still necessary?

5. Could you expand further on the way Gaussian Mixture Models work, for non-specialists?

References

Krull A, Vicar T, Jug F, Probabilistic Noise2Void: Unsupervised content-aware denoising (2019), arXiv:1906.00651v2
Laine S, Karras T, Lehtinen J, Aila T, High-quality self-supervised deep image denoising (2019), arXiv:1901.10277
Krull A, Buchholz TO, Jug F, Noise2Void – Learning denoising from single noisy images (2018), arXiv:1811.10980v2
Prakash M, Lalit M, Tomancak P, Krull A, Jug F, Fully unsupervised probabilistic Noise2Void (2019), arXiv:1911.12291v1
Weigert M, Schmidt U, Boothe T, Müller A, Dibrov A, Jain A, Wilhelm B, Schmidt D, Broaddus C, Culley S, Rocha-Martins M, Segovia-Miranda F, Norden C, Henriques R, Zerial M, Solimena M, Rink J, Tomancak P, Royer L, Jug F, Myers EW, Content-aware image restoration: pushing the limits of fluorescence microscopy, (2018), Nature Methods, 15(12):1090-1097, doi: 10.1038/s41592-018-0216-7.

Acknowledgement

I thank Mate Palfy for his feedback on this preLight highlight, and to Florian Jug, Alexander Krull, Mangal Prakash, and Manan Lalit for their feedback and kindly answering questions related to their preprint.

doi: https://doi.org/10.1242/prelights.16203

Read preprint

(No Ratings Yet)

Author's response

Mangal Prakash, Manan Lalit, Alexander Krull, Florian Jug shared

1. In your preprint you touch on the question why self-supervised methods are not competitive with models trained on image pairs. Can you expand further on your explanation?

This is an interesting question and it is important to answer with respect to the problem one intends to address. N2V, PN2V, and Bootstapped PN2V are methods to remove pixel-noises from image data. Supervised methods, such as the original CARE (5) or methods based on Noise2Noise training, can address various other image restoration problems as well. These problems include deconvolution, upsampling, superimposed text removal, or joint denoising and image projections. For all such tasks PN2V is not applicable and therefore, by definition, not competitive.
Still, if the goal is to remove pixel-noises from your images, i.e. photon shot-noise or readout-noise, we show on multiple examples that PN2V (as well as Bootstrapped PN2V) are now absolutely competitive with the best supervised methods that require ground-truth data during training.

Our methods do not care about the nature of the physical image formation process and will always learn to remove pixel-noises by extracting a content-aware prior from the signal surrounding any given pixel. Hence, N2V and PN2V will work out-of-the-box on, working on lots of data modalities, including TEM projections or SEM images. The same is not true for MRI volumina because their noise is of a very different kind. This is also true for EM tomograms, where we need Noise2Noise training in order to obtain useful image restoration models (see e.g. our cryoCARE method for more details).

3. What are general limitations you would expect using different types of samples (eg. live vs. fixed; strong markers vs. weak), or different imaging methods (eg. MRI, EM, cryo-EM, SR, etc)?

As we described above, N2V, PN2V, and Boostrapped PN2V will do their job (removing pixel-noises) independent of the type of hardware that acquired the images one desires to denoise.

4. Are there specific cases where your improved PN2V model is still unsuccessful and image pairs are still necessary?

Most of the original CARE applications (5) are, for example, not possible to with N2V or PN2V. The same holds true for other applications that go beyond the removal of pixel-noises. The future will tell in what ways one can continue to improve self-supervised image restoration methods. We are certainly looking forward to seeing new ideas being proposed and are clearly planning to continue working in this exciting research direction ourselves.

5. Could you expand further on the way Gaussian Mixture Models work, for non-specialists?

Gaussian Mixture Models are a powerful tool for approximating arbitrary probability distributions. A GMM is simply the combination of multiple normal distributions, each having their own mean and standard deviation, as well as a weight that determines its influence on the mixture. It is important to note that a GMM can be used to approximate quite complex functions, in our case the measured distribution of noisy pixels intensities.
An excellent and comprehensive introduction to GMMs can be found in https://towardsdatascience.com/gaussian-mixture-models-explained-6986aaf5a95.