Applications, Promises, and Pitfalls of Deep Learning for Fluorescence Image Reconstruction

Chinmay Belthangady , Loic A. Royer

Preprint posted on December 11, 2018

Belthangady & Royer present a comprehensive and accessible perspective on the current and future impacts of Artificial Intelligence on microscopy. Their vision for the field is inspiring and sets the ground for an AI revolution.

Selected by Romain F. Laine

Background of the preprint

The development of artificial intelligence (AI) is driving fundamental changes in the way we travel (self-driving cars), we work (automation) and we interact with the world. It is also being exploited to improve diagnosis and detection of cancer. But there is no reason to stop here, fundamental research can also benefit from these advances and this has triggered the start of a revolution in the field of microscopy. The past couple of years have seen a growing literature on application of artificial intelligence to all kind of imaging techniques for biomedical research. Here, Chinmay Belthangady & Loic Royer from the Chan Zuckerberg Biohub, San Francisco, USA review the recent and promising developments in the field. This review aims at clearly presenting the wide range of applications of AI and its pitfalls and gives perspective to Loic Royer’s seminal work on fluorescence microscopy image restoration using AI [1].

Key points of the preprint

The paper first introduces a number of fundamental definitions about AI in the context of fluorescence microscopy. AI can generate an accurate mathematical transform that is able to take a corrupted microscopy image (e.g. with a lot of noise, or a low resolution) and correct for this image corruption. The way it does this is by learning how the data corruption occurs through looking at a lot of examples of data where the same type of corruption is present (the so-called ‘training dataset’). The authors describe how the typical neural networks used to carry this out can be thought of, e.g. a sequence of mathematical functions that progressively extract the important features in the image. The notion that AI acts as a black box is commonly founded on the fact that these features can be very abstract and therefore cannot be easily understood by the researchers.

The recent applications of AI to microscopy have showed that it can significantly improve noise or out-of-focus light removal and even provide super-resolution in unprecedented ways. The authors highlight that one major advantage of this is to be able to significantly reduce illumination on the sample because the images can be corrected post-acquisition, therefore enabling non-invasive long-term live sample imaging.

The authors then present a number of exciting directions where AI could be useful and have a major impact, e.g. massive multi-color imaging, image registration and alignment, intelligent and AI-guided image acquisition, or even providing a higher level of data mining: “In future, we expect that Deep Learning models trained end-to-end will blur the frontier between image reconstruction and image analysis.”

Potential applications of Deep Learning in fluorescence microscopy and key concepts. (a) Learning to reduce scattered light ’haze’ in light-sheet microscopy. (b) Learning spectral unmixing of simultaneous multi-color acquisitions. (c) Learning to reconstruct super-resolved images from structured illumination acquisitions. (d) Learning to straighten live dynamic samples with weak supervision against a template shape. (e) Fluorescence optical flow with Deep Learning. Forward model simulations can produce time-lapse data for given vector fields, and a network can be trained to compute the inverse: vector fields from fluorescence spatio-temporal fluctuations.

This manuscript also elegantly explains what the limitations of the approach are. When used with the wrong type of training dataset or when too few examples are available for it to learn, it can create hallucinations and therefore make up very realistic features that were not actually present in the underlying sample. The authors make a convincing case about looking beyond the scepticism sometimes found around applying AI to microscopy and focussing on where it is appropriate to use it and its performance.

What I like about this preprint

This preprint provides the accessible overview about AI applied to microscopy, that I have been waiting for, for a couple of years now. It is very well written and presents complex concepts with well-chosen examples. The authors make a balanced case of the pros and cons of AI, but highlight the areas where its potential remains untapped.

Future directions

It is clear to me that AI will change fundamentally the way we acquire and analyse our microscopy data. But in what ways? While this still remains to be established, this preprint sets the ground for an exciting path full of potential. For AI to become mainstream in microscopy, the bottleneck in my opinion still is usability and accessibility of these complex mathematical tools. Will every research department or institute soon need their own in-house AI team? Will these be run like some bioinformatics facilities are today?

[1] Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, (2018).

Tags: artificial intelligence, deep learning, fluorescence microscopy

Posted on: 9th January 2019

Read preprint (4 votes)

  • Have your say

    Your email address will not be published. Required fields are marked *

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

    Sign up to customise the site to your preferences and to receive alerts

    Register here

    Also in the developmental biology category: