Menu

Close

Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, Maurício Rocha-Martins, Fabián Segovia-Miranda, Caren Norden, Ricardo Henriques, Marino Zerial, Michele Solimena, Jochen Rink, Pavel Tomancak, Loic Royer, Florian Jug, Eugene W. Myers

Preprint posted on January 23, 2018 https://www.biorxiv.org/content/early/2018/01/23/236463.1

Article now published in Nature Methods at http://dx.doi.org/10.1038/s41592-018-0216-7

Deep learning-based image restoration overcomes previously intractable barriers to high resolution imaging

Selected by Uri Manor

Summary

Deep learning-based image restoration overcomes previously intractable barriers.

 

How this work fits the bigger picture

Machine learning has been used by microscopists for many years, most famously for image segmentation and analysis. This paper presents the first nearly comprehensive demonstration of how deep learning-based image restoration has the potential to overthrow previous barriers to imaging such as phototoxicity, slow imaging speed, and limited resolution. While image restoration methods such as denoising, deconvolution, and compressed sensing have long been employed with great success, they are computationally demanding and often suffer from debilitating artifacts. These artifacts usually arise from mismatches between reality and the assumptions that are intrinsic to the mathematical equations which are used to process the images. Biology is complex, and biological samples are often “messy”, and to model the lightpath from excitation to emission to detection in even a single cell would require an equation far too complex to derive. The number of parameters are not only computationally daunting – they often cannot even be measured in the same sample that is being imaged. Thus, “rules-based” computational image restoration approaches are often compromises – estimates – of what would be calculated in an ideal world.

 

Enter this paper: With CARE™ (content aware restoration), the number of parameters is no longer a real issue. Highly parallelized GPU-based hardware, along with a properly constructed algorithm, is amply equipped to identify and adjust as many parameters as can be thrown at it for most practical purposes. By training the algorithm with sufficient pairs of degraded images alongside “ground truth” images (either the same sample imaged at high vs. low resolution or signal-to-noise ratio, or an analogously generated synthetic dataset pair), the algorithm will “learn” how to reconstruct the “ground truth” image from the input. Weigert et al. demonstrate this capability beautifully, with a focus on three major areas of improvement for microscopy:

  • Denoising (so we can image with lower laser power, and/or lower concentrations of fluorophores)
  • Axial undersampling (so we can image fewer z-slices, thereby increasing imaging speed and decreasing photoxicity)
  • Superresolution (so we can get high resolution images at much faster acquisition speeds) – below is an example of how deep learning compares with deconvolution for low SNR .
Top row: Acquired image of secretory granules (magenta) and microtubules (green) in insulin-secreting INS-1 cells. Middle row: Deep learning-based restoration of the image in the top row. Bottom row: Standard deconvolution-based restoration of the acquired image in the top row. Note the higher contrast, lower background, and greater signal-to-noise in the deep learning-based restoration in comparison to both the original acquired image as well as the deconvolved dataset.

 

Open questions  

How does CARE compare with more advanced denoising or deconvolution software? ER-Decon from the Sedat/Agard groups does particularly well with both denoising and deconvolution, and is cited in the paper, but wasn’t used for comparing with CARE – this would be much more stiff competition!

How would CARE (or any other deep learning architecture) work if it was trained only on images of one type of structure vs. a very broad range of structures? How broad a range of structures would be necessary to enable the software to “converge” on the “common denominator” of fluorescence microscopy? In other words, when can we hope to develop a generalized deconvolution software, as opposed to software that knows how to refine linear and/or dot structures (e.g. microtubules and punctate cellular structures) that it was trained to detect? How do we best construct the software, and/or how do we best train said software?

 

Tags: artificial intelligence, deep learning, fluorescence, imaging, microscopy, superresolution

Posted on: 22nd February 2018

Read preprint (4 votes)




  • Have your say

    Your email address will not be published. Required fields are marked *

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

    Sign up to customise the site to your preferences and to receive alerts

    Register here

    preLists in the cell biology category:

    FENS 2020

    A collection of preprints presented during the virtual meeting of the Federation of European Neuroscience Societies (FENS) in 2020

     



    List by Ana Dorrego-Rivas

    Planar Cell Polarity – PCP

    This preList contains preprints about the latest findings on Planar Cell Polarity (PCP) in various model organisms at the molecular, cellular and tissue levels.

     



    List by Ana Dorrego-Rivas

    BioMalPar XVI: Biology and Pathology of the Malaria Parasite

    [under construction] Preprints presented at the (fully virtual) EMBL BioMalPar XVI, 17-18 May 2020 #emblmalaria

     



    List by Gautam Dey, Samantha Seah

    1

    Cell Polarity

    Recent research from the field of cell polarity is summarized in this list of preprints. It comprises of studies focusing on various forms of cell polarity ranging from epithelial polarity, planar cell polarity to front-to-rear polarity.

     



    List by Yamini Ravichandran

    TAGC 2020

    Preprints recently presented at the virtual Allied Genetics Conference, April 22-26, 2020. #TAGC20

     



    List by Maiko Kitaoka, Madhuja Samaddar, Miguel V. Almeida, Sejal Davla, Jennifer Ann Black, Gautam Dey

    3D Gastruloids

    A curated list of preprints related to Gastruloids (in vitro models of early development obtained by 3D aggregation of embryonic cells)

     



    List by Paul Gerald L. Sanchez and Stefano Vianello

    ECFG15 – Fungal biology

    Preprints presented at 15th European Conference on Fungal Genetics 17-20 February 2020 Rome

     



    List by Hiral Shah

    ASCB EMBO Annual Meeting 2019

    A collection of preprints presented at the 2019 ASCB EMBO Meeting in Washington, DC (December 7-11)

     



    List by Madhuja Samaddar, Ramona Jühlen, Amanda Haage, Laura McCormick, Maiko Kitaoka

    EMBL Seeing is Believing – Imaging the Molecular Processes of Life

    Preprints discussed at the 2019 edition of Seeing is Believing, at EMBL Heidelberg from the 9th-12th October 2019

     



    List by Gautam Dey

    Autophagy

    Preprints on autophagy and lysosomal degradation and its role in neurodegeneration and disease. Includes molecular mechanisms, upstream signalling and regulation as well as studies on pharmaceutical interventions to upregulate the process.

     



    List by Sandra Malmgren Hill

    Lung Disease and Regeneration

    This preprint list compiles highlights from the field of lung biology.

     



    List by Rob Hynds

    Cellular metabolism

    A curated list of preprints related to cellular metabolism at Biorxiv by Pablo Ranea Robles from the Prelights community. Special interest on lipid metabolism, peroxisomes and mitochondria.

     



    List by Pablo Ranea Robles

    BSCB/BSDB Annual Meeting 2019

    Preprints presented at the BSCB/BSDB Annual Meeting 2019

     



    List by Gautam Dey

    MitoList

    This list of preprints is focused on work expanding our knowledge on mitochondria in any organism, tissue or cell type, from the normal biology to the pathology.

     



    List by Sandra Franco Iborra

    Biophysical Society Annual Meeting 2019

    Few of the preprints that were discussed in the recent BPS annual meeting at Baltimore, USA

     



    List by Joseph Jose Thottacherry

    ASCB/EMBO Annual Meeting 2018

    This list relates to preprints that were discussed at the recent ASCB conference.

     



    List by Gautam Dey, Amanda Haage

    Also in the developmental biology category:

    Society for Developmental Biology 79th Annual Meeting

    Preprints at SDB 2020

     



    List by Irepan Salvador-Martinez, Martin Estermann

    FENS 2020

    A collection of preprints presented during the virtual meeting of the Federation of European Neuroscience Societies (FENS) in 2020

     



    List by Ana Dorrego-Rivas

    Planar Cell Polarity – PCP

    This preList contains preprints about the latest findings on Planar Cell Polarity (PCP) in various model organisms at the molecular, cellular and tissue levels.

     



    List by Ana Dorrego-Rivas

    Cell Polarity

    Recent research from the field of cell polarity is summarized in this list of preprints. It comprises of studies focusing on various forms of cell polarity ranging from epithelial polarity, planar cell polarity to front-to-rear polarity.

     



    List by Yamini Ravichandran

    TAGC 2020

    Preprints recently presented at the virtual Allied Genetics Conference, April 22-26, 2020. #TAGC20

     



    List by Maiko Kitaoka, Madhuja Samaddar, Miguel V. Almeida, Sejal Davla, Jennifer Ann Black, Gautam Dey

    3D Gastruloids

    A curated list of preprints related to Gastruloids (in vitro models of early development obtained by 3D aggregation of embryonic cells)

     



    List by Paul Gerald L. Sanchez and Stefano Vianello

    ASCB EMBO Annual Meeting 2019

    A collection of preprints presented at the 2019 ASCB EMBO Meeting in Washington, DC (December 7-11)

     



    List by Madhuja Samaddar, Ramona Jühlen, Amanda Haage, Laura McCormick, Maiko Kitaoka

    EDBC Alicante 2019

    Preprints presented at the European Developmental Biology Congress (EDBC) in Alicante, October 23-26 2019.

     



    List by Sergio Menchero, Jesus Victorino, Teresa Rayon, Irepan Salvador-Martinez

    EMBL Seeing is Believing – Imaging the Molecular Processes of Life

    Preprints discussed at the 2019 edition of Seeing is Believing, at EMBL Heidelberg from the 9th-12th October 2019

     



    List by Gautam Dey

    SDB 78th Annual Meeting 2019

    A curation of the preprints presented at the SDB meeting in Boston, July 26-30 2019. The preList will be updated throughout the duration of the meeting.

     



    List by Alex Eve

    Lung Disease and Regeneration

    This preprint list compiles highlights from the field of lung biology.

     



    List by Rob Hynds

    Young Embryologist Network Conference 2019

    Preprints presented at the Young Embryologist Network 2019 conference, 13 May, The Francis Crick Institute, London

     



    List by Alex Eve

    Pattern formation during development

    The aim of this preList is to integrate results about the mechanisms that govern patterning during development, from genes implicated in the processes to theoritical models of pattern formation in nature.

     



    List by Alexa Sadier

    BSCB/BSDB Annual Meeting 2019

    Preprints presented at the BSCB/BSDB Annual Meeting 2019

     



    List by Gautam Dey

    Zebrafish immunology

    A compilation of cutting-edge research that uses the zebrafish as a model system to elucidate novel immunological mechanisms in health and disease.

     



    List by Shikha Nayar
    Close