Menu

Close

Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, Maurício Rocha-Martins, Fabián Segovia-Miranda, Caren Norden, Ricardo Henriques, Marino Zerial, Michele Solimena, Jochen Rink, Pavel Tomancak, Loic Royer, Florian Jug, Eugene W. Myers

Preprint posted on January 23, 2018 https://www.biorxiv.org/content/early/2018/01/23/236463.1

Deep learning-based image restoration overcomes previously intractable barriers to high resolution imaging

Selected by Uri Manor

Summary

Deep learning-based image restoration overcomes previously intractable barriers.

 

How this work fits the bigger picture

Machine learning has been used by microscopists for many years, most famously for image segmentation and analysis. This paper presents the first nearly comprehensive demonstration of how deep learning-based image restoration has the potential to overthrow previous barriers to imaging such as phototoxicity, slow imaging speed, and limited resolution. While image restoration methods such as denoising, deconvolution, and compressed sensing have long been employed with great success, they are computationally demanding and often suffer from debilitating artifacts. These artifacts usually arise from mismatches between reality and the assumptions that are intrinsic to the mathematical equations which are used to process the images. Biology is complex, and biological samples are often “messy”, and to model the lightpath from excitation to emission to detection in even a single cell would require an equation far too complex to derive. The number of parameters are not only computationally daunting – they often cannot even be measured in the same sample that is being imaged. Thus, “rules-based” computational image restoration approaches are often compromises – estimates – of what would be calculated in an ideal world.

 

Enter this paper: With CARE™ (content aware restoration), the number of parameters is no longer a real issue. Highly parallelized GPU-based hardware, along with a properly constructed algorithm, is amply equipped to identify and adjust as many parameters as can be thrown at it for most practical purposes. By training the algorithm with sufficient pairs of degraded images alongside “ground truth” images (either the same sample imaged at high vs. low resolution or signal-to-noise ratio, or an analogously generated synthetic dataset pair), the algorithm will “learn” how to reconstruct the “ground truth” image from the input. Weigert et al. demonstrate this capability beautifully, with a focus on three major areas of improvement for microscopy:

  • Denoising (so we can image with lower laser power, and/or lower concentrations of fluorophores)
  • Axial undersampling (so we can image fewer z-slices, thereby increasing imaging speed and decreasing photoxicity)
  • Superresolution (so we can get high resolution images at much faster acquisition speeds) – below is an example of how deep learning compares with deconvolution for low SNR .
Top row: Acquired image of secretory granules (magenta) and microtubules (green) in insulin-secreting INS-1 cells. Middle row: Deep learning-based restoration of the image in the top row. Bottom row: Standard deconvolution-based restoration of the acquired image in the top row. Note the higher contrast, lower background, and greater signal-to-noise in the deep learning-based restoration in comparison to both the original acquired image as well as the deconvolved dataset.

 

Open questions  

How does CARE compare with more advanced denoising or deconvolution software? ER-Decon from the Sedat/Agard groups does particularly well with both denoising and deconvolution, and is cited in the paper, but wasn’t used for comparing with CARE – this would be much more stiff competition!

How would CARE (or any other deep learning architecture) work if it was trained only on images of one type of structure vs. a very broad range of structures? How broad a range of structures would be necessary to enable the software to “converge” on the “common denominator” of fluorescence microscopy? In other words, when can we hope to develop a generalized deconvolution software, as opposed to software that knows how to refine linear and/or dot structures (e.g. microtubules and punctate cellular structures) that it was trained to detect? How do we best construct the software, and/or how do we best train said software?

 

Tags: artificial intelligence, deep learning, fluorescence, imaging, microscopy, superresolution

Posted on: 22nd February 2018

Read preprint (4 votes)




  • Have your say

    Your email address will not be published. Required fields are marked *

    Sign up to customise the site to your preferences and to receive alerts

    Register here

    Also in the cell biology category:

    SABER enables highly multiplexed and amplified detection of DNA and RNA in cells and tissues

    Jocelyn Y. Kishi, Brian J. Beliveau, Sylvain W. Lapan, et al.



    Selected by Yen-Chung Chen

    PIKfyve/Fab1 is required for efficient V-ATPase and hydrolase delivery to phagosomes, phagosomal killing, and restriction of Legionella infection

    Catherine M Buckley, Victoria L Heath, Aurelie Gueho, et al.



    Selected by Giuliana Clemente

    1

    Memory sequencing reveals heritable single cell gene expression programs associated with distinct cellular behaviors

    Sydney M Shaffer, Benjamin L Emert, Ann E. Sizemore, et al.



    Selected by Leighton Daigh

    1

    Aurora A depletion reveals centrosome-independent polarization mechanism in C. elegans

    Kerstin Klinkert, Nicolas Levernier, Peter Gross, et al.

    AND

    Centrosome Aurora A gradient ensures a single PAR-2 polarity axis by regulating RhoGEF ECT-2 localization in C. elegans embryos

    Sachin Kotak, Sukriti Kapoor



    Selected by Giuliana Clemente

    Anti-angiogenic effects of VEGF stimulation on endothelium deficient in phosphoinositide recycling

    Amber N Stratman, Olivia M Farrelly, Constantinos M Mikelis, et al.



    Selected by Coert Margadant

    Neural crest cells regulate optic cup morphogenesis by promoting extracellular matrix assembly

    Chase Dallas Bryan, Rebecca Lynne Pfeiffer, Bryan William Jones, et al.



    Selected by Ashrifia Adomako-Ankomah

    1

    mRNA localisation in endothelial cells regulates blood vessel sprouting

    Guilherme Costa, Nawseen Tarannum, Shane Herbert



    Selected by Andreas van Impel

    Local protein synthesis in axon terminals and dendritic spines differentiates plasticity contexts

    Anne-Sophie Hafner, Paul Donlin-Asp, Beulah Leitch, et al.



    Selected by Dipen Rajgor

    The cytoskeleton as a smart composite material: A unified pathway linking microtubules, myosin-II filaments and integrin adhesions

    Nisha Mohd Rafiq, Yukako Nishimura, Sergey V. Plotnikov, et al.



    Selected by Coert Margadant

    Quantitative, real-time, single cell analysis in tissue reveals expression dynamics of neurogenesis

    Cerys S Manning, Veronica Biga, James Boyd, et al.



    Selected by Teresa Rayon

    Profiling the surface proteome identifies actionable biology for TSC1 mutant cells beyond mTORC1 signaling

    Junnian Wei, Kevin K. Leung, Charles Truillet, et al.



    Selected by Rob Hynds

    1

    Optogenetic dissection of mitotic spindle positioning in vivo

    Lars-Eric Fielmich, Ruben Schmidt, Daniel J Dickinson, et al.



    Selected by Angika Basant

    1

    Excitable RhoA dynamics drive pulsed contractions in the early C. elegans embryo.

    Jonathan B Michaux, Francois B Robin, William M McFadden, et al.



    Selected by Sundar Naganathan

    Moving beyond P values: Everyday data analysis with estimation plots

    Joses Ho, Tayfun Tumkaya, Sameer Aryal, et al.



    Selected by Gautam Dey

    1

    A limited number of double-strand DNA breaks are sufficient to delay cell cycle progression.

    Jeroen van den Berg, Anna G. Manjon, Karoline Kielbassa, et al.



    Selected by Leighton Daigh

    Optogenetic manipulation of medullary neurons in the locust optic lobe

    Hongxia Wang, Richard B. Dewell, Markus U. Ehrengruber, et al.



    Selected by Ana Patricia Ramos

    Also in the developmental biology category:

    Polyacrylamide Bead Sensors for in vivo Quantification of Cell-Scale Stress in Zebrafish Development

    Nicole Traeber, Klemens Uhlmann, Salvatore Girardo, et al.



    Selected by Jacky G. Goetz

    millepattes micropeptides are an ancient developmental switch required for embryonic patterning

    Suparna Ray, Miriam I Rosenberg, Hélène Chanut-Delalande, et al.



    Selected by Erik Clark

    Aurora A depletion reveals centrosome-independent polarization mechanism in C. elegans

    Kerstin Klinkert, Nicolas Levernier, Peter Gross, et al.

    AND

    Centrosome Aurora A gradient ensures a single PAR-2 polarity axis by regulating RhoGEF ECT-2 localization in C. elegans embryos

    Sachin Kotak, Sukriti Kapoor



    Selected by Giuliana Clemente

    Anti-angiogenic effects of VEGF stimulation on endothelium deficient in phosphoinositide recycling

    Amber N Stratman, Olivia M Farrelly, Constantinos M Mikelis, et al.



    Selected by Coert Margadant

    SOL1 and SOL2 Regulate Fate Transition and Cell Divisions in the Arabidopsis Stomatal Lineage

    Abigail R Simmons, Kelli A Davies, Wanpeng Wang, et al.



    Selected by Martin Balcerowicz

    Analysis of the role of Nidogen/entactin in basement membrane assembly and morphogenesis in Drosophila

    Jianli Dai, Beatriz Estrada, Sofie Jacobs, et al.



    Selected by Nargess Khalilgharibi

    Neural crest cells regulate optic cup morphogenesis by promoting extracellular matrix assembly

    Chase Dallas Bryan, Rebecca Lynne Pfeiffer, Bryan William Jones, et al.



    Selected by Ashrifia Adomako-Ankomah

    1

    Simultaneous production of diverse neuronal subtypes during early corticogenesis

    Elia Magrinelli, Robin Jan Wagener, Denis Jabaudon



    Selected by Boyan Bonev

    1

    mRNA localisation in endothelial cells regulates blood vessel sprouting

    Guilherme Costa, Nawseen Tarannum, Shane Herbert



    Selected by Andreas van Impel

    LADL: Light-activated dynamic looping for endogenous gene expression control

    Mayuri Rege, Ji Hun Kim, Jacqueline Valeri, et al.



    Selected by Ivan Candido-Ferreira

    Quantitative, real-time, single cell analysis in tissue reveals expression dynamics of neurogenesis

    Cerys S Manning, Veronica Biga, James Boyd, et al.



    Selected by Teresa Rayon

    The cis-regulatory logic underlying abdominal Hox-mediated repression versus activation of regulatory elements in Drosophila



    Selected by Clarice Hong

    1

    Optogenetic dissection of mitotic spindle positioning in vivo

    Lars-Eric Fielmich, Ruben Schmidt, Daniel J Dickinson, et al.



    Selected by Angika Basant

    1

    Excitable RhoA dynamics drive pulsed contractions in the early C. elegans embryo.

    Jonathan B Michaux, Francois B Robin, William M McFadden, et al.



    Selected by Sundar Naganathan

    A transition from SoxB1 to SoxE transcription factors is essential for progression from pluripotent blastula cells to neural crest cells

    Elsy Buitrago-Delgado, Elizabeth Schock, Kara Nordin, et al.



    Selected by Amanda Haage

    Moving beyond P values: Everyday data analysis with estimation plots

    Joses Ho, Tayfun Tumkaya, Sameer Aryal, et al.



    Selected by Gautam Dey

    1

    Also in the physiology category:

    Optogenetic manipulation of medullary neurons in the locust optic lobe

    Hongxia Wang, Richard B. Dewell, Markus U. Ehrengruber, et al.



    Selected by Ana Patricia Ramos

    Targeting light-gated chloride channels to neuronal somatodendritic domain reduces their excitatory effect in the axon

    Jessica Messier, Hongmei Chen, Zhao-Lin Cai, et al.

    AND

    High-efficiency optogenetic silencing with soma-targeted anion-conducting channelrhodopsins

    Mathias Mahn, Lihi Gibor, Katayun Cohen-Kashi Malina, et al.



    Selected by Mahesh Karnani

    2

    Rearing temperature and fatty acid supplementation jointly affect membrane fluidity and heat tolerance in Daphnia

    Dominik Martin-Creuzburg, Bret L. Coggins, Dieter Ebert, et al.



    Selected by Alexander Little

    Long-term live imaging of the Drosophila adult midgut reveals real-time dynamics of cell division, differentiation, and loss

    Judy Martin, Erin Nicole Sanders, Paola Moreno-Roman, et al.



    Selected by Natalie Dye

    Individual- and population-level drivers of consistent foraging success across environments

    Lysanne Snijders, Ralf HJM Kurvers, Stefan Krause, et al.



    Selected by Rasmus Ern

    Zebrafish as a model to investigate the effects of exercise in cancer

    Alexandra Yin, Nathaniel R. Campbell, Lee W. Jones, et al.



    Selected by Jacky G. Goetz

    Feedback control of neurogenesis by tissue packing

    Tom W. Hiscock, Joel B. Miesfeld, Kishore R. Mosaliganti, et al.



    Selected by Sarah Morson

    1

    Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

    Martin Weigert, Uwe Schmidt, Tobias Boothe, et al.



    Selected by Uri Manor

    Galleria mellonella as an Insect Model for P. destructans, the Cause of White-Nose Syndrome in Bats

    Chapman N Beekman, Lauren Meckler, Eleanor Kim, et al.



    Selected by Heath MacMillan

    Genome-wide selection scans integrated with association mapping reveal mechanisms of physiological adaptation across a salinity gradient in killifish

    Reid S. Brennan, Timothy M. Healy, Heather J. Bryant, et al.



    Selected by Andy Turko

    From Armament to Ornament: Performance Trade-Offs in the Sexual Weaponry of Neotropical Electric Fishes

    Kory M. Evans, Maxwell J. Bernt, Matthew A. Kolmann, et al.



    Selected by Cassandra Donatelli
    Close