Menu

Close

Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, Maurício Rocha-Martins, Fabián Segovia-Miranda, Caren Norden, Ricardo Henriques, Marino Zerial, Michele Solimena, Jochen Rink, Pavel Tomancak, Loic Royer, Florian Jug, Eugene W. Myers

Preprint posted on January 23, 2018 https://www.biorxiv.org/content/early/2018/01/23/236463.1

Deep learning-based image restoration overcomes previously intractable barriers to high resolution imaging

Selected by Uri Manor

Summary

Deep learning-based image restoration overcomes previously intractable barriers.

 

How this work fits the bigger picture

Machine learning has been used by microscopists for many years, most famously for image segmentation and analysis. This paper presents the first nearly comprehensive demonstration of how deep learning-based image restoration has the potential to overthrow previous barriers to imaging such as phototoxicity, slow imaging speed, and limited resolution. While image restoration methods such as denoising, deconvolution, and compressed sensing have long been employed with great success, they are computationally demanding and often suffer from debilitating artifacts. These artifacts usually arise from mismatches between reality and the assumptions that are intrinsic to the mathematical equations which are used to process the images. Biology is complex, and biological samples are often “messy”, and to model the lightpath from excitation to emission to detection in even a single cell would require an equation far too complex to derive. The number of parameters are not only computationally daunting – they often cannot even be measured in the same sample that is being imaged. Thus, “rules-based” computational image restoration approaches are often compromises – estimates – of what would be calculated in an ideal world.

 

Enter this paper: With CARE™ (content aware restoration), the number of parameters is no longer a real issue. Highly parallelized GPU-based hardware, along with a properly constructed algorithm, is amply equipped to identify and adjust as many parameters as can be thrown at it for most practical purposes. By training the algorithm with sufficient pairs of degraded images alongside “ground truth” images (either the same sample imaged at high vs. low resolution or signal-to-noise ratio, or an analogously generated synthetic dataset pair), the algorithm will “learn” how to reconstruct the “ground truth” image from the input. Weigert et al. demonstrate this capability beautifully, with a focus on three major areas of improvement for microscopy:

  • Denoising (so we can image with lower laser power, and/or lower concentrations of fluorophores)
  • Axial undersampling (so we can image fewer z-slices, thereby increasing imaging speed and decreasing photoxicity)
  • Superresolution (so we can get high resolution images at much faster acquisition speeds) – below is an example of how deep learning compares with deconvolution for low SNR .
Top row: Acquired image of secretory granules (magenta) and microtubules (green) in insulin-secreting INS-1 cells. Middle row: Deep learning-based restoration of the image in the top row. Bottom row: Standard deconvolution-based restoration of the acquired image in the top row. Note the higher contrast, lower background, and greater signal-to-noise in the deep learning-based restoration in comparison to both the original acquired image as well as the deconvolved dataset.

 

Open questions  

How does CARE compare with more advanced denoising or deconvolution software? ER-Decon from the Sedat/Agard groups does particularly well with both denoising and deconvolution, and is cited in the paper, but wasn’t used for comparing with CARE – this would be much more stiff competition!

How would CARE (or any other deep learning architecture) work if it was trained only on images of one type of structure vs. a very broad range of structures? How broad a range of structures would be necessary to enable the software to “converge” on the “common denominator” of fluorescence microscopy? In other words, when can we hope to develop a generalized deconvolution software, as opposed to software that knows how to refine linear and/or dot structures (e.g. microtubules and punctate cellular structures) that it was trained to detect? How do we best construct the software, and/or how do we best train said software?

 

Tags: artificial intelligence, deep learning, fluorescence, imaging, microscopy, superresolution

Read preprint (4 votes)




  • Have your say

    Your email address will not be published. Required fields are marked *

    Sign up to customise the site to your preferences and to receive alerts

    Register here

    Also in the cell biology category:

    Spatiotemporally controlled Myosin relocalization and internal pressure cause biased cortical extension to generate sibling cell size asymmetry

    Tri Thanh Pham, Arnaud Monnard, Jonne Helenius, et al.



    Selected by Giuliana Clemente

    Nuclear decoupling is part of a rapid protein-level cellular response to high-intensity mechanical loading

    Hamish T J Gilbert, Venkatesh Mallikarjun, Oana Dobre, et al.



    Selected by Rebecca Quelch

    1

    A robust method for transfection in choanoflagellates illuminates their cell biology and the ancestry of animal septins

    David Booth, Heather Middleton, Nicole King



    Selected by Maya Emmons-Bell

    SWI/SNF remains localized to chromatin in the presence of SCHLAP1

    Jesse R Raab, Keriayn N Smith, Camarie C Spear, et al.



    Selected by Carmen Adriaens

    1

    Clathrin plaques form mechanotransducing platforms

    Agathe Franck, Jeanne Laine, Gilles Moulay, et al.



    Selected by Amanda Haage

    Cellular Crowding Influences Extrusion and Proliferation to Facilitate Epithelial Tissue Repair

    Jovany Jeomar Franco, Youmna Maryline Atieh, Chase Dallas Bryan, et al.



    Selected by Helen Weavers

    A non-cell autonomous actin redistribution enables isotropic retinal growth

    Marija Matejcic, Guillaume Salbreux, Caren Norden



    Selected by Yara E. Sánchez Corrales

    1

    Rearing temperature and fatty acid supplementation jointly affect membrane fluidity and heat tolerance in Daphnia

    Dominik Martin-Creuzburg, Bret L. Coggins, Dieter Ebert, et al.



    Selected by Alexander Little

    Live-cell imaging of marked chromosome regions reveals dynamics of mitotic chromosome resolution and compaction

    John K Eykelenboom, Marek Gierlinski, Zuojun Yue, et al.

    AND

    Quantitative imaging of chromatin decompaction in living cells

    Elisa Dultz, Roberta Mancini, Guido Polles, et al.



    Selected by Carmen Adriaens, Gautam Dey

    Optogenetic reconstitution reveals that Dynein-Dynactin-NuMA clusters generate cortical spindle-pulling forces as a multi-arm ensemble

    Masako Okumura, Toyoaki Natsume, Masato T Kanemaki, et al.



    Selected by Arnaud Monnard

    1

    Optogenetic reconstitution reveals that Dynein-Dynactin-NuMA clusters generate cortical spindle-pulling forces as a multi-arm ensemble

    Masako Okumura, Toyoaki Natsume, Masato T Kanemaki, et al.



    Selected by Ben Craske, Thibault Legal and Toni McHugh

    A novel microtubule nucleation pathway for meiotic spindle assembly in oocytes

    Pierre ROME, Hiroyuki OHKURA



    Selected by Binyam Mogessie

    ERM proteins: The missing actin linkers in clathrin-mediated endocytosis

    Audun Sverre Kvalvaag, Kay Oliver Schink, Andreas Brech, et al.



    Selected by Nicola Stevenson

    Cell type-specific interchromosomal interactions as a mechanism for transcriptional diversity

    Adan Horta, Kevin Monahan, Lisa Bashkirova, et al.



    Selected by Boyan Bonev

    A non-canonical role for dynamin-1 in regulating early stages of clathrin-mediated endocytosis in non-neuronal cells

    Saipraveen Srinivasan, Christoph J. Burckhardt, Madhura Bhave, et al.



    Selected by Penelope La-Borde

    Atomic model of microtubule-bound tau

    Elizabeth H Kellogg, Nisreen M.A. Hejab, Simon Poepsel, et al.



    Selected by Satish Bodakuntla

    1

    Also in the developmental biology category:

    Human macrophages survive and adopt activated genotypes in living zebrafish

    Colin D. Paul, Alexus Devine, Kevin Bishop, et al.



    Selected by Giuliana Clemente

    Altering the temporal regulation of one transcription factor drives sensory trade-offs

    Ariane Ramaekers, Simon Weinberger, Annelies Claeys, et al.



    Selected by Mariana R.P. Alves

    Presence of midline cilia supersedes the expression of Lefty1 in forming the midline barrier during the establishment of left-right asymmetry

    Natalia A Shylo, Dylan A Ramrattan, Scott D Weatherbee



    Selected by Hannah Brunsdon

    Genetic compensation is triggered by mutant mRNA degradation

    Mohamed El-Brolosy, Andrea Rossi, Zacharias Kontarakis, et al.



    Selected by Andreas van Impel

    1

    Cellular Crowding Influences Extrusion and Proliferation to Facilitate Epithelial Tissue Repair

    Jovany Jeomar Franco, Youmna Maryline Atieh, Chase Dallas Bryan, et al.



    Selected by Helen Weavers

    Photoperiod sensing of the circadian clock is controlled by ELF3 and GI

    Usman Anwer, Amanda Davis, Seth Jon Davis, et al.



    Selected by Annika Weimer

    Limb- and tendon-specific Adamtsl2 deletion identifies a soft tissue mechanism modulating bone length

    Dirk Hubmacher, Stetson Thacker, Sheila M Adams, et al.



    Selected by Alberto Rosello-Diez

    A non-cell autonomous actin redistribution enables isotropic retinal growth

    Marija Matejcic, Guillaume Salbreux, Caren Norden



    Selected by Yara E. Sánchez Corrales

    1

    Rearing temperature and fatty acid supplementation jointly affect membrane fluidity and heat tolerance in Daphnia

    Dominik Martin-Creuzburg, Bret L. Coggins, Dieter Ebert, et al.



    Selected by Alexander Little

    A SoxB gene acts as an anterior gap gene and regulates posterior segment addition in the spider Parasteatoda tepidariorum

    Christian L. B. Paese, Anna Schoenauer, Daniel J. Leite, et al.



    Selected by Erik Clark

    1

    Cell type-specific interchromosomal interactions as a mechanism for transcriptional diversity

    Adan Horta, Kevin Monahan, Lisa Bashkirova, et al.



    Selected by Boyan Bonev

    Germ layer specific regulation of cell polarity and adhesion gives insight into the evolution of mesoderm.

    Miguel Salinas-Saavedra, Amber Q. Rock, Mark Q. Martindale



    Selected by ClaireS & SophieM

    1

    Wnt/β-catenin regulates an ancient signaling network during zebrafish scale development

    Andrew J Aman, Alexis N Fulbright, David M Parichy



    Selected by Andreas van Impel

    Tissue flow induces cell shape changes during organogenesis

    Gonca Erdemci-Tandogan, Madeline J.Clark, Jeffrey D. Amack, et al.



    Selected by Jacky G. Goetz

    Temporal Control of Transcription by Zelda in living Drosophila embryos

    Jeremy Dufourt, Antonio Trullo, Jennifer Hunter, et al.



    Selected by Teresa Rayon

    1

    An atlas of silencer elements for the human and mouse genomes

    Naresh Doni Jayavelu, Ajay Jajodia, Arpit Mishra, et al.



    Selected by Rafael Galupa

    1

    Also in the physiology category:

    Rearing temperature and fatty acid supplementation jointly affect membrane fluidity and heat tolerance in Daphnia

    Dominik Martin-Creuzburg, Bret L. Coggins, Dieter Ebert, et al.



    Selected by Alexander Little

    Long-term live imaging of the Drosophila adult midgut reveals real-time dynamics of cell division, differentiation, and loss

    Judy Martin, Erin Nicole Sanders, Paola Moreno-Roman, et al.



    Selected by Natalie Dye

    Individual- and population-level drivers of consistent foraging success across environments

    Lysanne Snijders, Ralf HJM Kurvers, Stefan Krause, et al.



    Selected by Rasmus Ern

    Zebrafish as a model to investigate the effects of exercise in cancer

    Alexandra Yin, Nathaniel R. Campbell, Lee W. Jones, et al.



    Selected by Jacky G. Goetz

    Feedback control of neurogenesis by tissue packing

    Tom W. Hiscock, Joel B. Miesfeld, Kishore R. Mosaliganti, et al.



    Selected by Sarah Morson

    1

    Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

    Martin Weigert, Uwe Schmidt, Tobias Boothe, et al.



    Selected by Uri Manor

    Galleria mellonella as an Insect Model for P. destructans, the Cause of White-Nose Syndrome in Bats

    Chapman N Beekman, Lauren Meckler, Eleanor Kim, et al.



    Selected by Heath MacMillan

    Genome-wide selection scans integrated with association mapping reveal mechanisms of physiological adaptation across a salinity gradient in killifish

    Reid S. Brennan, Timothy M. Healy, Heather J. Bryant, et al.



    Selected by Andy Turko

    From Armament to Ornament: Performance Trade-Offs in the Sexual Weaponry of Neotropical Electric Fishes

    Kory M. Evans, Maxwell J. Bernt, Matthew A. Kolmann, et al.



    Selected by Cassandra Donatelli
    Close

    We want to make our website, and the services we provide, useful and reliable. This sometimes involves placing small amounts of information called cookies on the device you used to access the internet. If you continue to use this website we will assume you are happy to accept our cookies.

    Accept