Menu

Close

Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, Maurício Rocha-Martins, Fabián Segovia-Miranda, Caren Norden, Ricardo Henriques, Marino Zerial, Michele Solimena, Jochen Rink, Pavel Tomancak, Loic Royer, Florian Jug, Eugene W. Myers

Preprint posted on January 23, 2018 https://www.biorxiv.org/content/early/2018/01/23/236463.1

Deep learning-based image restoration overcomes previously intractable barriers to high resolution imaging

Selected by Uri Manor

Summary

Deep learning-based image restoration overcomes previously intractable barriers.

 

How this work fits the bigger picture

Machine learning has been used by microscopists for many years, most famously for image segmentation and analysis. This paper presents the first nearly comprehensive demonstration of how deep learning-based image restoration has the potential to overthrow previous barriers to imaging such as phototoxicity, slow imaging speed, and limited resolution. While image restoration methods such as denoising, deconvolution, and compressed sensing have long been employed with great success, they are computationally demanding and often suffer from debilitating artifacts. These artifacts usually arise from mismatches between reality and the assumptions that are intrinsic to the mathematical equations which are used to process the images. Biology is complex, and biological samples are often “messy”, and to model the lightpath from excitation to emission to detection in even a single cell would require an equation far too complex to derive. The number of parameters are not only computationally daunting – they often cannot even be measured in the same sample that is being imaged. Thus, “rules-based” computational image restoration approaches are often compromises – estimates – of what would be calculated in an ideal world.

 

Enter this paper: With CARE™ (content aware restoration), the number of parameters is no longer a real issue. Highly parallelized GPU-based hardware, along with a properly constructed algorithm, is amply equipped to identify and adjust as many parameters as can be thrown at it for most practical purposes. By training the algorithm with sufficient pairs of degraded images alongside “ground truth” images (either the same sample imaged at high vs. low resolution or signal-to-noise ratio, or an analogously generated synthetic dataset pair), the algorithm will “learn” how to reconstruct the “ground truth” image from the input. Weigert et al. demonstrate this capability beautifully, with a focus on three major areas of improvement for microscopy:

  • Denoising (so we can image with lower laser power, and/or lower concentrations of fluorophores)
  • Axial undersampling (so we can image fewer z-slices, thereby increasing imaging speed and decreasing photoxicity)
  • Superresolution (so we can get high resolution images at much faster acquisition speeds) – below is an example of how deep learning compares with deconvolution for low SNR .
Top row: Acquired image of secretory granules (magenta) and microtubules (green) in insulin-secreting INS-1 cells. Middle row: Deep learning-based restoration of the image in the top row. Bottom row: Standard deconvolution-based restoration of the acquired image in the top row. Note the higher contrast, lower background, and greater signal-to-noise in the deep learning-based restoration in comparison to both the original acquired image as well as the deconvolved dataset.

 

Open questions  

How does CARE compare with more advanced denoising or deconvolution software? ER-Decon from the Sedat/Agard groups does particularly well with both denoising and deconvolution, and is cited in the paper, but wasn’t used for comparing with CARE – this would be much more stiff competition!

How would CARE (or any other deep learning architecture) work if it was trained only on images of one type of structure vs. a very broad range of structures? How broad a range of structures would be necessary to enable the software to “converge” on the “common denominator” of fluorescence microscopy? In other words, when can we hope to develop a generalized deconvolution software, as opposed to software that knows how to refine linear and/or dot structures (e.g. microtubules and punctate cellular structures) that it was trained to detect? How do we best construct the software, and/or how do we best train said software?

 

Tags: artificial intelligence, deep learning, fluorescence, imaging, microscopy, superresolution

Posted on: 22nd February 2018

Read preprint (4 votes)




  • Have your say

    Your email address will not be published. Required fields are marked *

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

    Sign up to customise the site to your preferences and to receive alerts

    Register here

    Also in the cell biology category:

    Microtubules Gate Tau Condensation to Spatially Regulate Microtubule Functions

    Ruensern Tan, Aileen J. Lam, Tracy Tan, et al.

    AND

    Kinetically distinct phases of tau on microtubules regulate kinesin motors and severing enzymes

    Valerie Siahaan, Jochen Krattenmacher, Amayra Hernandez-Vega, et al.



    Selected by Satish Bodakuntla

    The modular mechanism of chromocenter formation in Drosophila

    Madhav Jagannathan, Ryan Cummings, Yukiko M Yamashita



    Selected by Maiko Kitaoka

    1

    The cell wall regulates dynamics and size of plasma-membrane nanodomains in Arabidopsis.

    Joseph Franics McKenna, Daniel Rolfe, Stephen E D Webb, et al.



    Selected by Marc Somssich

    Psychiatric risk gene NT5C2 regulates protein translation in human neural progenitor cells

    Rodrigo R.R. Duarte, Nathaniel D. Bachtel, Marie-Caroline Cotel, et al.



    Selected by Joanna Cross

    Old fibroblasts secrete inflammatory cytokines that drive variability in reprogramming efficiency and may affect wound healing between old individuals

    Salah Mahmoudi, Elena Mancini, Alessandra Moore, et al.



    Selected by Shikha Nayar

    DNA microscopy: Optics-free spatio-genetic imaging by a stand-alone chemical reaction

    Joshua A. Weinstein, Aviv Regev, Feng Zhang



    Selected by Theo Sanderson

    2

    The Toll pathway inhibits tissue growth and regulates cell fitness in an infection-dependent manner

    Federico Germani, Daniel Hain, Denise Sternlicht, et al.



    Selected by Rohan Khadilkar

    Drosophila kinesin-8 stabilises kinetochore-microtubule interaction

    Tomoya Edzuka, Gohta Goshima



    Selected by Ben Craske, Thibault Legal and Toni McHugh

    1

    Unjamming overcomes kinetic and proliferation arrest in terminally differentiated cells and promotes collective motility of carcinoma.

    Andrea Palamidessi, Chiara Malinverno, Emanuela FRITTOLI, et al.



    Selected by Tim Fessenden

    1

    Tension on kinetochore substrates is insufficient to prevent Aurora-triggered detachment

    Anna K de Regt, Charles L Asbury, Sue Biggins



    Selected by Angika Basant

    1

    Single-cell RNA sequencing reveals novel cell differentiation dynamics during human airway epithelium regeneration

    Sandra Ruiz Garcia, Marie Deprez, Kevin Lebrigand, et al.



    Selected by Rob Hynds

    1

    Actomyosin-driven tension at compartmental boundaries orients cell division independently of cell geometry in vivo

    Elena Scarpa, Cedric Finet, Guy Blanchard, et al.



    Selected by Ivana Viktorinová

    Molecular organization of integrin-based adhesion complexes in mouse Embryonic Stem Cells

    Shumin Xia, Evelyn K.F. Yim, Pakorn Kanchanawong

    AND

    Superresolution architecture of pluripotency guarding adhesions

    Aki Stubb, Camilo Guzmán, Elisa Närvä, et al.



    Selected by Nicola Stevenson, Amanda Haage

    Transcriptional initiation and mechanically driven self-propagation of a tissue contractile wave during axis elongation

    Anais Bailles, Claudio Collinet, Jean-Marc Philippe, et al.



    Selected by Sundar Naganathan

    1

    Bridging the divide: bacteria synthesizing archaeal membrane lipids

    Laura Villanueva, F. A. Bastiaan von Meijenfeldt, Alexander B. Westbye, et al.

    AND

    Extensive transfer of membrane lipid biosynthetic genes between Archaea and Bacteria

    Gareth A. Coleman, Richard D. Pancost, Tom A. Williams



    Selected by Gautam Dey

    1

    Revealing the nanoscale morphology of the primary cilium using super-resolution fluorescence microscopy

    Joshua Yoon, Colin J. Comerci, Lucien E. Weiss, et al.



    Selected by Gautam Dey

    Also in the developmental biology category:

    The cell wall regulates dynamics and size of plasma-membrane nanodomains in Arabidopsis.

    Joseph Franics McKenna, Daniel Rolfe, Stephen E D Webb, et al.



    Selected by Marc Somssich

    Psychiatric risk gene NT5C2 regulates protein translation in human neural progenitor cells

    Rodrigo R.R. Duarte, Nathaniel D. Bachtel, Marie-Caroline Cotel, et al.



    Selected by Joanna Cross

    A Scube2-Shh feedback loop links morphogen release to morphogen signaling to enable scale invariant patterning of the ventral neural tube

    Zach Collins, Kana Ishimatsu, Tony Tsai, et al.



    Selected by Teresa Rayon

    1

    Epiblast formation by Tead-Yap-dependent expression of pluripotency factors and competitive elimination of unspecified cells

    Masakazu Hashimoto, Hiroshi Sasaki



    Selected by Sarah Bowling, Teresa Rayon

    Single cell transcriptomics reveals spatial and temporal dynamics of gene expression in the developing mouse spinal cord

    Julien Delile, Teresa Rayon, Manuela Melchionda, et al.



    Selected by Reena Lasrado

    1

    The Toll pathway inhibits tissue growth and regulates cell fitness in an infection-dependent manner

    Federico Germani, Daniel Hain, Denise Sternlicht, et al.



    Selected by Rohan Khadilkar

    Rapid embryonic cell cycles defer the establishment of heterochromatin by Eggless/SetDB1 in Drosophila

    Charles A Seller, Chun-Yi Cho, Patrick H O'Farrell



    Selected by Gabriel Aughey

    A histidine kinase gene is required for large radius root tip circumnutation and surface exploration in rice

    Kevin R Lehner, Isaiah Taylor, Erin N McCaskey, et al.



    Selected by Martin Balcerowicz

    Actomyosin-driven tension at compartmental boundaries orients cell division independently of cell geometry in vivo

    Elena Scarpa, Cedric Finet, Guy Blanchard, et al.



    Selected by Ivana Viktorinová

    Molecular organization of integrin-based adhesion complexes in mouse Embryonic Stem Cells

    Shumin Xia, Evelyn K.F. Yim, Pakorn Kanchanawong

    AND

    Superresolution architecture of pluripotency guarding adhesions

    Aki Stubb, Camilo Guzmán, Elisa Närvä, et al.



    Selected by Nicola Stevenson, Amanda Haage

    Transcriptional initiation and mechanically driven self-propagation of a tissue contractile wave during axis elongation

    Anais Bailles, Claudio Collinet, Jean-Marc Philippe, et al.



    Selected by Sundar Naganathan

    1

    Signaling dynamics control cell fate in the early Drosophila embryo

    Heath E Johnson, Stanislav Y Shvartsman, Jared E Toettcher



    Selected by Yara E. Sánchez Corrales

    1

    Three-dimensional tissue stiffness mapping in the mouse embryo supports durotaxis during early limb bud morphogenesis

    Min Zhu, Hirotaka Tao, Mohammad Samani, et al.



    Selected by Natalie Dye

    PUMILIO hyperactivity drives premature aging of Norad-deficient mice

    Florian Kopp, Mehmet Yalvac, Beibei Chen, et al.



    Selected by Carmen Adriaens

    Synergy with TGFβ ligands switches WNT pathway dynamics from transient to sustained during human pluripotent cell differentiation

    Joseph Massey, Yida Liu, Omar Alvarenga, et al.



    Selected by Pierre Osteil

    1

    3D Tissue elongation via ECM stiffness-cued junctional remodeling

    Dong-Yuan Chen, Justin Crest, Sebastian J Streichan, et al.



    Selected by Sundar Naganathan

    Also in the physiology category:

    The Toll pathway inhibits tissue growth and regulates cell fitness in an infection-dependent manner

    Federico Germani, Daniel Hain, Denise Sternlicht, et al.



    Selected by Rohan Khadilkar

    Optogenetic manipulation of medullary neurons in the locust optic lobe

    Hongxia Wang, Richard B. Dewell, Markus U. Ehrengruber, et al.



    Selected by Ana Patricia Ramos

    Targeting light-gated chloride channels to neuronal somatodendritic domain reduces their excitatory effect in the axon

    Jessica Messier, Hongmei Chen, Zhao-Lin Cai, et al.

    AND

    High-efficiency optogenetic silencing with soma-targeted anion-conducting channelrhodopsins

    Mathias Mahn, Lihi Gibor, Katayun Cohen-Kashi Malina, et al.



    Selected by Mahesh Karnani

    2

    Rearing temperature and fatty acid supplementation jointly affect membrane fluidity and heat tolerance in Daphnia

    Dominik Martin-Creuzburg, Bret L. Coggins, Dieter Ebert, et al.



    Selected by Alexander Little

    Long-term live imaging of the Drosophila adult midgut reveals real-time dynamics of cell division, differentiation, and loss

    Judy Martin, Erin Nicole Sanders, Paola Moreno-Roman, et al.



    Selected by Natalie Dye

    Individual- and population-level drivers of consistent foraging success across environments

    Lysanne Snijders, Ralf HJM Kurvers, Stefan Krause, et al.



    Selected by Rasmus Ern

    Zebrafish as a model to investigate the effects of exercise in cancer

    Alexandra Yin, Nathaniel R. Campbell, Lee W. Jones, et al.



    Selected by Jacky G. Goetz

    Feedback control of neurogenesis by tissue packing

    Tom W. Hiscock, Joel B. Miesfeld, Kishore R. Mosaliganti, et al.



    Selected by Sarah Morson

    1

    Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

    Martin Weigert, Uwe Schmidt, Tobias Boothe, et al.



    Selected by Uri Manor

    Galleria mellonella as an Insect Model for P. destructans, the Cause of White-Nose Syndrome in Bats

    Chapman N Beekman, Lauren Meckler, Eleanor Kim, et al.



    Selected by Heath MacMillan

    Genome-wide selection scans integrated with association mapping reveal mechanisms of physiological adaptation across a salinity gradient in killifish

    Reid S. Brennan, Timothy M. Healy, Heather J. Bryant, et al.



    Selected by Andy Turko

    From Armament to Ornament: Performance Trade-Offs in the Sexual Weaponry of Neotropical Electric Fishes

    Kory M. Evans, Maxwell J. Bernt, Matthew A. Kolmann, et al.



    Selected by Cassandra Donatelli
    Close