Deep learning-enhanced light-field imaging with continuous validation
Preprint posted on 31 July 2020 https://www.biorxiv.org/content/10.1101/2020.07.30.228924v1
Article now published in Nature Methods at http://dx.doi.org/10.1038/s41592-021-01136-0
Capturing highly dynamic physiological processes happening on milli-second time scales across large areas in living organisms requires imaging methods capable of such resolution. An attractive candidate for high-speed 3D image acquisition is light-field microscopy (LFM), which has already opened new avenues in the fields of neurobiology and cardiovascular dynamics. While some technical hindrances of this tool have been overcome since its conception, the widespread use of LFM has been hampered by a computationally demanding, iterative image reconstruction process that requires a complex computational infrastructure and adequate data management. Multiple algorithms derived from deep learning and convolutional neural networks have recently been proposed to replace iterative deconvolution procedures, and offer new methods for deblurring, denoising and super-resolution. While many of these methods have excellent performance in various biologically relevant settings, many are not optimal for dynamic imaging with LFM given the complexity of dynamic processes in small animals. In their work, Wagner and Beuttenmueller et al (1) present a novel framework consisting of a hybrid light-field light-sheet microscope (HyLFM) and deep-learning-based volume reconstruction. In it, single light-sheet acquisitions continuously serve as training data and validation for the convolutional neural network (termed HyLFM-Net) reconstructing the LFM volume.
Key findings and developments
A simultaneous selective-plane illumination microscopy (SPIM) modality was added into a standard LFM microscope, allowing the generation of high-resolution ground truth images of single planes for validation, training, and refinement for the convolutional neural network. Training can be done on static sample volumes, or dynamically from a single SPIM plane going through the volume during 3D image acquisition. An automated image processing pipeline ensures that LFM and SPIM volumes are co-registered in a common reference volume and coordinate system with high precision. This is important for convolutional neural network training and validation, and the systems’ ability to acquire 2D and 3D training data is key for reliable convolutional neural network reconstructions, including data never seen in training. Altogether, the HyLFM-Net is trained on pairs of SPIM-LFM images.
To evaluate the performance of the HyLFM system, the authors imaged sub-diffraction sized fluorescent beads suspended in agarose, and quantified the improvement in spatial resolution and image quality compared to standard iterative light-field deconvolution. They concluded that HyLFM-Net correctly inferred the 3D imaging volume from the raw light-field data, with better resolution than that which could be obtained by light field deconvolution, and without artifacts commonly found in light field deconvolution. The authors point out the importance of training on diverse datasets, to avoid biases in performance.
As proof of principle, the authors explore the capabilities of the HyLFM system by imaging the dynamics of a hatchling medaka fish heart in vivo, to demonstrate the capability of the system to correctly capture dynamic cellular movements in 3D. The HyLFM-Net allowed acquiring high image quality metrics compared to SPIM, and allowed 3D volume inference at up to 18Hz, with at least 1000-fold reconstruction speed compared to light field deconvolution. The authors note that the network trained on dynamically acquired SPIM single planes performed equally well or better than the network trained on fully static volumes.
The authors also tested the HyLFM system on transgenic larval zebrafish brains expressing calcium indicators, to monitor neural activity. The ground truth data enabled the HyLFM system to faithfully learn and infer structural, as well as intensity-based information. The authors conclude HyLFM is thus an attractive method for visualizing neural activity.
Altogether, the new system allows reconstructing light-field volumes at sub-second rates, eliminating the main computational hindrances for light field imaging. Moreover, the system enables acquiring appropriate training data simultaneously and on-the-fly, while allowing continuous validation and fine-tuning. The network over time learns on the actual experimental data, rather than requiring pre-acquisition of training images in separate microscopes, solving the hindrance of transferability.
What I like about this preprint
I like that the authors address a hugely important technical gap in a fast-advancing microscopy area. It has not been uncommon over the past few decades that major advances in microscopy tools occur, and the methods for image analysis stay behind. This at some point becomes a limiting factor in itself. I think that tools addressing this gap to overcome those limitations are key, and ultimately allow the different microscopy tools to be used to their full potential, and become widespread.
1. Wagner N, Beuttenmueller F, et al, Deep learning-enganced light-field imaging with continuous validation, bioRxiv, 2020.
Posted on: 4 November 2020Read preprint
Also in the bioengineering category:
Actin-driven protrusions generate rapid long-range membrane tension propagation in cells
|Selected by||Nicolaes Hyun-Kee Min|
PHYTOMap: Multiplexed single-cell 3D spatial gene expression analysis in plant tissue
|Selected by||Gwendolyn K. Kirschner, Marc Somssich|
Quantification of cell penetrating peptide mediated delivery of proteins in plant leaves
|Selected by||Gwendolyn K. Kirschner|
Also in the physiology category:
A single cell spatial temporal atlas of skeletal muscle reveals cellular neighborhoods that orchestrate regeneration and become disrupted in aging
|Selected by||Jessica Chevallier|
All-optical visualization of specific molecules in the ultrastructural context of brain tissue
|Selected by||Nadja Hümpfer|
Local IL-17 orchestrates skin aging
|Selected by||Andrea Irazoki|
preListsbioengineering category:in the
EMBL Synthetic Morphogenesis: From Gene Circuits to Tissue Architecture (2021)
A list of preprints mentioned at the #EESmorphoG virtual meeting in 2021.
|List by||Alex Eve|
A curated list of preprints related to Gastruloids (in vitro models of early development obtained by 3D aggregation of embryonic cells). Updated until July 2021.
|List by||Paul Gerald L. Sanchez and Stefano Vianello|
ASCB EMBO Annual Meeting 2019
A collection of preprints presented at the 2019 ASCB EMBO Meeting in Washington, DC (December 7-11)
|List by||Madhuja Samaddar et al.|
EMBL Seeing is Believing – Imaging the Molecular Processes of Life
Preprints discussed at the 2019 edition of Seeing is Believing, at EMBL Heidelberg from the 9th-12th October 2019
|List by||Dey Lab|
Lung Disease and Regeneration
This preprint list compiles highlights from the field of lung biology.
|List by||Rob Hynds|
Advances in microscopy
This preList highlights exciting unpublished preprint articles describing advances in microscopy with a focus on light-sheet microscopy.
|List by||Stephan Daetwyler|
Also in the physiology category:
The advances in fibroblast biology preList explores the recent discoveries and preprints of the fibroblast world. Get ready to immerse yourself with this list created for fibroblasts aficionados and lovers, and beyond. Here, my goal is to include preprints of fibroblast biology, heterogeneity, fate, extracellular matrix, behavior, topography, single-cell atlases, spatial transcriptomics, and their matrix!
|List by||Osvaldo Contreras|
A collection of preprints presented during the virtual meeting of the Federation of European Neuroscience Societies (FENS) in 2020
|List by||Ana Dorrego-Rivas|
Preprints recently presented at the virtual Allied Genetics Conference, April 22-26, 2020. #TAGC20
|List by||Maiko Kitaoka et al.|
Preprints on autophagy and lysosomal degradation and its role in neurodegeneration and disease. Includes molecular mechanisms, upstream signalling and regulation as well as studies on pharmaceutical interventions to upregulate the process.
|List by||Sandra Malmgren Hill|
A curated list of preprints related to cellular metabolism at Biorxiv by Pablo Ranea Robles from the Prelights community. Special interest on lipid metabolism, peroxisomes and mitochondria.
|List by||Pablo Ranea Robles|