BIAFLOWS: A collaborative framework to benchmark bioimage analysis workflows
Preprint posted on November 29, 2019 https://www.biorxiv.org/content/10.1101/707489v2
Article now published in Patterns at http://dx.doi.org/10.1016/j.patter.2020.100040
BIAFLOWS: a tool for sharing, deploying, and comparing bioimage analysis workflows. Tackling reproducible and open image analysis.
Selected by Mariana De NizCategories: bioinformatics, cell biology, microbiology, plant biology, systems biology
Background
With constant and exciting advances in microscopy, biomedical and life scientists can acquire images of increasing size and complexity. Thus, manual processing has become limiting, and the use of computational methods allows the extraction of quantitative information from these datasets in an accurate and high-throughput manner.
Automated image analysis is key to extract quantitative information from microscopy images. However, an important limitation faced currently in this time of transition, is that the methods of image analysis developed and used across labs, can be complex to apply and relatively difficult to describe using written protocols. This threatens their reproducibility and calls for sharing the methods used to produce scientific data from images directly as software implementations (workflows). On top of this, most published image analysis methods are not easily searchable and test images as well as the parameters used at every step of a workflow are often not provided, making it difficult to understand the baseline for valid results.
One solution that has arisen to address this issue, is a movement towards making scientific datasets publicly available, through open data initiatives and web-based software. While an improvement, this still does not represent a unified platform for image analysis which allows access to, and comparison of bioimage analysis (BIA) workflows.
In an aim to tackle this issue, the work by Rubens et al (1) (COST Action CA15124) presents BIAFLOWS (Figure 1), a web-based framework to unify, make available, and benchmark automated image analysis workflows.

Key findings and developments
General developments: BIAFLOWS
- Within the Network of European Bioimage Analysis (NEUBIAS), an important body of work focuses on ensuring a better access to and assessment of existing BIA software. BIAFLOWS is presented as a web-based platform to benchmark bioimage analysis workflows on publicly shared annotated multidimensional imaging data. That is, to assess the accuracy of the workflows by comparing them to a ground truth reference (e.g. human annotation) and quantifying the discrepancy by computing relevant performance metrics. The aim is:
- To help integrate and compare image analysis methods, hence enforcing highest quality standards, and stimulating the continuous improvements of BIA methods.
- To offer a complete framework that allows a) handling multidimensional annotated images, b) integrating BIA workflows (standalone or running on BIA platforms), c) remotely visualizing images and workflow results and d) assessing workflows performance from widely accepted performance metrics.
- All content including images, workflows, ground-truth annotations, workflow results and benchmark metrics, can be browsed and interactively explored through the web interface.
- An online instance of BIAFLOWS is available from this URL: https://biaflows.neubias.org/ (user: guest, password: guest)
- BIAFLOWS can also be installed locally as a local image management and image analysis platform and all content can be easily migrated between existing instances.
Specific features for users and developers
- BIAFLOWS is based on Cytomine (2), an open source web platform developed for the collaborative annotation of high resolution biomedical images.
- The features that were extended or developed for BIAFLOWS are:
- Support for the upload of microscopy multidimensional images in OME-TIFF format, as well as their remote visualization in a viewer enabling navigation through image slices, adjusting contrast and toggling annotation layers.
- Support for the remote execution of BIA workflows by encapsulating the workflows and their original software environment in Docker images.
- Monitoring trusted spaces to automatically pull new or updated workflows whenever a new release is triggered from their associated source code repository.
- Version the workflows and make them permanently accessible from the system.
- Make the workflows ready for any type of computational resources (including high performance computing).
- Migration tools to transfer content between existing BIAFLOWS instances.
- BIAFLOWS is versatile: the workflows currently integrated to the online instance consist of a mixture of standalone software and scripts (ImageJ/Fiji, ICY, CellProfiler, Vaa3D, Ilastik, Python and Jupyter notebooks).
- Interested developers can package their workflows and make them available for benchmarking from https://biaflows.neubias.org/.
- BIAFLOWS workflow can also be used to process local images on a machine running Docker, independently of any BIAFLOW server.
What I like about this paper
I first heard of BIAFLOWS at the NEUBIAS (COST Action CA15124) Image analysis training schools in Porto, Portugal, when its concept and details were presented by author Sébastien Tosi. It caught my attention immediately, as it represents a much expected tool for users and developers of bioimage analysis workflows. While Sébastien Tosi discussed the current challenges of sharing BIA workflows, I could identify all very well: through my career, it has been difficult to search for, find, share, compare, and validate workflows. Equally, most imaging datasets from published work, are hardly available. A tool that compiles the many components necessary for BIA workflows is a huge step forward into openness and reproducibility into the important step of image analysis- an ever growing field of increasing importance.
Open questions
- Note: all questions with answers from authors are at the bottom of this page.
- The idea of BIAFLOWS is exciting. How did you come to create this, and how do you see it developing further as the fields grow (both of bioimage analysis and of microscopy)?
- Under the lines of the previous question- the creators of several open source tools offer training to involve and benefit as many scientists as possible. For those scientists who are new to imaging and image analysis (eg young PhD students, or scientists later in their careers beginning in either microscopy or image analysis) how will you tailor this excellent tool so that users at all levels of knowledge can benefit from it?
- How do you envisage integrating BIAFLOWS in the publication process? For instance, at present, most journals do not require depositing original imaging datasets from which quantifications or analyses are presented. In the past this was the case for bioinformatics analyses until open distribution/access became a norm. Do you envisage that BIAFLOWS could help make this possible? A repository of images that helps reproducibility and transparency in every way?
- In your paper you mention that in many cases of image analysis, imaging datasets from microscopy are a minority compared to medical imaging. In your opinion, what makes this difference between medical and biomedical-oriented imaging?
- You specify the features that BIAFLOWS currently possesses. We know that constantly, imaging methods, microscopes, and image analysis tools are being developed. With such rate, what is the workforce that will keep BIAFLOWS updated, and the results of comparisons of different workflows published and available? And do you see a need for increased funding and developers’ involvement worldwide to integrate into this effort?
References
- Rubens et al (2019). BIAFLOWS: A collaborative network framework to benchmark bioimage analysis workflows, bioRxiv, doi: 10.1101/707489
- Maree et al (2016). Collaborative analysis of multi-gigapixel imaging data with Cytomine, Bioinformatics, 32 (9): 1395-401.
Acknowledgements
I thank Sébastien Tosi and Raphael Maree for their time discussing their work, and their input on this highlight. I also thank Mate Palfy for his helpful comments. Finally, for this particular highlight, I thank the NEUBIAS (COST Action CA15124) Image analysis training school in Porto, Portugal, which allowed me to become acquainted with this interesting and promising work in the first place.
Posted on: 29th November 2019 , updated on: 30th November 2019
doi: https://doi.org/10.1242/prelights.15323
Read preprintSign up to customise the site to your preferences and to receive alerts
Register hereAlso in the bioinformatics category:
Dissecting Mammalian Spermatogenesis Using Spatial Transcriptomics
Selected by | Martin Estermann |
The hourglass model of evolutionary conservation during embryogenesis extends to developmental enhancers with signatures of positive selection
Selected by | Irepan Salvador-Martinez |
1
The mutational landscape of human somatic and germline cells
Selected by | Kerryn Elliott |
Also in the cell biology category:
Also in the microbiology category:
Gut microbiome features are associated with sepsis onset and outcomes
Selected by | Josie Gibson |
AMP-activated protein kinase: A key enzyme to manage nutritional stress responses in parasites with complex life cycles
Selected by | Mariana De Niz |
1
A mating-induced reproductive gene promotes Anopheles tolerance to Plasmodium falciparum infection
Selected by | Mariana De Niz |
1
Also in the plant biology category:
Creating Clear and Informative Image-based Figures for Scientific Publications
Selected by | Mariana De Niz |
1
Major components in the KARRIKIN INSENSITIVE2-ligand signaling pathway are conserved in the liverwort, Marchantia polymorpha
AND
The Physcomitrium (Physcomitrella) patens PpKAI2L receptors for strigolactones and related compounds highlight MAX2 dependent and independent pathways
Selected by | Facundo Romani |
Hydration-dependent phase separation of a prion-like protein regulates seed germination during water stress
Selected by | Martin Balcerowicz |
Also in the systems biology category:
Downstream of gasdermin D cleavage, a Ragulator-Rag-mTORC1 pathway promotes pore formation and pyroptosis
Selected by | Connor Rosen |
Bromodomain Inhibition Blocks Inflammation-Induced Cardiac Dysfunction and SARS-CoV2 Infection in Pre-Clinical Models
Selected by | Alexander Ward, Osvaldo Contreras |
1
Generative modeling of single-cell population time series for inferring cell differentiation landscapes
Selected by | Yen-Chung Chen |
preListsbioinformatics category:
in theSingle Cell Biology 2020
A list of preprints mentioned at the Wellcome Genome Campus Single Cell Biology 2020 meeting.
List by | Alex Eve |
Antimicrobials: Discovery, clinical use, and development of resistance
Preprints that describe the discovery of new antimicrobials and any improvements made regarding their clinical use. Includes preprints that detail the factors affecting antimicrobial selection and the development of antimicrobial resistance.
List by | Zhang-He Goh |
Also in the cell biology category:
FENS 2020
A collection of preprints presented during the virtual meeting of the Federation of European Neuroscience Societies (FENS) in 2020
List by | Ana Dorrego-Rivas |
Planar Cell Polarity – PCP
This preList contains preprints about the latest findings on Planar Cell Polarity (PCP) in various model organisms at the molecular, cellular and tissue levels.
List by | Ana Dorrego-Rivas |
BioMalPar XVI: Biology and Pathology of the Malaria Parasite
[under construction] Preprints presented at the (fully virtual) EMBL BioMalPar XVI, 17-18 May 2020 #emblmalaria
List by | Dey Lab, Samantha Seah |
1
Cell Polarity
Recent research from the field of cell polarity is summarized in this list of preprints. It comprises of studies focusing on various forms of cell polarity ranging from epithelial polarity, planar cell polarity to front-to-rear polarity.
List by | Yamini Ravichandran |
TAGC 2020
Preprints recently presented at the virtual Allied Genetics Conference, April 22-26, 2020. #TAGC20
List by | Maiko Kitaoka, Madhuja Samaddar, Miguel V. Almeida, Sejal Davla, Jennifer Ann Black, Dey Lab |
3D Gastruloids
A curated list of preprints related to Gastruloids (in vitro models of early development obtained by 3D aggregation of embryonic cells). Preprint missing? Don't hesitate to let us know.
List by | Paul Gerald L. Sanchez and Stefano Vianello |
ECFG15 – Fungal biology
Preprints presented at 15th European Conference on Fungal Genetics 17-20 February 2020 Rome
List by | Hiral Shah |
ASCB EMBO Annual Meeting 2019
A collection of preprints presented at the 2019 ASCB EMBO Meeting in Washington, DC (December 7-11)
List by | Madhuja Samaddar, Ramona Jühlen, Amanda Haage, Laura McCormick, Maiko Kitaoka |
EMBL Seeing is Believing – Imaging the Molecular Processes of Life
Preprints discussed at the 2019 edition of Seeing is Believing, at EMBL Heidelberg from the 9th-12th October 2019
List by | Dey Lab |
Autophagy
Preprints on autophagy and lysosomal degradation and its role in neurodegeneration and disease. Includes molecular mechanisms, upstream signalling and regulation as well as studies on pharmaceutical interventions to upregulate the process.
List by | Sandra Malmgren Hill |
Lung Disease and Regeneration
This preprint list compiles highlights from the field of lung biology.
List by | Rob Hynds |
Cellular metabolism
A curated list of preprints related to cellular metabolism at Biorxiv by Pablo Ranea Robles from the Prelights community. Special interest on lipid metabolism, peroxisomes and mitochondria.
List by | Pablo Ranea Robles |
BSCB/BSDB Annual Meeting 2019
Preprints presented at the BSCB/BSDB Annual Meeting 2019
List by | Dey Lab |
MitoList
This list of preprints is focused on work expanding our knowledge on mitochondria in any organism, tissue or cell type, from the normal biology to the pathology.
List by | Sandra Franco Iborra |
Biophysical Society Annual Meeting 2019
Few of the preprints that were discussed in the recent BPS annual meeting at Baltimore, USA
List by | Joseph Jose Thottacherry |
ASCB/EMBO Annual Meeting 2018
This list relates to preprints that were discussed at the recent ASCB conference.
List by | Dey Lab, Amanda Haage |
Also in the microbiology category:
BioMalPar XVI: Biology and Pathology of the Malaria Parasite
[under construction] Preprints presented at the (fully virtual) EMBL BioMalPar XVI, 17-18 May 2020 #emblmalaria
List by | Dey Lab, Samantha Seah |
1
ECFG15 – Fungal biology
Preprints presented at 15th European Conference on Fungal Genetics 17-20 February 2020 Rome
List by | Hiral Shah |
EMBL Seeing is Believing – Imaging the Molecular Processes of Life
Preprints discussed at the 2019 edition of Seeing is Believing, at EMBL Heidelberg from the 9th-12th October 2019
List by | Dey Lab |
Antimicrobials: Discovery, clinical use, and development of resistance
Preprints that describe the discovery of new antimicrobials and any improvements made regarding their clinical use. Includes preprints that detail the factors affecting antimicrobial selection and the development of antimicrobial resistance.
List by | Zhang-He Goh |
Also in the plant biology category:
Also in the systems biology category:
Single Cell Biology 2020
A list of preprints mentioned at the Wellcome Genome Campus Single Cell Biology 2020 meeting.
List by | Alex Eve |
ASCB EMBO Annual Meeting 2019
A collection of preprints presented at the 2019 ASCB EMBO Meeting in Washington, DC (December 7-11)
List by | Madhuja Samaddar, Ramona Jühlen, Amanda Haage, Laura McCormick, Maiko Kitaoka |
EMBL Seeing is Believing – Imaging the Molecular Processes of Life
Preprints discussed at the 2019 edition of Seeing is Believing, at EMBL Heidelberg from the 9th-12th October 2019
List by | Dey Lab |
Pattern formation during development
The aim of this preList is to integrate results about the mechanisms that govern patterning during development, from genes implicated in the processes to theoritical models of pattern formation in nature.
List by | Alexa Sadier |