BIAFLOWS: A collaborative framework to benchmark bioimage analysis workflows

Ulysse Rubens, Romain Mormont, Volker Baecker, Gino Michiels, Lassi Paavolainen, Graeme Ball, Devrim Unay, Benjamin Pavie, Anatole Chessel, Leandro A. Scholz, Martin Maska, Renaud Hoyoux, Remy Vandaele, Stefan G. Stanciu, Ofra Golani, Natasa Sladoje, Perrine Paul-Gilloteaux, Raphael Maree, Sebastien Tosi

Posted on: 29 November 2019 , updated on: 30 November 2019

Preprint posted on 29 November 2019

Article now published in Patterns at http://dx.doi.org/10.1016/j.patter.2020.100040

BIAFLOWS: a tool for sharing, deploying, and comparing bioimage analysis workflows. Tackling reproducible and open image analysis.

Selected by Mariana De Niz

Categories: bioinformatics, cell biology, microbiology, plant biology, systems biology

Background

With constant and exciting advances in microscopy, biomedical and life scientists can acquire images of increasing size and complexity. Thus, manual processing has become limiting, and the use of computational methods allows the extraction of quantitative information from these datasets in an accurate and high-throughput manner.

Automated image analysis is key to extract quantitative information from microscopy images. However, an important limitation faced currently in this time of transition, is that the methods of image analysis developed and used across labs, can be complex to apply and relatively difficult to describe using written protocols. This threatens their reproducibility and calls for sharing the methods used to produce scientific data from images directly as software implementations (workflows). On top of this, most published image analysis methods are not easily searchable and test images as well as the parameters used at every step of a workflow are often not provided, making it difficult to understand the baseline for valid results.

One solution that has arisen to address this issue, is a movement towards making scientific datasets publicly available, through open data initiatives and web-based software. While an improvement, this still does not represent a unified platform for image analysis which allows access to, and comparison of bioimage analysis (BIA) workflows.

In an aim to tackle this issue, the work by Rubens et al (1) (COST Action CA15124) presents BIAFLOWS (Figure 1), a web-based framework to unify, make available, and benchmark automated image analysis workflows.

Figure 1. Web interface of BIAFLOWS, allowing users to browse datasets based on a bioimage analysis problem, and to select a workflow that allows them to process images and adjust relevant parameters. Upon completion, the results of the workflow can be visualized, and the associated benchmark metrics reported.

Key findings and developments

General developments: BIAFLOWS

Within the Network of European Bioimage Analysis (NEUBIAS), an important body of work focuses on ensuring a better access to and assessment of existing BIA software. BIAFLOWS is presented as a web-based platform to benchmark bioimage analysis workflows on publicly shared annotated multidimensional imaging data. That is, to assess the accuracy of the workflows by comparing them to a ground truth reference (e.g. human annotation) and quantifying the discrepancy by computing relevant performance metrics. The aim is:
- To help integrate and compare image analysis methods, hence enforcing highest quality standards, and stimulating the continuous improvements of BIA methods.
- To offer a complete framework that allows a) handling multidimensional annotated images, b) integrating BIA workflows (standalone or running on BIA platforms), c) remotely visualizing images and workflow results and d) assessing workflows performance from widely accepted performance metrics.

All content including images, workflows, ground-truth annotations, workflow results and benchmark metrics, can be browsed and interactively explored through the web interface.
An online instance of BIAFLOWS is available from this URL: https://biaflows.neubias.org/ (user: guest, password: guest)
BIAFLOWS can also be installed locally as a local image management and image analysis platform and all content can be easily migrated between existing instances.

Specific features for users and developers

BIAFLOWS is based on Cytomine (2), an open source web platform developed for the collaborative annotation of high resolution biomedical images.
The features that were extended or developed for BIAFLOWS are:
- Support for the upload of microscopy multidimensional images in OME-TIFF format, as well as their remote visualization in a viewer enabling navigation through image slices, adjusting contrast and toggling annotation layers.
- Support for the remote execution of BIA workflows by encapsulating the workflows and their original software environment in Docker images.
- Monitoring trusted spaces to automatically pull new or updated workflows whenever a new release is triggered from their associated source code repository.
- Version the workflows and make them permanently accessible from the system.
- Make the workflows ready for any type of computational resources (including high performance computing).
- Migration tools to transfer content between existing BIAFLOWS instances.

BIAFLOWS is versatile: the workflows currently integrated to the online instance consist of a mixture of standalone software and scripts (ImageJ/Fiji, ICY, CellProfiler, Vaa3D, Ilastik, Python and Jupyter notebooks).
Interested developers can package their workflows and make them available for benchmarking from https://biaflows.neubias.org/.
BIAFLOWS workflow can also be used to process local images on a machine running Docker, independently of any BIAFLOW server.

What I like about this paper

I first heard of BIAFLOWS at the NEUBIAS (COST Action CA15124) Image analysis training schools in Porto, Portugal, when its concept and details were presented by author Sébastien Tosi. It caught my attention immediately, as it represents a much expected tool for users and developers of bioimage analysis workflows. While Sébastien Tosi discussed the current challenges of sharing BIA workflows, I could identify all very well: through my career, it has been difficult to search for, find, share, compare, and validate workflows. Equally, most imaging datasets from published work, are hardly available. A tool that compiles the many components necessary for BIA workflows is a huge step forward into openness and reproducibility into the important step of image analysis- an ever growing field of increasing importance.

Open questions

Note: all questions with answers from authors are at the bottom of this page.

The idea of BIAFLOWS is exciting. How did you come to create this, and how do you see it developing further as the fields grow (both of bioimage analysis and of microscopy)?
Under the lines of the previous question- the creators of several open source tools offer training to involve and benefit as many scientists as possible. For those scientists who are new to imaging and image analysis (eg young PhD students, or scientists later in their careers beginning in either microscopy or image analysis) how will you tailor this excellent tool so that users at all levels of knowledge can benefit from it?
How do you envisage integrating BIAFLOWS in the publication process? For instance, at present, most journals do not require depositing original imaging datasets from which quantifications or analyses are presented. In the past this was the case for bioinformatics analyses until open distribution/access became a norm. Do you envisage that BIAFLOWS could help make this possible? A repository of images that helps reproducibility and transparency in every way?
In your paper you mention that in many cases of image analysis, imaging datasets from microscopy are a minority compared to medical imaging. In your opinion, what makes this difference between medical and biomedical-oriented imaging?
You specify the features that BIAFLOWS currently possesses. We know that constantly, imaging methods, microscopes, and image analysis tools are being developed. With such rate, what is the workforce that will keep BIAFLOWS updated, and the results of comparisons of different workflows published and available? And do you see a need for increased funding and developers’ involvement worldwide to integrate into this effort?

References

Rubens et al (2019). BIAFLOWS: A collaborative network framework to benchmark bioimage analysis workflows, bioRxiv, doi: 10.1101/707489
Maree et al (2016). Collaborative analysis of multi-gigapixel imaging data with Cytomine, Bioinformatics, 32 (9): 1395-401.

Acknowledgements

I thank Sébastien Tosi and Raphael Maree for their time discussing their work, and their input on this highlight. I also thank Mate Palfy for his helpful comments. Finally, for this particular highlight, I thank the NEUBIAS (COST Action CA15124) Image analysis training school in Porto, Portugal, which allowed me to become acquainted with this interesting and promising work in the first place.

Tags: image analysis, imaging, microscopy, workflows

doi: https://doi.org/10.1242/prelights.15323

Read preprint

(No Ratings Yet)

Author's response

Sebastien Tosi and Raphael Marée shared

Open questions:

1. The idea of BIAFLOWS is exciting. How did you come to create this, and how do you see it developing further as the fields grow (both of bioimage analysis and of microscopy)?

One of the first needs we identified within NEUBIAS was to publicly index most existing bioimage analysis tools and workflows so that both, biologists and image analysis experts could efficiently find them. Toward this goal, a workgroup designed a web platform enabling to efficiently tag content and unify semantic fields: the Bioimage Informatics Search Engine (BISE, https:\\biii.eu). Now, when you do image analysis, you quickly learn that there are often many ways to tackle the same problem, but that they are not all equal! And obviously, if you get multiple hits while searching for a suitable image analysis method, you want to be taken by the hand towards the best pick. We hence naturally got interested in describing the internals of the methods and comparing their performance. BISE workgroup worked on the first item. For the second item, we needed an infrastructure to store images together with ground truth annotations (e.g. human object annotations), process the images by BIA workflows, assess performance, and make all this content viewable online! We also needed the content itself: a pool of annotated images recapitulating important BIA problems and associated workflows. I guess you quickly understand that it is a complex task and that it had to be carefully planned so that every component could smoothly interoperate. BIAFLOWS is the results of this work and we are now proud to release it as an open source solution. From this point on, our main effort will be to increase the content, in terms of problems covered, workflows and annotated datasets. This will be mainly achieved by opening calls for contribution and dissemination. Only when we reach a critical mass for all these items, will the benchmarking results become fully meaningful. This should in turn increase BIAFLOWS impact, further adoption and content…

2.Under the lines of the previous question- the creators of several open source tools offer training to involve and benefit as many scientists as possible. For those scientists who are new to imaging and image analysis (eg young PhD students, or scientists later in their careers beginning in either microscopy or image analysis) how will you tailor this excellent tool so that users at all levels of knowledge can benefit from it?

Actually, in its current form anybody can browse the platform and should enjoy the experience! It is as simple as selecting a BIA problem from the ones illustrated, opening the associated images and workflow results, and spotting where a given workflow stands in the ranking. To guide users, video tutorials are provided from https://biaflows.neubias.org/ and a detailed documentation is available from https://neubias-wg5.github.io/ . Workflows coming from any platform, as long as it fulfils some minimal openness requirements can be integrated, so we are really inclusive. Now, again, an enabling factor to truly turn it into a reference asset for the communities involved is that on one hand developers adopt the platform massively as a way to compare and promote their software, and on the other hand that researchers contribute human annotated microscopy datasets to consolidate the benchmarking. Specifically, we need to set the basis of representative datasets for common sample preparation and imaging protocols as a community and consensual effort. So in other words at this point the question is not so much how BIAFLOWS can benefit to you, but how could you contribute to BIAFLOWS to make it useful for all?

3.How do you envisage integrating BIAFLOWS in the publication process? For instance, at present, most journals do not require depositing original imaging datasets from which quantifications or analyses are presented. In the past this was the case for bioinformatics analyses until open distribution/access became a norm. Do you envisage that BIAFLOWS could help make this possible? A repository of images that helps reproducibility and transparency in every way?

BIAFLOWS could definitely be used as a repository of images “living together” with the workflows that were used to process them and produce scientific results in the context of a research project. This is actually the direction open science should pursue. There is no current design limitation to meet this goal but, of course, the storage and computational infrastructure BIAFLOWS relies on has to be accordingly dimensioned and maintained. This obviously requires a strong commitment and adequate funding from the publishers or agencies promoting open science. In our view, it is totally justified since adopting a unique public platform, standardizing data and workflows formats, and ensuring reproducibility would make the whole process truly open, sustainable, and simpler for all.

4.In your paper you mention that in many cases of image analysis, imaging datasets from microscopy are a minority compared to medical imaging. In your opinion, what makes this difference between medical and biomedical-oriented imaging?

I think it is both a question of field maturity and standardization. Biomedical image analysis predates microscopy bioimage analysis and it is more standardized in terms of imaging techniques (how to prepare and image samples) and image analysis (what to measure). This makes it easier to collect representative sample datasets to illustrate common BIA tasks. Fluorescence microscopy, just to mention it, is a world on its own… just have a look at the number of acronyms and recent developments in the field of lightsheet microscopy and super-resolution! Not to mention the infinite combinations of sample preparation techniques including protein fluorescence tagging, antibody staining, sample clearing… Still, I believe that at least BIA workflows extracting quantitative information from classical microscopy assays deserve to be scientifically compared since they are so widely used. And we are working at it!

5. You specify the features that BIAFLOWS currently possesses. We know that constantly, imaging methods, microscopes, and image analysis tools are being developed. With such rate, what is the workforce that will keep BIAFLOWS updated, and the results of comparisons of different workflows published and available? And do you see a need for increased funding and developers’ involvement worldwide to integrate into this effort?

The architecture of BIAFLOWS is extensible by design, which means that supporting new image formats, image analysis ecosystems and BIA problem types is straightforward. This of course requires some workforce and developments but, as mentioned previously, the main driving force to bring BIAFLOWS to the next level is its adoption by developers as a mean to reproducibly package their workflows, simplify their deployment and compare their performance. For that reason, our current effort is now mainly about disseminating BIAFLOWS. Besides opening calls and promoting the platform in the different communities involved, a way to speed up this process is to reach out to the organizers of bioimage analysis challenges. The match is clear since they would directly benefit from automating benchmarking and opening the results to the community and we would benefit from the content they create (workflows and annotated datasets). Going in the same direction, there is nowadays a huge interest in making annotated datasets readily available since they are the basis to train deep learning networks, a class of algorithms that is quickly revolutionizing the field of bioimage analysis.

Have your say Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Sign up to customise the site to your preferences and to receive alerts

Also in the bioinformatics category:

STEAMBOAT: Attention-based multiscale delineation of cellular interactions in tissues

Shaoheng Liang, Junjie Tang, Guanghan Wang, et al.

Selected by 25 April 2025

Benjamin Dominik Maier

Discussion

Decoding the Molecular Language of Proteins with Evolla

Xibin Zhou, Chenchen Han, Yingqi Zhang, et al.

Selected by 14 March 2025

Jawdat Sandakly

Tidyplots empowers life scientists with easy code-based data visualization

Jan Broder Engler

Selected by 21 February 2025

Felipe Del Valle Batalla

Also in the cell biology category:

Mechanical cues organize planar cell polarity during vertebrate morphogenesis and embryonic wound repair

Chih-Wen Chu, Satheeja Velayudhan, Jakob H. Schauser, et al.

Selected by 02 July 2025

Aline Grata

The essential role of connective-tissue cells during axolotl limb regeneration

Damián García-García, Dunja Knapp, Minjoo Kim, et al.

Selected by 26 June 2025

Dhanuush Balakannan

Succinate Dehydrogenase loss causes cascading metabolic effects that impair pyrimidine biosynthesis

Madeleine L. Hart, Kristian Davidsen, Serwah Danquah, et al.

Selected by 26 May 2025

Hannah Pletcher

Also in the microbiology category:

Direct cell-to-cell transmission of retrotransposons

Maya Voichek, Andreas Bernhard, Maria Novatchkova, et al.

Selected by 21 May 2025

Ally Simpson et al.

Discussion

A prophage-encoded anti-phage defense system that prevents phage DNA packaging by targeting the terminase complex

Gil Azulay, Nadejda Sigal, Keren Tzohar Rabinovich, et al.

Selected by 19 May 2025

Briana Guzman et al.

Discussion

Green synthesized silver nanoparticles from Moringa: Potential for preventative treatment of SARS-CoV-2 contaminated water

Adebayo J. Bello, Omorilewa B. Ebunoluwa, Rukayat O. Ayorinde, et al.

Selected by 14 November 2024

Safieh Shah, Benjamin Dominik Maier

Discussion

Also in the plant biology category:

Conservation and divergence of regulatory architecture in nitrate-responsive plant gene circuits

C Bian, GS Demirer, MT Oz, et al.

Selected by 14 March 2025

Jeny Jose

New evidence for the presence and function of phosphoinositides (PPIs) in the chloroplast

Mastoureh Sedaghatmehr, Frieda Rößler, Alexander P. Hertle

Selected by 12 December 2024

Shreya Pramanik

Plasmodesmal closure elicits stress responses

Estee E. Tee, Andrew Breakspear, Diana Papp, et al.

Selected by 12 June 2024

Yueh Cho

Also in the systems biology category:

Modular control of time and space during vertebrate axis segmentation

Ali Seleit, Ian Brettell, Tomas Fitzgerald, et al.

AND

Natural genetic variation quantitatively regulates heart rate and dimension

Jakob Gierten, Bettina Welz, Tomas Fitzgerald, et al.

Selected by 24 June 2024

Girish Kale, Jennifer Ann Black

Expressive modeling and fast simulation for dynamic compartments

Till Köster, Philipp Henning, Tom Warnke, et al.

Selected by 18 April 2024

Benjamin Dominik Maier

Clusters of lineage-specific genes are anchored by ZNF274 in repressive perinucleolar compartments

Martina Begnis, Julien Duc, Sandra Offner, et al.

Selected by 10 April 2024

Silvia Carvalho

preLists in the bioinformatics category:

Keystone Symposium – Metabolic and Nutritional Control of Development and Cell Fate

This preList contains preprints discussed during the Metabolic and Nutritional Control of Development and Cell Fate Keystone Symposia. This conference was organized by Lydia Finley and Ralph J. DeBerardinis and held in the Wylie Center and Tupper Manor at Endicott College, Beverly, MA, United States from May 7th to 9th 2025. This meeting marked the first in-person gathering of leading researchers exploring how metabolism influences development, including processes like cell fate, tissue patterning, and organ function, through nutrient availability and metabolic regulation. By integrating modern metabolic tools with genetic and epidemiological insights across model organisms, this event highlighted key mechanisms and identified open questions to advance the emerging field of developmental metabolism.

BIAFLOWS: A collaborative framework to benchmark bioimage analysis workflows

Background

Key findings and developments

General developments: BIAFLOWS

Specific features for users and developers

What I like about this paper

Open questions

References

Acknowledgements

Share this:

Have your say Cancel reply

Sign up to customise the site to your preferences and to receive alerts

Also in the bioinformatics category:

STEAMBOAT: Attention-based multiscale delineation of cellular interactions in tissues

Decoding the Molecular Language of Proteins with Evolla

Tidyplots empowers life scientists with easy code-based data visualization

Also in the cell biology category:

Mechanical cues organize planar cell polarity during vertebrate morphogenesis and embryonic wound repair

The essential role of connective-tissue cells during axolotl limb regeneration

Succinate Dehydrogenase loss causes cascading metabolic effects that impair pyrimidine biosynthesis

Also in the microbiology category:

Direct cell-to-cell transmission of retrotransposons

A prophage-encoded anti-phage defense system that prevents phage DNA packaging by targeting the terminase complex

Green synthesized silver nanoparticles from Moringa: Potential for preventative treatment of SARS-CoV-2 contaminated water

Also in the plant biology category:

Conservation and divergence of regulatory architecture in nitrate-responsive plant gene circuits

New evidence for the presence and function of phosphoinositides (PPIs) in the chloroplast

Plasmodesmal closure elicits stress responses

Also in the systems biology category:

Modular control of time and space during vertebrate axis segmentation

Natural genetic variation quantitatively regulates heart rate and dimension

Expressive modeling and fast simulation for dynamic compartments

Clusters of lineage-specific genes are anchored by ZNF274 in repressive perinucleolar compartments

preLists in the bioinformatics category:

Keystone Symposium – Metabolic and Nutritional Control of Development and Cell Fate

‘In preprints’ from Development 2022-2023

9th International Symposium on the Biology of Vertebrate Sex Determination

Alumni picks – preLights 5th Birthday

Fibroblasts

Single Cell Biology 2020

Antimicrobials: Discovery, clinical use, and development of resistance

Also in the cell biology category:

June in preprints – the CellBio edition

May in preprints – the CellBio edition

Keystone Symposium – Metabolic and Nutritional Control of Development and Cell Fate

April in preprints – the CellBio edition

March in preprints – the CellBio edition

Biologists @ 100 conference preList

February in preprints – the CellBio edition

Community-driven preList – Immunology

January in preprints – the CellBio edition

December in preprints – the CellBio edition

November in preprints – the CellBio edition

BSCB-Biochemical Society 2024 Cell Migration meeting

‘In preprints’ from Development 2022-2023

preLights peer support – preprints of interest

The Society for Developmental Biology 82nd Annual Meeting

CSHL 87th Symposium: Stem Cells

Journal of Cell Science meeting ‘Imaging Cell Dynamics’

9th International Symposium on the Biology of Vertebrate Sex Determination

Alumni picks – preLights 5th Birthday

CellBio 2022 – An ASCB/EMBO Meeting

Fibroblasts

EMBL Synthetic Morphogenesis: From Gene Circuits to Tissue Architecture (2021)

FENS 2020

Planar Cell Polarity – PCP

BioMalPar XVI: Biology and Pathology of the Malaria Parasite

Cell Polarity

TAGC 2020

3D Gastruloids

ECFG15 – Fungal biology

ASCB EMBO Annual Meeting 2019

EMBL Seeing is Believing – Imaging the Molecular Processes of Life

Autophagy

Lung Disease and Regeneration

Cellular metabolism

BSCB/BSDB Annual Meeting 2019

MitoList

Biophysical Society Annual Meeting 2019

ASCB/EMBO Annual Meeting 2018