Menu

Close

Moving beyond P values: Everyday data analysis with estimation plots

Joses Ho, Tayfun Tumkaya, Sameer Aryal, Hyungwon Choi, Adam Claridge-Chang

Preprint posted on July 26, 2018 https://www.biorxiv.org/content/early/2018/07/26/377978

A visual, intuitive and widely accessible tool could finally help us move from asking “does it?” to “how much?”

Selected by Gautam Dey

Context

Statistical analysis in the biological sciences has long been dominated by null-hypothesis significance testing (NHST). Statisticians and quantitatively-minded biologists alike have been crying themselves hoarse about the fallacies and intrinsic limitations associated with this approach for, believe it or not, approximately 75 years1,2. Unfortunately, there has been little consensus on the practical steps needed to achieve significant reform.

The authors illustrate the key limitations of NHST as well as their proposed solution using an experimental setup we are all too familiar with: one containing two groups of data points, representing a control and a test/intervention sample. Such an experiment would be traditionally visualized using bar graphs (Fig 1A), box plots (Fig 1B) or perhaps scatterplots (Fig 1C) and analyzed by a Student’s t-test or related NHST variant.

Figure 1: Reproduced from Figure 1 of Ho et al. 2018 under a CC-BY-NC-ND 4.0 international license. 2-groups data represented by bar plots (A), box plots (B), and scatter plots with jitter (C). (D) Histogram-like scatter plots with jitter, with null-hypothesis distribution and p-value (red segment). (E) Estimation plot with difference of means distribution and 95% CI (red line).

 

What is wrong with the status quo?  

  • The NHST focuses purely on a binary decision3 to accept or reject the null hypothesis (that the means of both groups are identical) and diverts attention away from the actual effect size; this is emphasized by bar plots and only moderately mitigated by box and scatter plots.
  • Visualizing the null distribution and the p-value threshold (Fig 1D, red tail) helps drive home the issues with NHST. First, even an infinitesimally small intervention to any real system will produce at least some effect, making the zero-effect hypothesis intrinsically flawed4. Second, since the p-value threshold (usually 0.05) actually lies within the tail of the null distribution, we are concluding that control and test samples are different by demonstrating that they are sometimes the same!

 

How to fix it?

  • Estimation plots focus on the difference of means (Fig 1E). The visual representation helps focus attention on the effect size, which is what we (should) actually care about. The 95% confidence interval5 (red bar in Fig 1E), that encompasses the bulk of the ∆ sampling-error distribution (by definition), is more intuitively grasped and much better behaved than the p-value. In this case, we are concluding that control and test samples are different by demonstrating that they are almost always different.

 

Why I chose this preprint  

I loved this preprint! The estimation plot provides a complete yet visually accessible description of the data- and working through the steps in Figure 1 has given me a visual framework to interpret what I thought I understood about hypothesis testing. More importantly, the authors go to great lengths to make estimation plotting broadly accessible- by providing 5 different ways in which to create them, ranging from Python code to a handy web tool that requires no programming experience whatsoever. Go ahead- try it out!

 

References:

  1. Berkson, J. Tests of Significance Considered as Evidence. J. Am. Stat. Assoc. 37, 325–335 (1942).
  2. Halsey, L. G., Curran-Everett, D., Vowler, S. L. & Drummond, G. B. The fickle P value generates irreproducible results. Nat. Methods 12, 179–185 (2015).
  3. McShane, B. B. & Gal, D. Statistical Significance and the Dichotomization of Evidence. J. Am. Stat. Assoc. 112, 885–895 (2017).
  4. Cohen, J. The earth is round (p < .05). Am. Psychol. 49, 997–1003 (1994).
  5. Cumming, G. Understanding The New Statistics. (Routledge, 2011). doi:10.4324/9780203807002

Tags: quantitative biology, significance testing, statistics for biology

Posted on: 1st August 2018 , updated on: 3rd August 2018

Read preprint (3 votes)




  • A brief interview with the authors

    Joses Ho and Adam Claridge-Chang shared

    Could you tell us a little bit about how the project started? For example, was the tool a side effect of your ongoing work on estimation statistics, motivated by the needs of other research projects in the group, or a directed effort to address a general shortcoming in the field?

    It started back when Adam and I overlapped at Oxford’s human genetics centre. It is a hub of activity around genome-wide association studies (GWAS), and uses a host of sophisticated statistical tools. As part of my PhD on language genetics, I became familiar with GWAS p-values and the odds ratio, a number GWAS uses to express relative disease risk. So that experience was my first contact with effect sizes.

    Around the same time, Adam, who does experimental neurogenetics, was frustrated by the p-value rollercoaster that so many experience: one day a phenotype is significant, next day it isn’t. He had also heard about effect sizes at Oxford, and when he moved to Singapore took to time to read some text books on the topic, including Statistics with Confidence by Douglas Altman and others, and Geoff Cumming’s Understanding the New Statistics. The concepts and tools in those books are pretty eye-opening.

    So when I graduated and returned to Singapore to start in Adam’s lab as the resident data scientist, he handed me a pile of these textbooks to read and retrain in estimation statistics. Since then, we have used meta-analysis (which is used widely in clinical settings) to synthesise thirty years of short-term memory in flies, and to systematically review over 300 preclinical studies in rodent anxiety. Our paper on fly anxiety-like behaviours used meta-analytic data to compare our results to rodent studies, and also used estimation statistics to analyse and present our results.

    Adam also loans new lab members his well-worn copy of Edward Tufte’s The Visual Display of Quantitative Information, and our group makes an effort to apply Tuftian principles when working on figures for manuscripts. In early 2016, Adam remarked to me that the confidence intervals for the effect sizes could be depicted the way Gardner and Altman did in their textbook (See Figure 1 and 2 in this PDF), and also that we could use bootstrap methods to obtain the full effect-size distribution (‘∆ curve’).

    The benefits of using the bootstrap were immediately obvious: we did not have to make assumptions about the underlying population (which Gardner, Altman, and Cumming do), and I could depict the confidence interval as a graded distribution, and so indicate a likelihood of values for the effect size rather than just a point estimate and hard error-bar boundaries.

    I started writing a version in Python for internal lab use, and along the way we gave it the name Data Analysis with Bootstrap-coupled ESTimation (DABEST). The first version of DABEST and the webapp estimationstats.com were released in late 2017.

    So it really grew out of our own frustration with significance testing, and a desire for better tools for ourselves. Then, once we were happy with it internally, it made sense to share it with everyone else.

    As you discuss in your paper, statisticians and biologists alike have been working on alternatives to NHST statistics for years without any sort of consensus in the community. Do you think the easy accessibility and visual nature of your tool could help shift the balance a bit? Your preprint has already triggered significant discussion on social media platforms- do you think this could be leveraged into lasting impact?

    Student’s t-test has incredible brand recognition among scientists, so a key motivation for the creation of the webapp was indeed an attempt to improve the branding of estimation methods. We’re not exactly marketing experts, but we hope that improving awareness and accessibility will encourage some to make the switch. Adam has given several talks in the past where he has tried to get scientists to use these estimation as an alternative to NHST. In doing this he realised he needed a simple handle people could easily grasp and remember, so decided on ‘estimation statistics’. I also attempted to get other laboratories to use my Python code, but the need to learn programming was a major barrier to adoption, so it became clear I needed to be able to say: “There’s an app for that.”

    While we were targeting basic biomedical researchers, one surprise is that our tool has gotten a fair amount of interest from other areas: ecologists,  sports scientists, psychologists and others. We do hope that estimation plots have the potential to change the data-analysis culture. Still, p-values having been under fire for over 75 years, and they are still going strong—so maybe we’ll be doomed to use them forever?

    3. Anything else you’d like to tell us about the paper, estimationstats.com, or what’s next for you and your research group?

    We’ve submitted the paper, and hope to see it in print, but are encouraged and pleased with the reception the preprint’s gotten. v0.1.4 of DABEST, which features aesthetic tweaks, will be released very shortly as well.

    Have your say

    Your email address will not be published. Required fields are marked *

    Sign up to customise the site to your preferences and to receive alerts

    Register here

    Also in the cancer biology category:

    Profiling the surface proteome identifies actionable biology for TSC1 mutant cells beyond mTORC1 signaling

    Junnian Wei, Kevin K. Leung, Charles Truillet, et al.



    Selected by Rob Hynds

    1

    Precise tuning of gene expression output levels in mammalian cells

    Yale S. Michaels, Mike B Barnkob, Hector Barbosa, et al.



    Selected by Tim Fessenden

    1

    JNK-mediated spindle reorientation in stem cells promotes dysplasia in the aging intestine

    Daniel Hu, Heinrich Jasper



    Selected by Maiko Kitaoka

    Mitotic chromosome alignment is required for proper nuclear envelope reassembly

    Cindy L Fonseca, Heidi LH Malaby, Leslie A Sepaniac, et al.



    Selected by Maiko Kitaoka

    SWI/SNF remains localized to chromatin in the presence of SCHLAP1

    Jesse R Raab, Keriayn N Smith, Camarie C Spear, et al.



    Selected by Carmen Adriaens

    1

    Single-cell Map of Diverse Immune Phenotypes Driven by the Tumor Microenvironment

    Elham Azizi, Ambrose J. Carr, George Plitas, et al.



    Selected by Tim Fessenden

    Cancer modeling by Transgene Electroporation in Adult Zebrafish (TEAZ)

    Scott J Callahan, Stephanie Tepan, Yan M Zhang, et al.



    Selected by Hannah Brunsdon

    1

    PDX Finder: A Portal for Patient-Derived tumor Xenograft Model Discovery

    Nathalie Conte, Jeremy Mason, Csaba Halmagyi, et al.



    Selected by Carmen Adriaens

    HIF1-alpha expressing cells induce a hypoxic-like response in neighbouring cancer cells

    Hannah Harrison, Henry J Pegg, Jamie Thompson, et al.



    Selected by Anh Hoang Le

    Long-term live imaging of the Drosophila adult midgut reveals real-time dynamics of cell division, differentiation, and loss

    Judy Martin, Erin Nicole Sanders, Paola Moreno-Roman, et al.



    Selected by Natalie Dye

    A role for RNA and DNA:RNA hybrids in the modulation of DNA repair by homologous recombination

    Giuseppina D'Alessandro, Marek Adamowicz, Donna Whelan, et al.



    Selected by Carmen Adriaens

    Nuclear envelope assembly defects link mitotic errors to chromothripsis

    Shiwei Liu, Mijung Kwon, Mark Mannino, et al.



    Selected by Gautam Dey

    Zebrafish as a model to investigate the effects of exercise in cancer

    Alexandra Yin, Nathaniel R. Campbell, Lee W. Jones, et al.



    Selected by Jacky G. Goetz

    Stopping Transformed Growth with Cytoskeletal Proteins: Turning a Devil into an Angel

    Bo Yang, Haguy Wolfenson, Naotaka Nakazawa, et al.



    Selected by Jon Humphries

    Cancer exosomes induce tumor neo-neurogenesis potentiating tumor growth

    Marianna Madeo, Paul L. Colbert, Daniel W. Vermeer, et al.



    Selected by Jacky G. Goetz

    1

    Also in the cell biology category:

    The cytoskeleton as a smart composite material: A unified pathway linking microtubules, myosin-II filaments and integrin adhesions

    Nisha Mohd Rafiq, Yukako Nishimura, Sergey V. Plotnikov, et al.



    Selected by Coert Margadant

    Quantitative, real-time, single cell analysis in tissue reveals expression dynamics of neurogenesis

    Cerys S Manning, Veronica Biga, James Boyd, et al.



    Selected by Teresa Rayon

    Profiling the surface proteome identifies actionable biology for TSC1 mutant cells beyond mTORC1 signaling

    Junnian Wei, Kevin K. Leung, Charles Truillet, et al.



    Selected by Rob Hynds

    1

    Optogenetic dissection of mitotic spindle positioning in vivo

    Lars-Eric Fielmich, Ruben Schmidt, Daniel J Dickinson, et al.



    Selected by Angika Basant

    1

    Excitable RhoA dynamics drive pulsed contractions in the early C. elegans embryo.

    Jonathan B Michaux, Francois B Robin, William M McFadden, et al.



    Selected by Sundar Naganathan

    Moving beyond P values: Everyday data analysis with estimation plots

    Joses Ho, Tayfun Tumkaya, Sameer Aryal, et al.



    Selected by Gautam Dey

    1

    A limited number of double-strand DNA breaks are sufficient to delay cell cycle progression.

    Jeroen van den Berg, Anna G. Manjon, Karoline Kielbassa, et al.



    Selected by Leighton Daigh

    Optogenetic manipulation of medullary neurons in the locust optic lobe

    Hongxia Wang, Richard B. Dewell, Markus U. Ehrengruber, et al.



    Selected by Ana Patricia Ramos

    JNK-mediated spindle reorientation in stem cells promotes dysplasia in the aging intestine

    Daniel Hu, Heinrich Jasper



    Selected by Maiko Kitaoka

    ER-to-Golgi trafficking of procollagen in the absence of large carriers.

    Janine McCaughey, Nicola Stevenson, Stephen Cross, et al.



    Selected by Gautam Dey

    1

    Template switching causes artificial junction formation and false identification of circular RNAs

    Chong Tang, Tian Yu, Yeming Xie, et al.



    Selected by Fabio Liberante

    Mechanosensitive binding of p120-Catenin at cell junctions regulates E-Cadherin turnover and epithelial viscoelasticity

    K. Venkatesan Iyer, Romina Piscitello-Gómez, Frank Jülicher, et al.



    Selected by Ivana Viktorinová

    Targeting light-gated chloride channels to neuronal somatodendritic domain reduces their excitatory effect in the axon

    Jessica Messier, Hongmei Chen, Zhao-Lin Cai, et al.

    AND

    High-efficiency optogenetic silencing with soma-targeted anion-conducting channelrhodopsins

    Mathias Mahn, Lihi Gibor, Katayun Cohen-Kashi Malina, et al.



    Selected by Mahesh Karnani

    2

    EFFECTORS OF THE SPINDLE ASSEMBLY CHECKPOINT BUT NOT THE MITOTIC EXIT NETWORK ARE CONFINED WITHIN THE NUCLEUS OF SACCHAROMYCES CEREVISIAE

    Lydia R Heasley, Jennifer G DeLuca, Steven M Markus



    Selected by Hiral Shah

    An atlas of the aging lung mapped by single cell transcriptomics and deep tissue proteomics

    Ilias Angelidis, Lukas M Simon, Isis E Fernandez, et al.



    Selected by Rob Hynds

    1

    Peculiar features of the plastids of the colourless alga Euglena longa and photosynthetic euglenophytes unveiled by transcriptome analyses

    Kristina Zahonova, Zoltan Fussy, Erik Bircak, et al.



    Selected by Ellis O'Neill

    1

    Also in the developmental biology category:

    LADL: Light-activated dynamic looping for endogenous gene expression control

    Mayuri Rege, Ji Hun Kim, Jacqueline Valeri, et al.



    Selected by Ivan Candido-Ferreira

    Quantitative, real-time, single cell analysis in tissue reveals expression dynamics of neurogenesis

    Cerys S Manning, Veronica Biga, James Boyd, et al.



    Selected by Teresa Rayon

    The cis-regulatory logic underlying abdominal Hox-mediated repression versus activation of regulatory elements in Drosophila



    Selected by Clarice Hong

    1

    Optogenetic dissection of mitotic spindle positioning in vivo

    Lars-Eric Fielmich, Ruben Schmidt, Daniel J Dickinson, et al.



    Selected by Angika Basant

    1

    Excitable RhoA dynamics drive pulsed contractions in the early C. elegans embryo.

    Jonathan B Michaux, Francois B Robin, William M McFadden, et al.



    Selected by Sundar Naganathan

    A transition from SoxB1 to SoxE transcription factors is essential for progression from pluripotent blastula cells to neural crest cells

    Elsy Buitrago-Delgado, Elizabeth Schock, Kara Nordin, et al.



    Selected by Amanda Haage

    Moving beyond P values: Everyday data analysis with estimation plots

    Joses Ho, Tayfun Tumkaya, Sameer Aryal, et al.



    Selected by Gautam Dey

    1

    Dynamic control of proinflammatory cytokines Il-1β and Tnf-α by macrophages is necessary for functional spinal cord regeneration in zebrafish

    Themistoklis M. Tsarouchas, Daniel Wehner, Leonardo Cavone, et al.



    Selected by Shikha Nayar

    1

    JNK-mediated spindle reorientation in stem cells promotes dysplasia in the aging intestine

    Daniel Hu, Heinrich Jasper



    Selected by Maiko Kitaoka

    ER-to-Golgi trafficking of procollagen in the absence of large carriers.

    Janine McCaughey, Nicola Stevenson, Stephen Cross, et al.



    Selected by Gautam Dey

    1

    Mechanosensitive binding of p120-Catenin at cell junctions regulates E-Cadherin turnover and epithelial viscoelasticity

    K. Venkatesan Iyer, Romina Piscitello-Gómez, Frank Jülicher, et al.



    Selected by Ivana Viktorinová

    A novel mechanism of gland formation in zebrafish involving transdifferentiation of renal epithelial cells and live cell extrusion

    Richard W Naylor, Alan J Davidson



    Selected by Giuliana Clemente

    1

    An intrinsic cell cycle timer terminates limb bud outgrowth

    Joseph Pickering, Kavitha Chinnaiya, Constance A Rich, et al.



    Selected by Ashrifia Adomako-Ankomah

    1

    Fbxw7 is a critical regulator of Schwann cell myelinating potential

    Breanne L Harty, Fernanda Coelho, Sarah D Ackerman, et al.



    Selected by Yen-Chung Chen

    Kinetic sculpting of the seven stripes of the Drosophila even-skipped gene

    Augusto Berrocal, Nicholas C Lammers, Hernan G Garcia, et al.



    Selected by Erik Clark

    TORC1 modulation in adipose tissue is required for organismal adaptation to hypoxia in Drosophila.

    Byoungchun Lee, Elizabeth C Barretto, Savraj S Grewal



    Selected by Sarah Bowling

    Also in the molecular biology category:

    Quantitative, real-time, single cell analysis in tissue reveals expression dynamics of neurogenesis

    Cerys S Manning, Veronica Biga, James Boyd, et al.



    Selected by Teresa Rayon

    The cis-regulatory logic underlying abdominal Hox-mediated repression versus activation of regulatory elements in Drosophila



    Selected by Clarice Hong

    1

    Moving beyond P values: Everyday data analysis with estimation plots

    Joses Ho, Tayfun Tumkaya, Sameer Aryal, et al.



    Selected by Gautam Dey

    1

    A limited number of double-strand DNA breaks are sufficient to delay cell cycle progression.

    Jeroen van den Berg, Anna G. Manjon, Karoline Kielbassa, et al.



    Selected by Leighton Daigh

    Template switching causes artificial junction formation and false identification of circular RNAs

    Chong Tang, Tian Yu, Yeming Xie, et al.



    Selected by Fabio Liberante

    Dynamics and interactions of ADP/ATP transporter AAC3 in DPC detergent are not functionally relevant

    Vilius Kurauskas, Audrey Hessel, François Dehez, et al.

    AND

    Major concerns with the integrity of the mitochondrial ADP/ATP carrier in dodecyl-phosphocholine used for solution NMR studies

    Martin S. King, Paul G. Crichton, Jonathan J. Ruprecht, et al.



    Selected by Reid Alderson

    1

    Targeting light-gated chloride channels to neuronal somatodendritic domain reduces their excitatory effect in the axon

    Jessica Messier, Hongmei Chen, Zhao-Lin Cai, et al.

    AND

    High-efficiency optogenetic silencing with soma-targeted anion-conducting channelrhodopsins

    Mathias Mahn, Lihi Gibor, Katayun Cohen-Kashi Malina, et al.



    Selected by Mahesh Karnani

    2

    Glutamic acid is a carrier for hydrazine during the biosyntheses of fosfazinomycin and kinamycin

    Kwo-Kwang Abraham Wang, Tai L. Ng, Peng Wang, et al.



    Selected by Ellis O'Neill

    The Histone H3-H4 Tetramer is a Copper Reductase Enzyme

    Narsis Attar, Oscar A Campos, Maria Vogelauer, et al.



    Selected by Lauren Neves

    Super-Mendelian inheritance mediated by CRISPR/Cas9 in the female mouse germline

    Hannah A. Grunwald, Valentino M. Gantz, Gunnar Poplawski, et al.



    Selected by Rebekah Tillotson

    1

    EFFECTORS OF THE SPINDLE ASSEMBLY CHECKPOINT BUT NOT THE MITOTIC EXIT NETWORK ARE CONFINED WITHIN THE NUCLEUS OF SACCHAROMYCES CEREVISIAE

    Lydia R Heasley, Jennifer G DeLuca, Steven M Markus



    Selected by Hiral Shah

    TTL proteins scaffold brassinosteroid signaling components at the plasma membrane to optimize signal transduction in plant cells

    Vitor Amorim-Silva, Alvaro Garcia-Moreno, Araceli G Castillo, et al.



    Selected by Martin Balcerowicz

    1

    Peculiar features of the plastids of the colourless alga Euglena longa and photosynthetic euglenophytes unveiled by transcriptome analyses

    Kristina Zahonova, Zoltan Fussy, Erik Bircak, et al.



    Selected by Ellis O'Neill

    1

    OptoGranules reveal the evolution of stress granules to ALS-FTD pathology

    Peipei Zhang, Baochang Fan, Peiguo Yang, et al.



    Selected by Srivats Venkataramanan

    1

    Large-scale, quantitative protein assays on a high-throughput DNA sequencing chip

    Curtis J Layton, Peter L McMahon, William J Greenleaf



    Selected by Samantha Seah

    A robust method for transfection in choanoflagellates illuminates their cell biology and the ancestry of animal septins

    David Booth, Heather Middleton, Nicole King



    Selected by Maya Emmons-Bell

    Also in the neuroscience category:

    Moving beyond P values: Everyday data analysis with estimation plots

    Joses Ho, Tayfun Tumkaya, Sameer Aryal, et al.



    Selected by Gautam Dey

    1

    Optogenetic manipulation of medullary neurons in the locust optic lobe

    Hongxia Wang, Richard B. Dewell, Markus U. Ehrengruber, et al.



    Selected by Ana Patricia Ramos

    Targeting light-gated chloride channels to neuronal somatodendritic domain reduces their excitatory effect in the axon

    Jessica Messier, Hongmei Chen, Zhao-Lin Cai, et al.

    AND

    High-efficiency optogenetic silencing with soma-targeted anion-conducting channelrhodopsins

    Mathias Mahn, Lihi Gibor, Katayun Cohen-Kashi Malina, et al.



    Selected by Mahesh Karnani

    2

    Phenotypic landscape of schizophrenia-associated genes defines candidates and their shared functions

    Summer B. Thyme, Lindsey M. Pieper, Eric H. Li, et al.



    Selected by Daniel Grimes

    Cerebellar contribution to preparatory activity in motor neocortex

    Francois Pierre Chabrol, Antonin Blot, Thomas D Mrsic-Flogel



    Selected by Mahesh Karnani

    1

    Fbxw7 is a critical regulator of Schwann cell myelinating potential

    Breanne L Harty, Fernanda Coelho, Sarah D Ackerman, et al.



    Selected by Yen-Chung Chen

    OptoGranules reveal the evolution of stress granules to ALS-FTD pathology

    Peipei Zhang, Baochang Fan, Peiguo Yang, et al.



    Selected by Srivats Venkataramanan

    1

    Mouse dLGN receives input from a diverse population of retinal ganglion cells with limited convergence

    Miroslav Román Rosón, Yannik Bauer, Philipp Berens, et al.



    Selected by Polona Jager

    Sparse recurrent excitatory connectivity in the microcircuit of the adult mouse and human cortex

    Stephanie C Seeman, Luke Campagnola, Pasha A Davoudian, et al.



    Selected by Mahesh Karnani

    1

    Cell type-specific interchromosomal interactions as a mechanism for transcriptional diversity

    Adan Horta, Kevin Monahan, Lisa Bashkirova, et al.



    Selected by Boyan Bonev

    Feedback control of neurogenesis by tissue packing

    Tom W. Hiscock, Joel B. Miesfeld, Kishore R. Mosaliganti, et al.



    Selected by Sarah Morson

    1

    Heterochromatin drives organization of conventional and inverted nuclei

    Martin Falk, Yana Feodorova, Natasha Naumova, et al.



    Selected by Boyan Bonev

    Precise temporal regulation of alternative splicing during neural development

    Sebastien M Weyn-Vanhentenryck, Huijuan Feng, Dmytro Ustianenko, et al.



    Selected by James Gagnon

    GSK3 Controls Migration of the Neural Crest Lineage

    Sandra G Gonzalez Malagon, Anna Lopez Munoz, Daniel Doro, et al.



    Selected by Amanda Haage
    Close