Close

Science in motion: A qualitative analysis of journalists’ use and perception of preprints

Alice Fleerackers, Laura Moorhead, Lauren A. Maggio, Kaylee Fagan, Juan Pablo Alperin

Preprint posted on 4 February 2022 https://www.biorxiv.org/content/10.1101/2022.02.03.479041v1.full

Preprints hitting the headlines! How do journalists report on preprinted research?

Selected by Helen Robertson, Jonny Coates

Introduction

The COVID-19 pandemic initiated a well-documented increase in the number of preprints posted, viewed, and cited (1). This influx of pandemic-related preprints also meant that preprinted research was, for the first time, frequently reported in the mainstream media. Prior to the pandemic, there was a general hesitancy on the part of journalists to report on preprints: a reluctance that could be attributed to the perceived risk of writing about yet-to-be peer-reviewed research for the general public, who might be less familiar with the concept of preprints or preliminary results. However, the urgency of scientists to report new COVID-19 research findings, and of journalists to report these findings in the media, meant that preprints became a much more dominant part of the research and science communication landscape from March 2020 onwards. The research to preprint to reporting pipeline is not always straightforward: some flawed preprints related to COVID-19 did receive considerable media attention, despite being incorrect or unfounded. It should be noted that the majority of these flawed preprints were largely disregarded or critiqued by the scientific community, and often withdrawn as a result. Further, peer-review is not infallible, and several problematic peer-reviewed papers have been widely reported on in the press prior to their withdrawal. However, reluctance to share unverified science might contribute to further hesitancy around reporting on preprinted research.

In this preprint, the authors examine how journalists say they are covering preprints, and whether the COVID-19 pandemic might have a long-term influence over the prevalence of preprints in the media. To do this, they frame journalistic use of preprints as a response to post-normal science (PNS), defined with four features, all with relevance to the COVID-19 pandemic:

1)       High levels of scientific uncertainty

2)      Science policy considerations that involve values, not just evidence

3)      High relevance to society

4)      An urgent need for political decision-making

As the authors note, how journalists pivot from ‘normal’ science reporting to those better-suited to PNS has not been well-documented. This preprint therefore investigates the use of a novel context (COVID-19 preprints) in a PNS scenario. In their study the authors address the following questions in interviews with 19 health and science journalists:

1)      What benefits and risks do journalists consider in deciding whether to cover preprints?

2)      What practices do journalists use to find, verify, and communicate the preprints they cover?

3)      How has the COVID-19 pandemic affected journalists’ use of preprints?

Main findings

–          Generally, the journalists interviewed in the study described reporting on preprints as a risk-benefit pay off. Interestingly a lot of the benefits of reporting on preprints are similar to those that come with depositing research on a preprint server – namely the timely nature of new findings (not delayed by slow peer review or editorial processes) and the accessibility of preprints (not behind a paywall).

–          The risk associated with reporting on preprints meant that a lot of journalists claimed to express caution in their use of preprinted research. Interestingly, when they did decide to use preprints in reports for widespread media outlets, journalists said their approach to verifying results was similar to normal science journalisms best practices: critical reading of methods and results, verifying results with other scientists, and comparisons to other published studies.

–          Journalists said that they were clear about the preprinted nature of new research, and in interviews said that they flagged research as unverified in articles. Whilst some journalists said they reported on preprints as if they were formally published papers, most journalists stated that they provided some context and limitations in their articles.

–          The response by journalists to the pandemic with regards reporting on preprints was surprisingly variable amongst those interviewed. Most journalists interviewed stated that the COVID-19 pandemic had changed their use of preprints, but whilst some experienced a total shift of never using preprints to reporting on them frequently, others were much more moderate in their views.

–          Generally, the journalists interviewed indicated that an increase in preprint reporting had been a ‘net positive’ in the context of science communication and the pandemic. The findings of this study suggest that reporting of preprints is likely to continue post-pandemic for science and health stories, marking a longer-term change to science reporting practices.

–          The authors of the preprint do note that if this trend continues, scientists should be conscious of journalistic bias or reporting errors in the way new research is reported, and caution that scientists should always be aware of how they share their findings with the public through journalists.

–          Lastly, and importantly, the authors of the preprint emphasise that both scientists and journalists have a role in the responsible reporting of science to the public. In helping journalists to verify preprinted research, scientists are playing a responsible role in science communication, and also feeding into the evolving preprint review landscape, which separates peer review and peer commenting on new research from exclusively being linked to a journal.

 

Why is this important?

As noted, preprints have seen a big increase in popularity as a result of the COVID-19 pandemic – not just in medical sciences, but across life science research (1). There are lots of benefits to researchers in preprinting their research, but also benefits to the public, many of which were highlighted in this study (2). Of course, reporting on any scientific research (not just preprints) to the general public comes with responsibility, and this preprint notes that journalists say they are generally aware of this when reporting on public-interest science, particularly in the context of preprints. Whilst there has been some controversy in the past over how preprints can and should be communicated to the public, this preprint shows that preprints are increasingly part of the science publishing and communication landscape.

Perhaps in the future, appropriate training and development for journalists by professional bodies could be implemented to give journalists confidence in reporting on preprints appropriately and remove the ‘fear’ associated with using preprinted research in public communication.

 

Posted on: 3 March 2022

doi: https://doi.org/10.1242/prelights.31559

Read preprint (No Ratings Yet)

Author's response

Alice shared

Q1) Did the experiences of journalists impact their responses? I.e., were younger journalists more likely to have positive attitudes to preprints?

We did not specifically look at how journalists’ lived experiences and demographics influenced their use of preprints. But there was some evidence that the type of outlet, and, by extension, type of audience, that journalists wrote for influenced the way they perceived, used, and covered preprints. For example, journalists who wrote for more niche science outlets—which target a more science-literate and science-interested audience—seemed to have more experience using preprints, more in-depth awareness about how and why scientists use them, and more knowledge of the scientific publishing system overall. In contrast, journalists who wrote for more general interest outlets often appeared to be less familiar with preprints or scientific publishing. Several stated that their audience might struggle to understand terms like “preprint” without a lengthy explanation of what the term actually means.

On a related note, we were surprised to find that even journalists with advanced science training (think a PhD in a STEM field) struggled with how to vet preprints. Lived experiences, background, and demographics certainly play a role in how journalists do their work, but our study focused on the perspectives that were shared by almost everyone we spoke with.

Q2) The challenge of “verification” appears to be a major issue with preprints, yet we know that peer review is not a foolproof verification process. However, the comments would suggest that journalists are relying on opaque peer review more than talking to independent experts (“almost all of the journalists said that an additional level of scrutiny was required to vet preprints than to vet peer-reviewed journal publications”). How do you think we can overcome this and promote more responsible reporting?

There’s a lot to unpack in this question.

While it’s true that journalists said they applied additional scrutiny to preprints, it’s not quite true that they relied more on “opaque” peer review than on talking to independent experts. In fact, as we report, most journalists agreed that interviewing independent experts was important when covering any research, but even more so when it came to preprints.

We may need to adjust the way we report our results to make this clearer. Yes, many journalists preferred peer reviewed publications to preprints. But many also acknowledged the limitations and imperfections of the peer review system, stressing that skepticism is a core part of the journalist’s job—whether the research they’re covering is peer reviewed or not.
In terms of promoting responsible reporting, we think that researchers can support journalists in several ways—all of which apply to peer reviewed research as well:

1. Use clear, accessible language, and perhaps include a lay summary. Several journalists expressed frustration with the jargon that scientists used in their papers (preprint and peer reviewed). Some ended up not covering studies because they were too challenging to understand.
2. Be very explicit about the limitations of your study, report any conflicts of interests, and do not oversell the importance of your findings. This will help journalists cover them more accurately.
3. If a journalist reaches out for a comment on a new study, take this role as seriously as you would a formal peer review request. Do your “review” with a public audience, rather than a peer audience, in mind. Why does this research matter for the public? What are the limitations or implications that are relevant to them?

We discuss these and other recommendations in more detail in our Discussion section, for those who are interested in learning more.

Q3) Preprint servers host comments and often link to public peer review. Were there any examples of journalists using these to help verify the preprints?

It’s not included in our manuscript, but we did actually find some evidence, at least among certain participants, that journalists used public forms of peer review (both formal and informal) to support their preprint coverage. For example, a couple of journalists described monitoring scientists’ Twitter conversations to see how new preprints were being received by other researchers. One journalist mentioned using platforms like PubPeer to identify the research that was most interesting to the academic community at that time. There weren’t a ton of journalists who reported using these forms of public peer review, but the few folks who did seemed to find it useful.

Have your say

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Sign up to customise the site to your preferences and to receive alerts

Register here
Close