A systematic examination of preprint platforms for use in the medical and biomedical sciences setting

Author's response

Naomi Penfold (NP) and Jessica Polka (JP) worked together on these answers. Initials are used to attribute perspectives, and both have reviewed each other’s work. shared

1. During the ongoing pandemic, some preprint platforms are changing their policies and not accepting certain kinds of work. Does this defy the purpose of preprints in your opinion?

NP: As we see from many researchers’ hesitation to read or post preprints, some element of quality control is important. Readers don’t want to have to filter through a lot of content that they don’t think is ready for them to read yet. And the current pandemic has really brought into focus the responsibility that any content hosts and disseminators have: we see that news spreads fast online, and it’s hard to revise any early impressions once the information is evaluated – many people will hear “X causes Y” and never spend time checking back in on whether that’s true in the end. As long as it’s easy for people to spread content before (and sometimes despite) any reviews come in and the community establishes what is agreed and disputed, I think it’s sensible for preprint platforms to adapt to the situation at hand, and prioritize protecting human health.

What I think is important in this case is that any quality control measures are decided by and the process involves a diverse set of researchers, and maybe even other stakeholders who have an interest in trusting the quality control process.

JP: Even preprint servers with very lightweight screening processes do exclude content they deem to be non-scientific or otherwise out of scope, so there is always some degree of subjective judgement there.

2. Are there additional screening measures that need to be employed by preprint repositories to limit the publishing of bad science?

NP: I think preprint platforms are useful for supporting more people to be included in scientific discourse, whether sharing their findings with a manuscript or assessing other people’s manuscripts and providing feedback and review.

Following on from your first question – I think we’re talking here about content that is not deemed harmful to human health. Readers should be prepared to critically evaluate preprints themselves, and I think it is useful to remind readers of that responsibility. Of course, platforms can help readers out by filtering out obviously inappropriate, potentially harmful or pseudoscience content so the reader needs only to read and evaluate scientific content. However, I think the concept of “bad science” is one that can easily lead to bias – if this form of gatekeeping is used, I think it needs to be done in a way that allows the research community to be able to understand and trust the process by which science is being judged as “bad” or “good’. Further, whether or not “bad science” is shared, I think it is useful for the community to share what they mean when they judge something as “bad science” – so anyone contributing to a field of research can be aware of the standards that the others in the field expect from them: as well as sharing feedback directly to the authors, I would support greater transparency in the overall decisions made by any preprint platforms as to whether or not a preprint submission should be posted online.

In time, we expect to see more peer review happening on preprints, and I hope preprint platforms will continue to support readers to assess any preprints for themselves, and benefit from any peer reviews of the content that are already available – whether a traditional review or other kinds of indicators. It would be useful to be able to evaluate how much review a preprint has received too, so the reader knows to what degree content has been critically evaluated by the time they are reading it.

JP: These standards for science can change rapidly, or the need to communicate them may change rapidly, and we have seen this recently with COVID-19 research. For example, not without some controversy, medRxiv and bioRxiv decided against considering papers that rely on in-silico predictions of drug efficacy, reasoning that the potential damage to public health outweighs the benefit of early dissemination.

Even informal comments on a preprint can be very valuable in correcting the record. For example, a notorious preprint claiming similarity between SARS-CoV-2 and HIV was withdrawn by authors within 2 days of posting after the paper received many critical comments.

3. One of the biggest benefits of preprints is that authors receive feedback from everyone. Are there any repositories that do not allow/limit user engagement?

NP: For providing feedback to authors, I don’t know of any platform for which the author names, affiliations and contact information are not clear, even if that is within the PDF of the preprint itself – so feedback provision is not limited.

Nonetheless, it’s interesting you use the word ‘engagement’ there. Allowing comments, annotations, likes, or other forms of interaction to provide public feedback on a preprint is one thing. But when feedback interactions like this happen, I think it’s important to notify the preprint author(s) and anyone who has shown an interest in the preprint (like you might bookmark something to follow or read later). This is what will bring them back to the preprint to check in on the new feedback and see if that affects their own understanding. I think one of the reasons Twitter is used for talking about new preprints is that people can easily get involved in the conversation and follow it. So, in answer to your question, I think that until there is a good experience with how authors and readers follow the conversation around a preprint, all true ‘engagement’ is limited.

4. Who initiates removal/retractation of preprints from the platform? Is it solely the responsibility of the platform?

NP: Most platforms will respond if any author or reader flags an issue that could mean withdrawing a preprint, and most platforms will be the ones who actually withdraw or remove the preprint, following their process. To maintain the scholarly record, it is preferable to withdraw (leave the content on the web, but clearly mark it as withdrawn and provide a reason where possible). Sometimes content is removed completely – if that’s in the first few days of its release then it may not affect the citation trail much. There are cases where content is removed after many months, and that makes it more likely that someone may have cited work that is now no longer available.

JP: We are working with attendees of a recent workshop, #bioPreprints2020, to propose standard definitions for withdrawal and removal (which is necessary under certain legal or ethical circumstances) and conditions under which those terms and procedures might be used.

5. Do First Look platforms that have partner journals offer preprints free-of-charge/without journal subscription?

NP: The content shared on some journal-associated platforms on SSRN, such as Preprints with the Lancet and Cell Sneak Peek, require readers to register and log-in in order to view the full content. These accounts are free-of-charge and are not the same as a paid subscription, as far as we are aware.

6. Are there any disadvantages to using platforms that do not provide DOI vs the ones that do?

NP: DOIs provide a persistent identifier and web address for content, even if the host’s URL for the content changes. This makes them useful for citation practices. However there are many citable forms of communication that do not have DOIs, including webpages and blog posts, so not having a DOI does not necessarily mean your preprint could not be cited. In academia, DOIs are also viewed as important for showing the legitimacy of a piece of content: whether in a reference list for your own manuscript or when presenting your own works in grant and job/promotion/tenure applications.

7. Do all preprint platforms archive manuscripts that are retracted?

NP: The majority of preprint platforms use a service to archive their content, and what this will mean in practice depends on the service agreement. However, it is common for journals and preprint platforms to use a mechanism called ‘dark archiving’, which means that the content is preserved but not accessible unless a condition is ‘triggered’, such as the preprint platform ceasing operations (and therefore not being able to host the content itself anymore). The service agreements are not publicly available, and we do not know whether retracted content is included in the archiving agreement for preprint platforms.

If you’d like to find out more, the service commonly used by preprint platforms is Portico, and their website describes their service in greater detail. You can look up what they preserve by searching by the publisher (for example, bioRxiv and medRxiv are preserved, listed under Cold Spring Harbor Laboratory as the publisher).