From amazing work to I beg to differ - analysis of bioRxiv preprints that received one public comment till September 2019

Background

As readers and writers of preLights know full well, feedback on preprints can be helpful to many different parties. Readers are drawn to read work they might have missed, and they gain additional context to help them understand the paper. Authors can collect input useful in improving their work. Along the way, the public conversation creates a sense of community and itself constitutes a form of collaboration. However, few studies have shed light on the content or impact of this commenting activity on preprints. 

Key findings

Malički and colleagues extracted information on 3,265 preprints, that in the study period (21 May 2015 – 9 Sept 2019), received one or more comments (as data was not available for the period before May 2015 through the bioRxiv API – they estimated less than 10% of all preprints on bioRxiv received comments till Sept 2019). They focused their subsequent analysis on 1,983 preprints that received a single public comment. Of these, 69% of the comments were posted by non-authors, with the remaining 31% posted by authors. 

Non-author comments were longer than comments posted by the authors, with a median word count of 38 for non-authors vs 18 for authors. Non-author comments were also posted faster after preprint posting, a median 23 days after the preprints vs 91 days for author comments (which was understandable due to the nature of the comments, see below).

12% of all non-author comments were full review reports resembling those prepared for a journal. Of these, about half (52%) were written by a single individual, with the remainder (48%) written by a group of authors. Among the non-author comments not resembling full peer review reports, the comments frequently contained praise (n=577, 42%), suggestions for improvement (n=399, 29%), and critique of some aspect of the preprint (n=226, 17%). 

The comments posted by authors frequently contained updates about the preprint’s publication status (n=354, 57%), additional information for the study (n=158, 26%), and requests for feedback for the preprint (n=65, 11%).

Malički and colleagues suggest several directions for future research and next steps. First, an obvious extension to the study would be to include comments on preprints receiving more than one comment. The authors suggest there may also be differences in the types of comments left on papers during the COVID-19 pandemic. The sample of comments for a future analysis would include cases where the authors responded to comments left by others on their preprint, enabling investigation of factors associated with authors’ willingness to respond to comments or incorporate feedback into the paper. Finally, Malički and colleagues highlight the need for user-friendly integration of comments from server platforms with those posted on social media (e.g. Twitter) and specialized review platforms. 

What we like about this preprint

While preprint commenting and review received increased attention in the context of the COVID-19 pandemic, there is a dearth of evidence around preprint commenting activities. The study by Malički and colleagues is a nice initial step into preprint commenting trends and the types of comments that are being posted. 

The study reports that less than 10% of preprints had at least one public comment. This may be viewed as low overall, but let’s place it in context with other public commenting initiatives. The rate is similar to that from an analysis of PLOS journals which reported that 5.2% of articles between 2003 and 2016 received an external comment (i.e. a comment not posted by the publisher, Wakeling et al.). In contrast, 0.16% of papers attracted comments via PubMed Commons, a commenting feature hosted by PubMed which operated from 2013 to 2018 (calculation based on comments posted 2014-2016 (Vaught et al.) vs publications indexed in PubMed in the same period). As an additional comparison, PubPeer, likely the most active among the commenting platforms for research outputs, hosted 96,000 comments in October 2020, which amounts to less than 0.1% of research outputs indexed in Dimensions. The commenting rate on preprints is thus higher or at least comparable to that on journal articles, and it can be argued that feedback on a preprint may be more motivating as it comes at an earlier stage of the work where authors can still incorporate the comments and suggestions.

Malički and colleagues report that half of the non-author comments that resemble journal peer review reports were posted by a group rather than an individual. There are similarities here with PubMed Commons, where a fifth of the posts were collaborative comments (Vaught et al.). Posting joint reviews may mitigate individual commenter concerns around the lack of anonymity or self-questioning on whether they are qualified to provide a full review of the paper. The group discussion allows different group members to provide complementary expertise into the assessment, and a safer space for individuals to share their contribution in a public forum.

The study reports that comments by non-authors are posted faster than those by authors; the median time for posting at 23 days is faster than the review timelines at many journals and highlights the potential for preprints to facilitate prompt community feedback.

Another interesting observation is that 80% of non-author comments recommended additional citations to the commenter’s own work. While it is not surprising for the commenter to mention their work (after all they may have read the preprint out of relevance to their own line of research), the question arises as to how this compares to reviewers’ citation requests during journal peer review.

We commend the authors for sharing their full dataset of comments, the categorization information, and the decision tree employed. These are fantastic resources to inspire future studies involving categorization of public comments on scholarly works; the use of a similar categorization approach would permit future comparisons of commenting trends between preprints and journal articles, over time, or across different disciplines.

Questions for the authors

1. The analysis focused on preprints posted on bioRxiv. There are other servers that provide commenting features (e.g. Research Square, medRxiv, OSF-hosted preprint servers). Will you carry out similar studies on comments posted on other servers, and would you expect to see similar or different patterns in public commenting?
2. It would be interesting to learn more about whether and how receiving comments results in any follow-up by the preprint authors, i.e. do the authors revise their preprint and update it with a new version? Is there any correlation between receiving review-type comments and the fate and time to publication at a journal? Are the items raised in the preprint comment incorporated into the version of the paper published at a journal?
3. Have you considered an analysis of comments per discipline? We would expect papers reporting findings that have high societal impact or involving potentially controversial topics to attract more comments than others. This may tie to your intention to look into preprint commenting trends in the context of COVID-19.
4. We fully agree with your call for further integration of comments on preprints posted across different platforms and we welcome initiatives such as Sciety and Early Evidence Base that are looking to aggregate preprint review activity. The current analysis looks at comments posted on the bioRxiv server itself; have you planned to look at comments available via other sites e.g. PREreview, or include reviews from review services such as eLife or Review Commons?
5. There are documented imbalances in gender and country representation in journal peer review. Preprint review opens opportunities for broadening participation, but we lack evidence on the characteristics of commenters on preprints and how they compare to journal peer review. Are you planning to look into gender and country distribution among commenters?
6. As a follow-up study, it may be interesting to carry out qualitative research to better understand non-authors’ motivations for commenting publicly, and to explore this for commenters who posted comments individually vs those who posted collaborative/group comments.

Tags: biorvix, preprint comments

doi: https://doi.org/10.1242/prelights.27585

Read preprint (No Ratings Yet)