Transparency in peer review: Exploring the content and tone of reviewers’ confidential comments to editors

 

Background

Peer-review is traditionally used in academic publishing to assist editors with publication decisions and to provide critical feedback to manuscript authors. For many journals, reviewer identities are blinded to authors with the belief that this anonymity permits an honest review of a manuscript without the risk of retaliation from the authors, particularly in cases of unequal power dynamics. However, this single-blinded peer review approach has come under criticism owing to the lack of conscientiousness that might arise when reviewers know their identity is concealed, and the absence of transparency around editorial decisions. An increasing number of journals are opting for an open peer-review process that not only requires reviewers to share their identity, but also to make their reviews publicly available alongside the published manuscript (1). However, even in these instances, the identity of the reviewer is not commonly known to the author until the point of publication. Despite this, the increased transparency associated with publishing peer reviews may help to improve quality and fairness of the peer-review process. Publishing peer reviews can also be more useful to readers of the final paper.

Alongside comments to authors, reviewers are often given the opportunity to provide confidential comments to the editor which are not shared with the manuscript authors. In most cases, this includes the reviewer’s decision recommendation (e.g., accept, minor revisions, major revisions, reject). Although reviewer guidelines tend to advise that reviewer comments to the author should include ‘most feedback’, confidential comments to the editor could compromise transparency of the peer review process and may conflict with the final editorial decision. Non-transparency for the author may hinder confidence in the legitimacy of the peer-review process. Further, hiding feedback from reviewers does not aid with improving manuscripts. On the other hand, the option of confidential comments allows reviewers to discuss conflict of interest, suspected plagiarism and to either agree or disagree with the other reviewer’s comments (in the cases a given reviewer has access to see other reviewer’s comments).

In this preprint, the authors aimed to investigate how confidential comments in an online peer-review system may impact transparency and quality of feedback. Specifically, they looked at how confidential comments to the editor were used, and how the content and tone of confidential comments from the reviewer differed from comments to the authors when they recommend a manuscript should be rejected.

Study design

The authors focused on papers submitted to Perspectives on Medical Education (PME), a peer-reviewed journal aimed at furthering clinical education. PME publishes ~65 articles a year from ~400 annual submissions. Manuscripts that proceed to peer review require at least two external reviews. Peer review is single-blind – the reviewer knows the author’s identity, but authors are blind to their reviewers – and all reviewers must submit their publication decision along with a statement. Reviewers may also submit private comments to the editor which are visible to the editorial board but not the manuscript authors.

Reviews of peer-reviewed manuscripts submitted between January 2019 and August 2020 were included for analysis. The dataset included final editorial decision, reviewer comments to the author and reviewer comments to the editor. Any identifying information with regards the author or reviewer was excluded from the dataset. Comments to the author and the editor were coded for tone into categories including critical, constructive, supportive or neutral, and compared for alignment in content and tone between author and editor feedback. Only those reviews that recommended ‘reject’ as an editorial decision were evaluated for alignment, given the greater anticipated discrepancy between the comments to the authors of the manuscript compared to the editors. We would like to note that the authors declared in the preprint that they have editorial roles in the journal. De-identified comments about the manuscripts are available on Zenodo (https://zenodo.org/record/5128724#.YPwyahNKjUo).

Main findings

Overview of dataset

The final dataset included 358 reviews of 168 articles, 65% of which were original research articles, and the rest comprising research reviews or brief preliminary data communications. 144 articles had a final decision from the editor, including 88 rejections (61%). 49% of the reviews included confidential comments to the editor. A larger percentage of reviews with comments to the editor recommended reject (27%) as opposed to those with no comments to the editor (21% reject). Nonetheless, final rejection rate was similar for reviews with (56%) and without (52%) comments to the editor.

How is the ‘comments to the editor’ in online peer-review platforms used by reviewers?

The authors analysed 176 reviews that included comments to the editor around three dimensions: content, section of the manuscript and tone of comments to the editor. In general, where comments to the editor occurred, they were brief and often referred to comments that had been made separately to the author.

They found that less than half of the comments to the editor (44%) stated their publication recommendation, whereas more than half remarked on the quality of the work and addressed research design or shortcomings. Most commonly this related to the methods or results section. A small percentage of comments (10% and 3% respectively) addressed concerns around suitability of the work for the journal or raised ethical concerns or potential conflicts of interest. The comments to the editor frequently included remarks on their own suitability as a reviewer, the journal editorial process, gratitude at being asked to review a manuscript, or to engage further with the journal.

Although the authors found that most comments to the editor were critical in tone (40%), this varied by reviewer recommendation. A more constructive tone was associated with a recommendation for revisions, and a critical tone was associated with a recommendation to reject the manuscript. Interestingly, most of the comments to the editor came from reviewers located in USA and Canada. Notably, there were no reviewers from Asia, South America or most of Africa.

How does the content and tone of reviewer’s comments differ between those to the author and those to the editor?

56% of reviews with a reject recommendation were found to have comments to the editor. The majority of these aligned in content between comments to the author and comments to the editor (65%).

In the 17 reviews (=35%) that differed in content, the authors of the preprint found some instances that compromised transparency around the peer-review process. As is cited in the preprint, a comment to the editor might state ‘We find this manuscript not suitable for publication’, compared to comments to the author suggesting changes to the manuscript which would make it suitable for publication, such as ‘We suggest to…’ or ‘This can be overcome by…’.

Interestingly, the authors of the preprint found that the tone of comments to the editor and the author generally aligned (85%). Where there was a discrepancy in tone between the two, comments to the editor were more critical than those to the author.

Why we chose this preprint

Peer review and the traditional scientific publishing process has come under criticism in recent years for a number of reasons, including concerns around transparency and bias. Whilst it has been suggested that peer-review may never be free of bias (2), the peer review process should still aim to minimise bias of reviewers as much as possible.

As preprinting has become more prevalent across scientific fields, the popularity of open peer review or community feedback on preprints has increased. Interestingly, a recent study in BMJ found no difference in the quality of review reports between open (reviewer identity known) vs anonymous review (reviewer identity hidden) (3). We are also seeing more journals publishing reviewer reports alongside formally published manuscripts in a bid to increase transparency of the review and decision-making process. These new avenues might avoid issues associated with the current system, but could also be open to potential novel sources of bias.

Interestingly, the findings of this preprint indicate that concerns around confidential comment to the editor, even in a more ‘open’ peer review process, could be unfounded. As they describe, the majority of comments to the editor align in tone and content with comments to the author, and comments to the editor were commonly used to make personal remarks about the journal or their review process. In these instances, comments to the editor could even be perceived positively, as a way of enriching the relationship between reviewer and editor in an otherwise impersonal and time-consuming process. However, we and the authors note that since the data was de-identified it is not possible to ascertain if the rejection or comments to the editor had any geographical, gender, or other source of bias.

As described in the preprint, if confidential comments to the editor persist during traditional peer-review, it might prove beneficial for journals to clearly specify which instances these should be used in, so as not to compromise the transparency of comments to the author. In some cases – ethical considerations, conflict-of-interest, or suitability as a reviewer – such confidential comments could still be warranted. With community feedback on preprints becoming more popular, it will be interesting to see how the dialogue between reviewers, authors and editors changes to reflect an increasing desire for transparency in peer-review and publishing.  It is hearting to see that at least in the context of this study in this particular journal, confidential comments to editors were given by reviewers in the spirit that they were intended to.

(1) https://elifesciences.org/articles/64910

(2) https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.22784

(3) https://www.bmj.com/content/318/7175/23

 

 

doi: https://doi.org/10.1242/prelights.30564

Read preprint (No Ratings Yet)