When I started writing this blog post I titled it “Peer review: the good, the bad and the ugly”. Then I googled “peer review” and found that was not a clever or original title (see here and here for starters), and certainly wouldn’t get past Reviewer 2.
“Peer review is the system used to assess the quality of a manuscript before it is published. Independent researchers in the relevant research area assess submitted manuscripts for originality, validity and significance to help editors determine whether a manuscript should be published in their journal.” https://www.biomedcentral.com/getpublished/peer-review-process
Peer Review Week (a global event celebrating the vital role peer review plays in maintaining scientific quality) has a focus on quality in 2019: how can you, the research community, provide and be provided with quality peer review?
“The goals of peer review are both lofty and mundane. It is the responsibility of journals to administer an effective review system. Peer review is designed to select technically valid research of significant interest. Referees are expected to identify flaws, suggest improvements and assess novelty. If the manuscript is deemed important enough to be published in a high visibility journal, referees ensure that it is internally consistent, thereby ferreting out spurious conclusions or clumsy frauds.” www.nature.com/nature-research/editorial-policies/peer-review
How do you know if you are reviewing well? There are some great resources written by researchers and publishers including Notes for Those New to Writing Peer Reviews, How to write a structured reviewer report and the hashtag #BeReviewer1 on Twitter. There is also plenty of information on how NOT to peer review well – see the hashtag #Reviewer2 on Twitter, or The Three Types of Peer Reviewers. It is always worth reading the peer review policy of key journals in your field, both so you know what to expect when you submit a paper and so you know what is expected from you as a peer reviewer.
“The review by peers is a tried and tested method …. it places great demands on reviewers whose assessments must be honest, fair, safe and independent.” https://publons.com/community/academy/
What makes a good reviewer?
- Responds to accept or decline review invite emails promptly and provides the review in a timely way
- Provides constructive criticism with justification
- Is helpful and collegiate, treating others as they expect to be treated
- Discloses conflicts of interest
- Has relevant expertise and informs the editor if any part of the paper falls outside their expertise
- Assesses validity of research, results and conclusions
- Raises ethical considerations in a balanced, respectful way
- Highlights if key literature is not referenced
Most journals with a traditional single blind peer review system (where the reviewers remain anonymous to the authors) require that the manuscript is only seen and commented on by the specified reviewer, not in collaboration with their research group of PhD students, for example Nature. However, some journals offer more flexibility such as EMBO Press journals:
“We subscribe to referee confidentiality rules. On the other hand, we are aware of the relatively common practice of handing a review onto someone else in the laboratory. If carried out correctly, this can in fact be an important part of training, but it should be reserved for experienced postdoctoral researchers. Thus, all EMBO Press journals allow co-refereeing with one other senior member of a referee’s laboratory as part of the mentoring process only if the primary referee has independently evaluated the manuscript and agrees with the report filed. For co-review, conflict of interest and confidentiality rules apply to both referees. In order to provide accountability and appropriate credit, we request that the name of the co-referee be documented to the editors.” https://www.embopress.org/transparent-process
Cartoon by Hilda Bastian, CC BY-NC-ND
Blind peer review can be affected by bias as explained in this PLOS blog post. Some journals allow, or mandate, open peer review meaning that the author knows who the reviewers are. Over 70 of the journals published by BMC have adopted open peer review, stating that “Open peer review facilitates accountability and recognition, and may help in training early career researchers about the peer review process.” https://www.biomedcentral.com/about/advancing-peer-review
Proceedings of the Royal Society A provides authors and reviewers the option of either blind or open peer review: “Where authors opt for open peer review, Proceedings A will make the editorial process transparent for all accepted papers, by publishing reviewer reports, the substantive part of decision letter after review and the associated author responses alongside published articles. This will allow readers to better assess the published paper and provide post-publication comments (which will also be published). If the reviewers have opted to disclose their names this will be included too. Referee reports are made public under the CC-BY open access licence.” https://royalsocietypublishing.org/rspa/open-peer-review
Researchers in some disciplines have for many years published pre-prints, drafts of research papers made available prior to peer review, to publicise initial findings and receive feedback before submission to a journal. arXiv (pronounced ‘archive’) is one of longest established pre-print servers (the University of Southampton recognises the value of arXiv and provides financial support). The internet has also enabled the development of post-publication peer review systems such as F1000 and allowing authors to make their review history public as with PeerJ. Peerage of Science uses a system called collegiate peer review, where many reviewers can assess a submission and review other reviews. Once a manuscript has been reviewed and amended, member journals can make an offer to publish it.
How can you receive requests to peer review? We recommend creating a free review profile on Publons, a platform that allows researchers to track, verify and be recognised for their peer review and editorial work. The Publons Academy is a free peer review training course for early career researchers and includes links to useful resources. Journal editors can use Publons to find reviewers, but networking at conferences and sending a copy of your CV to relevant journals can be good ways of getting noticed.
Publons also provides publishing advice and guidance from researchers. I read this piece offering advice for early career peer reviewers published for Peer Review Week 2015 with interest – I agree with many of the points but I personally think there can be merit in open peer review. If you are in the mood for a longer read that includes open peer review and issues with preprints and post-publication review, I recommend The state of the art in peer review, an article by J. Tennant in FEMS Microbiology Letters, the follow-up interview that took place in Peer Review Week 2018 and a recent preprint by the same author on The limitations to our understanding of peer review, feeding in responses from this Twitter thread.
The University of Southampton is a member of Vitae, a non-profit programme providing professional development for all types of researchers that has a wealth of resources available when you create a free account. The Vitae Researcher Development Framework sets out the knowledge, behaviours and attributes of researchers into clearly defined areas with quick tips, advice sheets with ‘action and evidence’ examples for each, enabling you to develop your expertise to provide honest, fair, valid and independent reviews for others.
Do you have any thoughts about quality in peer review? We would love to hear your comments.