How to be an effective peer reviewer

Dr Diana MarshallHead of Reviewer Programmes

Taylor & Francis Group

Dr Jia YangReviewer Engagement Manager

Taylor & Francis Group

Overview of peer review• Overview of the process

• Why peer review is important

• Types of peer review

Who is involved?

Editor• Assesses the article• Usually selects suitable

reviewers• Makes decisions

Reviewers• Assesses the detail• Give advice and expertise to

the Editor

Journal staff• Check format and journal

requirements• Manage communications• Production processes once

article accepted• Maintain journal systems and

websites

Why is peer review important?

• Editors cannot be experts on the detail of all

subjects

• Peer review is a conversation between the

authors, the editors, and the reviewers -

improve the quality of research and enable

effective communication of knowledge

• 90% of researchers believed their last paper

was improved through peer review - Sense

about Science Peer Review Survey 2019

“After authors, reviewers are

the lifeblood of any journal.”

Mike J. Smith, Editor-in-Chief, Journal of Maps.

Types of peer review

• Reviewers know the identity of the authors• Authors do not know the identity of the reviewers• Most common model of peer review

• Reviewers do not know the identity of the authors• Authors do not know the identity of the reviewers• May fail to hide author identity in 25-50% of cases*

• Reviewers know the identity of the authors• Authors know the identity of the reviewers• Reviewer reports may be published with reviewer

names if article accepted

*O’Connor E, Cousar M, Lentini J, Castillo M, Halm K, Zeffiro T. Efficacy of double-blind peer review in an imaging subspecialty journal. AJNR Am J Neuroradiol. 2017;38(2):230-235.

Single-blind/

Single-anonymous

Double-blind/

Double-anonymous

Open peerreview

Other models of review

• Transparent peer review: reports published without reviewer names• Post-publication peer review: article published without review and process open online

Responsibilities and expectations on reviewers• Expectations on reviewers

• When you receive an invitation letter

Give readers:

• Trusted research integrity of the article

• Ensuring adequately detailed methods to allow readers to judge the merit of the study design

Give authors:

• Detailed feedback

• Highlights any errors or gaps in literature

• Assists with making the manuscript more applicable to the journal readership

Give editors:

• Summarized information on research merit, originality, and validity of results

• Allow editors to assess the suitability of manuscript for publication in the journal

Reviewers

Editors

ReadersAuthors

Expectations on reviewers

When you receive an invitation letter:

• Meeting the deadline is essential for the whole peer review progress• Peer review is not a one-off task, you may be asked to review further versions of that manuscript

Can you complete the review in a timely fashion?

• From the information in the invitation, do you think that you can assess all or part of the manuscript?

Do you have the right expertise?

• Conflicts of interest are connections to an author, idea, or funding organization that could interfere with your ability to be objective

Are there any potential conflicts of interest?

Are you happy with the type of review used by the journal?

When you receive an invitation letter:

• Accept

• Decline - Indicate the reason‒ Declare conflicts of interest if any ‒ Give keywords to make sure journal editor knows your expertise, if the invitation is

not within your subject area‒ Suggest replacement reviewers if you are able (DO NOT contact potential reviewers)

• Unavailable‒ Specify when you will be available ‒ Editors may get back to you with an extended deadline

The Editor would rather have a quick decline than a report which does not help them make a decision

It’s OK to say no

Bear in mind…• DO submit your report and

recommendation in a timely manner.• DO declare any potential conflict of interest

before agreeing to review and any relationship that may potentially bias your review

• DO keep the peer review process confidential – from the moment you get the invitation

• DO judge manuscript on its merits, without regard to race, religion, nationality, sex, seniority, or institutional affiliation of the author(s)

• DON'T use any of the results or novel approaches reported in the manuscript until it is published

• DON'T share or discuss any aspect of the manuscript with anybody else

• DON'T make personal attacks on the author rather than providing assessment of the work itself

• DON'T ask the authors to cite your own or colleagues work unnecessarily

COPE Ethical Guidelines for Peer Reviewers

Critically assessing a manuscript• What reviewers look for

• Getting an overview of the paper

• Detailed assessment

• Should peer review detect fraud and misconduct?

• Assessing language

What do peer reviewers look for?

Presentation Quality

Research significance

OriginalityValidity

• Contribution to current knowledge

• Interest and importance

• Original contribution• Novelty

• Soundness of study• Appropriateness of the

conclusions• Research ethics

• Quality of language• The style of the paper

4 DIMENSIONS OF PEER REVIEW

Getting an overview of the paper

1. Get an idea of the direction that the paper is going• Abstract

• Methods and figures

• Conclusions

2. Read through beginning to end

• Make brief notes if needed

• No need to pick up all the details yet

15

Once you have an overview go back to the beginning and work through section by section for your detailed assessment

Detailed assessment:

Concise, accurate, and informative

A clear, short summary of the full manuscript

Why the study matters and putting the research in context

Appropriate to answer the research question and enough detail to repeat

Clearly describe the outcomes

Interprets the results, discusses limitations

Title

Abstract

Introduction

Method

Result (incl. table and figures)

Discussion &conclusion

Manuscript

Consider what is required from each section of the article

Find our checklist online for the details to check in all the sections

Detailed assessment:

• Need to be suitable to answer the question and enough description to be repeatable

• Consider the methods impartially

Ask questions when assessing:

• Are the study design and methods appropriate for the research question?

• Is there enough detail to repeat the experiments?

• Is it clear how samples were collected or how participants were recruited?

• Is there any potential bias in the sample or in the recruitment of participants?

• Are the correct controls/ validation included?

• Has any randomization been done correctly?

• Is the time-frame of the study sufficient to see outcomes?

• Is there sufficient power and appropriate statistics?

• Do you have any ethical concerns?

Researchers gave articles with the same methodology and different outcomes to 75 reviewers:• More likely to notice errors in methodology if

the results were negative• More positive about results which matched

their theoretical preference

Mahoney 1977, Cognitive Therapy and Research 1(2)Mahoney, M.J. Cogn Ther Res (1977) 1: 161

https://doi.org/10.1007/BF01173636

Pay particular attention to the methods

• Clearly describing outcomes which

would be derived from the methods

presented

• Data should not be selectively reported

• Results should be descriptive and

detailed, not presenting explanation or

discussion of the data

• It is ok for results to be negative

Checklist

• Do the results presented match the methods?

• Have all the relevant data been included?

• Are the results presented clearly and accurately?

• Is there any risk of patients or participants being identified?

• Is the data described in the text consistent with the data showed in figures and tables?

Detailed assessment:

Results

Detailed assessment:

• Is the data presented clearly and accurately in the figures and tables?

• Is the presentation of tables and figures consistent with the description

in text?

• Are the figure legends and table headings detailed enough to stand

alone from the text?

• Do you have any concerns about the manipulation of data?

• Is there any data in the main article which should be in the

supplementary files?

Look carefully at the tables and figures, including the supplementary data

• Interpretation of the findings

• Comparison with current knowledge

• Importance for future research

• Correspond with the aims of study

• Supported by data presented

• Realistic about limitations and what next steps might be needed

Checklist

• Do the authors logically explain the findings?

• Do the authors compare the findings with current findings in the research field?

• Are the implications of the findings for future research and potential applications discussed?

• Are the conclusions supported by the data presented?

• Are any limitations of the study discussed?

• Are any contradictory data discussed?

Detailed assessment:

Discussion and Conclusion

Reporting guidelines

Checklists of key points to cover in different biomedical articles – can help identify what needs covering in a reviewhttp://www.equator-network.org/

Taylor & Francis Author Survey, 2015

Should peer review detect fraud and misconduct?

“Peer review is not intended for, and is not an efficient or effective means for, the

detection of deliberate research fraud, or indeed other forms of misconduct.”

Mark Ware, Peer Review: An Introduction and Guide

• Peer review is not primarily to detect misconduct• But peer reviewers are looking closely at the article

and may see things that others have missed

Research misconduct Publishing misconduct

Author misconduct

If you have concerns then contact the journal

Data fabrication

and falsification

Plagiarism & self-plagiarism

Research ethics

• Deliberately made up data (fabrication) or change data (falsification)

• Re‐use without appropriate citation of the original source

• Any concerns about treatment of patients/ participants/ animals

Multiple publication

Others

• Submitting the same paper to more than one journal at the same time

• Improper author contribution or attribution

• Undeclaration of conflicts of interest

• DO contact the journal editor‒ If you want to see the original

files‒ If you are very concerned then

email immediately‒ If not urgent, could be included

in comments to editors‒ Give the journal specific details‒ Using neutral wording and

reporting

What to do if you are concerned

• DON’T try to contact the author directly

• DON’T accuse the author of deliberate manipulation

• DON’T investigate in detail e.g. run a plagiarism check yourself

Assessing languageShould you correct spelling and grammar errors?

Difficult to understand

Consider rejection orsuggest to editor revisions required

Not clear but understandable

Request a language polishing in review

report

Minor spelling and grammar issues

Ignore, there will be copyediting prior to

publication

79% of recommendations by reviewers were

influenced by grammar and writing style.M Shattell, et al., Journal of Nursing Scholarship, 42(1), 58-65, 2010

Reviewing non-research articles

Review article• Summary of current state of research• Balance and accuracy

Systematic review• Focus on specific methodology• Review the methods in detail

Software articles• Describing development of a new software tool• Suitable testing in relevant conditions• Is the tool available? e.g. code open source on

GitHub as required

Methodology• Detailed description and testing of a novel method• Rationale for having the new method and is it possible

to replicate

Opinion (commentary, editorial)• Variety of articles where the author gives their views

on a topic• Based on current knowledge and contributed to the

research field?• Well-argued?

Check the journal’s instructions for authors

Case report• Describing clinical progression of an unusual

disease, presentation of a disease or side-effect• Diagnosis/treatment/outcomes clearly described

Qualitative research

• Is the aim of the study clear?

• Do authors provide sufficient information about context of the research, data collection and analysis process?

• Is the methodology well described and suitable for the research question?

• Is the questionnaire standard or suitably validated?

• Are the findings and analysis aligned with used methodology?

• What contribution does the study make to knowledge?

Qualitative research seeks in-depth understanding of social phenomena.

It focuses on the “why” and “how” rather than the "what" and “how often” types of questions

Bear in mind…• Neutrally assess the research which is there, not what you wish the

authors had done

• Most research is not perfect, as long as the limitations are clear it

doesn’t need to be

• Take notes constantly by assessing not only the issues to improve but

also the strengths of the manuscript

• It’s not the reviewer’s responsibility to investigate author misconduct

• It’s not the reviewer’s responsibility to polish grammar or spelling

• Communicate with the editor/journal if you have any problems or

concerns

Writing a review report• Structure of the report

• Making your recommendation

• Style of the report

• Re-review

Structure of the review report

• What the manuscript is about• Key findings and conclusions• Contribution of the manuscript• Strengths and weakness

• Essential points the authors must address for publication

• Fundamental points for the current study

Comments and recommendation should match

SUMMARY

MAJOR COMMENTS

MINOR COMMENTS• Still important but will not affect the overall

conclusions• Good suggestions, suggested but not essential

comments to the authors

Major vs. minor issues

Minor issuesStill important but will not affect the overall conclusions• Missing references• Technical clarifications • Unclear labelling of figures and

tables• Spelling, grammar and phrasing

issues

Major issuesEssential points the authors need to address before the manuscript can proceed• Problems with study design • Issues with data or analysis• Experiments not following best

practice

Help the authors improve the manuscript before publicationThese comments can be seen by both editor and authors

Comments to the authors

• Summarize the strengths of manuscript, give positive feedback• Do not mention whether the paper should be published unless the

journal requests this• Be specific so the authors know what they need to do to improve• Number your comments and provide one major comment per

number• Refer to page and line numbers in the manuscript

• Give editors an overall picture about the manuscript and help editors to make their decision

• These comments will not be seen by authors• Do not include detailed concerns about the manuscript which the

authors need to be aware of

Confidential comments to editor

• Comments about the manuscript’s novelty, research significance or suitability for the journal

• Comments not to be shared with the authors‒ Concerns over suspected

author misconduct ‒ Anything that may disclose

your identity‒ Any issues relating to conflicts

of interest

Key questions:• Can your concern be

addressed?• How long will it take to

address your concern?

Recommendation• Methodologically sound

• Fits the journal scope

• Completeness

• Correctness

• Need to redo the study or add significant work

• Major flaws in the methodology • Lack of significance/novelty• Not suitable for this journal

May need no further reviewing

Suitable for publication in its current form

Further reviewingNo guarantee for acceptance

• Grammar/ Format

• Clarifications

• Accept

• Minor revision

• Major revision

• Reject

Style of the report:

• Fair, polite, clearly, and not personal

• Objective & constructive

• Key message is clear and consistent

• Specific not brief

• Check your own spelling and grammar for clarity Cartoon by Nick D Kim, strange-matter.net

Review as you would want to be reviewed

avoid saying… vs. could say…“The writing is terrible.” or “The

writing is so bad, it is practically

unreadable.”

“The authors should revise the language to

improve readability.” or “While the study appears

to be sound, the language is unclear, making it

difficult to follow. I advise the authors improve

the flow and readability of the text.”

“The technical details don’t make

sense.”

“The technical details should be expanded and

clarified to ensure that readers understand exactly

what the researchers studied.”

“The manuscript is fatally flawed.” “The study appears to be missing a key point” or

“Additional work is required to support the

conclusions”.

Example comments

Re-review

• Focus on how well your original comments have been addressed• Avoid raising new concerns unless they have to do with the author’s

revisions• Avoid asking for further experimentation• If the authors decided not to follow your suggestions evaluate their

reasons fairly, don’t take it personally

How to re-review

• The journal may ask you to re-review to confirm whether the points have been addressed

• Consider as part of the commitment of original review• Look carefully at emails

Agree to re-review

Bear in mind…• DO give positive feedback first in your report

• DO keep the focus on the research and not the

author

• BE professional and respectful.

• BE aware of the possibility for bias in your

review

• DO proofread your report and check

everything one more time before submitting it

• DO raise your suspicions in the comments to

editors if you suspect plagiarism, fraud or have

other ethical concerns, providing as much

detail as possible

• DON'T say whether the paper should be

published or not in the comments to authors

• DON'T try to tell the authors exactly how to

revise their manuscript

• DON'T use the review to promote your own

research or hypotheses

• DON'T feel under pressure to detect fraud and

misconduct

• DON'T make comments that could make your

identity obvious to the authors

How to be an effective reviewer?1. It is ok to say no to an invitation

2. Start by getting an overview of the article

3. Consider what is required from each section of the article

4. Pay particular attention to the methods/methodology

5. Look carefully at the tables and figures, including any

supplementary data

6. Start your report with a summary

7. Make it clear which comments are essential

8. Be specific

9. Review as you would want to be reviewed

10. Contact the journal or Editor if you have any questions or problems

Questions?

https://editorresources.taylorandfrancis.com/reviewer-guidelines/

For more information about reviewer training with Taylor & Francis:https://editorresources.taylorandfrancis.com/reviewer-guidelines/peer-review-training/