Getting peer review right

A guide for early career researchers

1

Getting Peer Review Right

Todays Speakers

Verity Warne
Wiley

John Langley
University of
Southampton

Michael Willis
Wiley and ISMTE
president-elect

Paul Trevorrow
Wiley

J. Matthias Starck

Emily Jesper

University of Munich (LMU)

Sense About Science

2

What is peer review, and why should I

care about it?
Michael Willis, Senior Manager, Peer Review, Wiley

[Cartoon by Nick D Kim, scienceandink.com. See website for terms of use.]

What is peer review not?

confined to scholarly research (it exists throughout society)

the entire process of validating research (its just part of it)

a perfect system (its only as strong as those engaged in it)

a one size fits all mechanism (there are different models)

What is peer review?

peer review is the central pillar of trust [in scholarly
communication].
On the one hand, researchers want to be published in
journals that have robust peer review and, on the
other, they feel secure in citing peer reviewed
material.
Trust and authority in scholarly communications in the light of the digital
transition, University of Tennessee and CIBER Research Ltd, December 2013

Peer review: a tool to evaluate

and improve
evaluate
is the research valid?
is the research original?
is the research significant?

improve
are there any omissions?
could the methods be reproduced?
have the authors overstated their conclusion?
is it written well?

Why should I volunteer to peer

review?
it helps you develop critical appraisal skills
it helps you to see your own research through the
lens of a reviewer
theres a demand
its good for your CV
its invaluable experience if youre interested in
publication of research

Conduct your
own research

Inform your
own research

The virtuous circle

of peer review and
research community
engagement

Critique the
research
undertaken
by others

Find out
what others
are doing in
your area of
research

Review
manuscripts
in your area
of research

The critical, constructive advice I received

enabled me to improve my work and to
progress as an academic.

The paper I had to review was absolutely

terrible. Yet, it still helped me to see errors in
my own work.

Testimonials
In my experience, the postdoc is often the
most valuable peer reviewer because he/she is
hot from the grilling of a Ph.D. programme, is
still excited about trying to keep up in the field,
and eager to please.

The Editors Perspective

Prof. Dr. J. Matthias Starck

Editor in Chief The Journal of Morphology
11

Its All about Science

Publications communicate science
Science should be:
innovative => present novel information
based on testable hypotheses
reproducible by anybody who wishes to do so =>
provide all information on material and methods to
enable peers repeating your experiments
discuss all possible interpretations
consider and refer to the work of others (following
ethical standards)

12

Decision making
Peer reviewing is a collegial process honesty and best ethical
practice are the foundations
Decisions about a manuscript are made by the section editor or
editor in chief, depending on the organization of a journal NOT
by the reviewers.
The editor / editor in chief takes the final responsibility for the
contents of a journal
Decisions include:

(1) all reviewer recommendations

(2) aims and scope of the journal
(3) novelty of research
(4) scientific and ethical standards
(5) integrity of the academic record
(6) quality of the journal / impact of papers
13

Reviewing guidelines
General
Are YOU the reviewer competent to review ALL aspects of the
manuscript. If your expertise is restricted to certain aspects of the
paper, please inform the editor immediately, so that additional
expert reviewers can be invited.
Are you unbiased (in any direction; positive as well as negative)?
Please inform the editor about any conflict of interest, potential
bias, interaction with the authors.
If you cannot review the manuscript in the requested time, please
inform the editor and ask for extension. Also, if for any
unforeseen reason your review is delayed, please inform the
editor and give a date when the review can be expected. Never
stop communicating with the editor / editorial office!
Do NOT communicate with the authors directly. Any
communication between reviewer and author that bypasses the
editor has the potential to corrupt editorial decisions.
14

Reviewing guidelines
General (cont.)
Is the research novel / innovative so that it deserves publication?
Is the data new and has it not been published anywhere else?
Are there any concerns about plagiarism or redundant publication?
Is there any question of violating good scientific practice, or human / animal welfare?
Is the language acceptable?
This concerns also figures. Please inform as if there is any suspicion of duplicate publishing.
Most journal can check using professional plagiarism detection software.
Most Wiley Journals follow the codes for publication ethics as developed by the Committee
of Publication Ethics (COPE for authors: http://publicationethics.org/about/guide/authors).
It is NOT the reviewers duty to correct language, or act as a copy editor. However, we are
grateful for your overall evaluation whether or not the written English is acceptable to
communicate the contents. Any advice to the authors will be welcome. If you are non-native
English speaking and do not wish to comment on the language, please indicate so in your
review.
15

Reviewing guidelines
Abstract and keywords
The abstract reports the key hypothesis, the main method(s)
and the key results!
Is the abstract concise, technical and informative?
The abstract should NOT report numbers or statistics.

Keywords should NOT be redundant with the manuscript title.

They represent additional search items that increase the
visibility of the paper and help finding it using online search
engines.

16

Reviewing guidelines
Introduction
The introduction presents the scientific justification for conducting the research
and provides explicit hypotheses that are tested in the paper.

Does the introduction presents a clear and explicit reason why the study was
conducted?

Does the introduction contains explicit hypotheses?

Does the introduction refers to the important previous studies in the field
(excessive references should be avoided)

Is the introduction presented in a concise style? Long philosophical

excursions are usually not necessary to introduce a morphological study.

Do results and discussion relate to the hypotheses presented in the

introduction?

Nothing is known about is not a valid reason to conduct a study and

certainly no hypothesis to test. Curiosity certainly drives science but it does not
replace hypotheses that can be tested using appropriate methods.

17

Reviewing guidelines
Material and Methods
Material and Methods should report in every detail about the
materials and methods used, so that the study is fully
reproducible for anybody who wish to do so.
Has the origin of the material, as well as the depository (where
the raw material is deposited) given in full detail (e.g., catalog
numbers for museum specimens; digital data depositories)?
Are necessary permits and ethical clearing given (collection
permit, animal experimentation).
Is sample size (for each method applied) made explicit?
Is sample size sufficient to conduct tests, exclude
pseudoreplications and support conclusions?
Are the methods employed (including statistics) appropriate to
test the hypotheses?
Are methods described in all necessary details?
18

Reviewing guidelines
Results
The results section should present ONLY results from the study. NO
references to earlier studies or published data from the literature should
be made in the results section. It must be explicit and clear what is own
and what is others research.
Does the results section reports data from all methods used?
Are all methods used described in the Materials and Methods section?
Are all results appropriately documented by text, figures, graphs, and
tables? Is the image quality appropriate?
Do images show the details reported in the text?
Are results presented in non-redundant and concise manner?
Can details for a specialist be moved to an online appendix?
Have you checked and reviewed supplementary online material?
An online appendix is an integral part of the paper and therefore can be
cited. All material presented in an online appendix is equally protected
by the authors copy right.
19

Reviewing guidelines
Discussion
The discussion should relate the results of the study to
previously published data. Repeating the results is not
necessary and not wanted.
Is the relevant literature covered?
Are results appropriately discussed and are the conclusions
sufficiently supported by the results?
Is the discussion presented in the most concise style?

20

Reviewers suggestions
Accept
Minor revision
Major revision
Reject (but allow resubmission)
Reject (no option for resubmission)

21

Accept
All review criteria for publication met
Reviewer should also provide positive statements (!) to all questions /
criteria relevant for publication.
It is work YES! And often even more difficult. Use our guidelines and
checklists to provide a complete review. When positive information is
not provided the editor does not see if you have considered all
important aspects.
Be always critical. It is easier to handle excessive critique than false
positive reviews.

22

False positive review

Worst Case Scenario for the Editorial Process
Authors will be disappointed a paper is rejected despite
positive reviews

Reviewers will be disappointed because their review is not

appreciated / taken into account. => Waste of time

Editor will be disappointed because the reviews were useless

and because authors are disappointed. There is alwas
additional work and rejection despite positive reviews requires
intensive reasoning.

23

Minor revision
This recommendation is generally made when FORMAT
CHANGES are required:
The research is OK and needs NO change
Language and presentation may require change
Some paragraphs may require rewriting (shortening, more
concise presentation, possibly literature added)
Figures and graphs may require format changes

NO CHANGES of content (text, results, figures)

Usually no re-review necessary

24

Major revision
This recommendation is made when changes of content are
necessary:
There is not enough material presented (add more
specimens)
Statistics are not appropriate and require corrections,
improvements
Additional research needs to be added to support better the
results of the paper
Figures need to be added / improved

This decision generally results in re-review of paper

because contents need to be changed
25

Reject
Paper fails to meet acceptance criteria of the journal because:
Topic inappropriate for the journal
No hypotheses tested (plain descriptive / narrative)
Research cannot be reproduced because details of material
and methods are missing
Lack of appropriate ethical clearing
Experiments / research flawed by application of incorrect
methods
Statistics not appropriate
Sample size not sufficient to support statements
26

How to write a great peer review

Dr John Langley and Paul Trevorrow
27

How to write a great peer review:

Rationale
Learn how to provide a top quality evaluation that is useful for
the authors in improving their manuscript, and editors in
making their decisions.

28

Structure of the session

1. Referees questionnaire
2. Narrative report for the author
3. Comments to the editor

29

30

The referees questionnaire

Novelty
Concise
Comprehensive
Accuracy
Abstract
Citations
Language

Structure

Decision

31

The narrative report

Typically in 2 parts:
1. A narrative report for the author
2. Confidential comments for the editor

32

The narrative report for the author

STRUCTURE OF THE REPORT
1. Summary
2. Major issues
3. Minor issues
4. Reflect on the questionnaire
5. Opinion

33

The narrative report for the author

1. THE SUMMARY
This article addresses assumptions made in previous
literature and methodically discusses these with
carefully designed experiments and discussion to
highlight these issues.
The findings improve on previous applications of the
FABMS technique, reveal new matrix effects and
ionization efficiencies which should be considered in
any FABMS ligand-cation investigation.

What is the paper about

and key findings?

Where does it fit in the

scientific literature*
Is it novel, how/why?

This is a quality submission, the work is well written albeit Quality of the work,
what are its
suffering from inaccuracies of structure, styling,
strengths/weaknesses?
nomenclature and presentation.
The findings will bring attention to previous FABMS
Any significant
cation-ligand studies where such considerations are
considerations in the
absent.
field?
34

The narrative report for the author

2. MAJOR ISSUES
The experimental design and methodology are strong and well presented
Prior work by Geraud in this area has been omitted see: Matrix
influence on stability constants of 15-crown-5-alkali metal complexes
measured by liquid secondary ion mass spectrometry (pages 169
175), Organic Mass Spectrometry 1994, 29, 169. The author should
state where the current work stands in relation to Gerauds findings.
The structure of the manuscript is not in accordance with the journal style,
the experimental section must be placed after the introduction.
The script should be re-written in the third person, passive voice. Avoid
we proposedwe notedwe chose. Pages 1,3 and 5 most affected.

35

Flaws in design,
interpretation of
results

Similar
unacknowledged
work

Nomenclature,
representational
flaws, language,
structure, data
presentation

Are there any

ethical issues?*

The narrative report for the author

3. MINOR ISSUES
FABMS should be defined at first use.
The abstract refers to protonated ions [M + H]+ and metallated ions [M +
Metal]+ this is incorrect. This should be protonated molecules and
cationised molecules respectively, a protonated or cationised ion would
be doubly charged.
All figures, whilst the ions of interest are clearly annotated the charge is
missing from all annotations. Ions are correctly described in the
manuscript, i.e. [M + H]+ but incorrectly written in the figures [M+H], note
the spaces before and after the + are important.
Full bibliographic data needs to be included in the references. Some are
missing volume and issue data.
Text corrections
Page 1, line 27: this sentence should terminate with a colon and (i) and (ii)
each start on a new line.
Page 5, line 41: protocol should be protocols
36

Sections with
unclear meaning?

Factual errors?

Figure quality /
correctly labelled?

Citations correct
and present? Are
they balanced?

Text corrections

The narrative report for the author

4. REFLECT ON THE QUESTIONNAIRE

37

The narrative report

5. OPINION
Summarise your opinion of the work
Suitability for target journal?
Recommendation*

A scientifically significant paper that addresses inaccuracies in literature

approaches. These are of particular interest to the readership of this journal. The
manuscript requires modification of its tone and presentation to meet the stylistic
criteria of the journal.

38

The narrative report

Comments to the editor

Elaborate further on opinion

Elaborate on concerns regarding any suspected misconduct
Include any evidence that is omitted from the narrative report
through necessity, for example; because it might identify you
Declaration of ability/inability to review any parts of the
manuscript
Conflict of interest*

39

Major / minor revision?

What is the difference

40

The Ethics of Peer Review

Emily Jesper, Assistant Director,

Sense About Science

The problem has been that people have believed the myth that science is a
pure objective activity.
It is not. Its a human activity and its prone to all the joys and downsides of
being a human activity. Weve fooled ourselves!
Dr Richard Smith, ex Vice Chair of COPE (Committee on publication ethics)

Reviewers need to be trustworthy and honest

Dr Irene Hames, Independent Publishing Consultant

Fake reviewer cases

Irene Hames, Sense About Science, Nov 2014

Reviewers role?

Keep it confidential.
If you plan to ask a colleague to do the review, ask the journal first and
make sure they receive the credit
Destroy submitted manuscripts after you have completed the review.

Provide a timely review (reviewers shouldnt slow down publication to enable

them to get a paper out first!)

Declare conflict of interest, either real or

perceived
Financial activities
Grants
Consulting fees or honoraria
Support for travel to meetings
Payment for writing manuscripts
Stock options
Royalties
Patentand the list goes on

Reviewers should disqualify themselves if they feel unable to provide an honest

and unbiased assessment.

When you review

Is it clear to the reader who funded the study? (If there is no specific funding,
then this should be stated.)

Dont make hostile, insulting or defamatory remarks. Rather support your

points with evidence.

Dont request that the author cites your own papers, (unless there is a strong
scholarly rationale for this).

Ask: Was the research ethical?

If the study involves humans, human tissues or animals

Was ethics approval gained?
Was the study ethical?
WMA Declaration of Helsinki - Ethical Principles for Medical Research
Involving Human Subjects
http://www.wma.net/en/30publications/10policies/b3/

Peer reviewers can help identify research

misconduct
data fabrication;
falsification
plagiarism
image manipulation
unethical research
biased reporting
authorship abuse
redundant or duplicate publication;
undeclared conflicts of interest

Can reviewers help detect plagiarism &

redundant publication (self plagiarism)?

Let the editor know if a manuscript you received overlaps significantly with
one that has already been published.
Journals use Cross Check: Plagiarism screening
Reviewers can check on eTBLAST http://etest.vbi.vt.edu/etblast3/

eTBLAST http://etest.vbi.vt.edu/etblast3/

@senseaboutsci

COPEs new Ethical Guidelines for Peer Reviewers: background, issues, and evolution,
http://c.ymcdn.com/sites/www.ismte.org/resource/resmgr/files/hames_article.pdf

Irene Hames, Sense About Science, Nov 2014

Resources

Best Practice Guidelines on Publishing Ethics

http://exchanges.wiley.com/ethicsguidelines

Peer review: The nuts and bolts

www.senseaboutscience.org/resources.php/99/pee
r-review-the-nuts-and-bolts

Committee on Publication Ethics

www.Publicationethics.org

Is it peer reviewed?
Share this useful first question with
the public

Dont let peer review be your best

kept secret!

http://bit.ly/1GeEF8c

Questions?

54

Interested in becoming a Wiley Advisor?

Wiley Advisors are early career researchers and professionals
who serve as a voice for their communities. They provide ideas,
insights, and feedback as we develop new products and
services. In exchange, they are eligible for special discounts,
events, contests, networking opportunities and more.

Visit http://exchanges.wiley.com/advisors for more information

or connect with us on twitter @WileyAdvisors

Thank you!
To keep up to date with our forthcoming
webinars, visit

https://www.brighttalk.com/channel/11201

56