Journal of Psychosomatic Research 79 (2015) 561565

Contents lists available at ScienceDirect

Journal of Psychosomatic Research

Comparison of self-citation by peer reviewers in a journal with

single-blind peer review versus a journal with open peer review
Alexander W. Levis a, Albert F.G. Leentjens b, James L. Levenson c, Mark A. Lumley d, Brett D. Thombs a,e,f,g,h,i,j,
a
Lady Davis Institute for Medical Research, Jewish General Hospital, Montral, QC, Canada
b
Department of Psychiatry, Maastricht University Medical Center, Maastricht, The Netherlands
c
Department of Psychiatry, Virginia Commonwealth University School of Medicine, Richmond, VA, USA
d
Department of Psychology, Wayne State University, Detroit, MI, USA
e
Department of Psychiatry, McGill University, Montreal, QC, Canada
f
Departments of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, QC, Canada
g
Department of Medicine, McGill University, Montreal, QC, Canada
h
Department of Educational and Counselling Psychology, McGill University, Montreal, QC, Canada
i
Department of Psychology, McGill University, Montreal, QC, Canada
j
School of Nursing, McGill University, Montreal, QC, Canada

a r t i c l e i n f o a b s t r a c t

Article history: Objective: Some peer reviewers may inappropriately, or coercively request that authors include references to the
Received 14 July 2015 reviewers' own work. The objective of this study was to evaluate whether, compared to reviews for a journal with
Received in revised form 14 August 2015 single-blind peer review, reviews for a journal with open peer review included (1) fewer self-citations; (2) a
Accepted 20 August 2015 lower proportion of self-citations without a rationale; and (3) a lower ratio of proportions of citations without
a rationale in self-citations versus citations to others' work.
Keywords:
Methods: Peer reviews for published manuscripts submitted in 2012 to a single-blind peer review journal, the
Peer review
Self-citation
Journal of Psychosomatic Research, were previously evaluated (Thombs et al., 2015). These were compared to
Journalology publically available peer reviews of manuscripts published in 2012 in an open review journal, BMC Psychiatry.
Publishing ethics Two investigators independently extracted data for both journals.
Open peer review Results: There were no signicant differences between journals in the proportion of all reviewer citations that
Blind peer review were self-citations (Journal of Psychosomatic Research: 71/225, 32%; BMC Psychiatry: 90/315, 29%; p = .50), or
in the proportion of self-citations without a rationale (Journal of Psychosomatic Research: 15/71, 21%; BMC
Psychiatry: 12/90, 13%; p = .21). There was no signicant difference between journals in the proportion of
self-citations versus citations to others' work without a rationale (p = .31).
Conclusion: Blind and open peer review methodologies have distinct advantages and disadvantages. The present
study found that, in reasonably similar journals that use single-blind and open review, there were no substantive
differences in the pattern of peer reviewer self-citations.
2015 Elsevier Inc. All rights reserved.

Introduction role of the peer reviewer, which is to provide expert evaluation of a

Peer review is used by biomedical and psychological journals
to evaluate submitted manuscripts and attempt to improve the quality
of those that will be published. Peer review has a central role in the
scientic process, but has been criticized as subjective, inconsistent,
and potentially abused [13]. In addition to these concerns, some peer
reviewers may make inappropriate, or even coercive recommendations
to authors to unnecessarily cite the peer reviewer's own work [46].
Coercive self-citation via the peer review process may inappropriately
highlight the peer reviewer's work and is in conict with the essential
Corresponding author at: Jewish General Hospital, 4333 Cote Ste Catherine Road,
Montral, Qubec H3T 1E4, Canada.
E-mail address: brett.thombs@mcgill.ca (B.D. Thombs).
manuscript's merit, along with recommendations for improvement
[5]. It may also put authors in the difcult position of having to decide
whether to comply with superuous requests or argue against such
requests, which may reduce the likelihood that their manuscript will
be accepted for publication [5,6]. The Committee on Publication Ethics
(COPE) Ethical Guidelines for Peer Reviewers indicate that peer re-
viewers should not suggest that authors include citations to the
reviewer's (or their associates') work merely to increase the reviewer's
(or their associates') citation count or to enhance the visibility of their or
their associates' work; suggestions must be based on valid academic or
technological reasons [7].
We recently evaluated peer reviews of all manuscripts that were
submitted to the Journal of Psychosomatic Research in 2012, in order to
assess the degree to which peer reviewer self-citation occurs [6]. The

http://dx.doi.org/10.1016/j.jpsychores.2015.08.004
0022-3999/ 2015 Elsevier Inc. All rights reserved.
562 A.W. Levis et al. / Journal of Psychosomatic Research 79 (2015) 561565
Journal of Psychosomatic Research uses single-blind peer review [8],
where the identity of peer reviewers is unknown to the authors of sub-
mitted manuscripts. Of the 616 peer reviews included in the study, 171
included at least one peer reviewer citation, with a total of 428 citations.
We found that 29% of all citations recommended by peer reviewers
were citations to the peer reviewer's own work. Self-citations were
more than twice as common in reviews recommending revision or
acceptance, where 33% of all citations were self-citations, compared to
reviews recommending rejection (15%). Furthermore, 21% of reviewer
self-citations were made without any rationale whatsoever, compared
to 5% of citations to others' work [6].
There are three principal forms of peer review: single-blind peer
review; double-blind peer review, where both author and reviewer
identities are concealed; and open peer review, in which the identities
of authors and reviewers are known to one another and, in some set-
tings, reviews are made publically available [8]. It is possible that open
review may discourage unnecessary or coercive recommendations by
peer reviewers to cite their own work, particularly if reviews are
made available to the public [6]. However, no studies have compared
the frequency and nature of peer reviewer self-citation in journals
with single- or double-blind peer review compared to open peer
review.
The objective of this study was to compare peer reviewer self-
citation in a journal with single-blind peer review, the Journal of
Psychosomatic Research, to a psychiatry journal with a similar impact
factor that uses open peer review and makes reviews of published
manuscripts available to the public online, BMC Psychiatry. To do this,
we compared a subset of reviews of manuscripts that were eventually
published in the Journal of Psychosomatic Research from our previous
study [6] to reviews of published articles in BMC Psychiatry. We hypoth-
esized that, compared to reviews of manuscripts accepted for publica-
tion in the Journal of Psychosomatic Research, reviews of manuscripts
accepted for publication in the journal with open peer review, BMC
Psychiatry, (1) would have a signicantly lower proportion of total
citations that were citations to peer reviewers' own work; (2) among
self-citations, a greater proportion would include a rationale for the
relevance of the citation; and (3) the ratio of the proportion of self-
citations without a rationale to the proportion of citations of others'
work without a rationale would be lower.
Methods
The coding manual and all methods used in the present study were
adapted from our previous study of peer reviewer self-citation [6]. How-
ever, because only peer reviews of published articles are provided on-
line by BMC Psychiatry, in the present study we only used data from
peer reviews of manuscripts accepted for publication by the Journal of
Psychosomatic Research, rather than from all peer reviews, as we did in
our previous study.
Journal of psychosomatic research data extraction
The Journal of Psychosomatic Research is an interdisciplinary research
journal that publishes various types of articles focusing on the interplay
between psychology, medical illness and health care. Our previous
study assessed all initial reviews of manuscripts (for full-length articles,
reviews, and short reports) submitted in 2012, excluding: (1) reviews of
manuscripts authored by journal editors who were the investigators of
the previous study [6] and the present study; and (2) peer reviews con-
ducted by study investigators. These exclusion criteria were used in
order to maintain peer reviewer anonymity of the single-blind peer
review process and to avoid conict of interest. The 2012 impact factor
for the journal was 3.3.
Two investigators independently extracted all data, with discrepan-
cies resolved by consensus. For each review, the total number of general
and specic citations was extracted, along with the number of these
citations that referred to work where the reviewer was an author or
co-author, which were classied as self-citations. A specic citation
was dened as having adequate information to nd a unique publica-
tion in a search of multiple electronic databases (e.g., specic author,
journal, year), whereas general citations referred more broadly to mul-
tiple works of an author or cohort of authors (e.g., a request to cite work
on a topic by an author without a journal or year). A specic citation was
coded as a self-citation if, after locating the cited publication in a search
of multiple electronic databases, the reviewer was an author or co-
author on the article. A general citation was coded as a self-citation if,
in a search of all work by the cited investigator or team, the peer review-
er was identied as an author or co-author on any published articles
that may have been referred to in the generally cited research. For
all self-citations, we noted whether the peer reviewer's own name
was listed in the review or whether a search was needed to conrm
the self-citation. In addition, we documented whether the peer
reviewer self-citation was used by the study authors in the nal
published article.
For each citation, we also coded whether the reviewer provided a ra-
tionale for including the citation. A rationale was dened as any indica-
tion given by the reviewer that the citation was included because
(1) authors failed to properly attribute facts or arguments presented
in the manuscript, (2) the citation addressed specic relevant informa-
tion missing from the manuscript, or (3) it addressed specic errors or
inaccuracies in the information presented in the manuscript. For the
present study, we included a sensitivity analysis for all comparisons
between journals, in which we excluded one peer review from the
Journal of Psychosomatic Research. Notably, this one review contained
11 self-citations, none including a rationale. This outlier review
was identied by the editors and not forwarded to the study authors
or taken into consideration in the evaluation of the submitted
manuscript.
BMC psychiatry data extraction
The open review journal BMC Psychiatry was selected due to its gen-
erally similar scope to the Journal of Psychosomatic Research and because
all peer reviews of manuscripts published in the journal are freely avail-
able online in the pre-publication history section of each article's
webpage [9]. BMC Psychiatry is an open access journal that focuses on
psychiatric disorders and related genetics, pathophysiology, and epide-
miology. The impact factor in 2012 was 2.2. All articles of the type
Research Article published in 2012 were downloaded, and for each
article, the initial review for all reviewers was also downloaded. In
cases where information was missing from the website (e.g., peer re-
view document present but with no reviewer comments), the journal
was contacted by email to request the information. Following the
methods of the previous study described above [6], citations in each
review were classied as either general or specic, a self-citation or a
citation to the work of others, and as providing a rationale or not.
Again, two investigators independently extracted and coded the
reviews, and discrepancies were resolved by consensus.
Statistical analysis
We compared (1) the proportion of citations that were self-citations
and (2) the proportion of self-citations that did not include a rationale in
the Journal of Psychosomatic Research versus BMC Psychiatry with two-
tailed Fisher's exact tests and = 0.05. We compared the proportions
of self-citations without a rationale (out of all self-citations) versus cita-
tions to others' work without a rationale (out of all citations to others'
work) between the two journals using a z-test for difference in log
(risk ratio) [10], also with = 0.05. All statistical analyses were
performed using the R statistical package, version 3.1.3 [11].
 A.W. Levis et al. / Journal of Psychosomatic Research 79 (2015) 561565 563

Fig. 1. Flow diagram of selection of peer reviews for the Journal of Psychosomatic Research in 20121.

Results A comparison of citation patterns in the Journal of Psychosomatic

Research and BMC Psychiatry is provided in Table 2. There was no signif-
The Journal of Psychosomatic Research received 298 article submis- icant difference in the proportion of citations that were self-citations in
sions in 2012, excluding articles authored by journal editors who were
investigators of the previous peer reviewer self-citation study [6] and
Table 1
the present study, as well as articles where all reviews were performed Distribution of peer reviews by number of self-citations.
by study investigators. Of these, 155 articles were eventually accepted
for publication by the journal. These 155 papers were associated with Number of peer reviews
(% of total reviews)
334 peer reviews and 304 peer reviewers, where reviews by original
study investigators were excluded (see Fig. 1). In the text of these Number of self-citations Journal of Psychosomatic BMC
in peer review Research Psychiatry
reviews, there were 225 total citations made by peer reviewers, of
which 13 (6%) were general citations, and 212 (94%) were specic 0 291 (87.1) 386 (86.7)
1 31 (9.3) 47 (10.6)
citations.
2 8 (2.4) 7 (1.6)
There were 196 research articles published by BMC Psychiatry in 3 0 (0.0) 1 (0.2)
2012 that were associated with 445 peer reviews by 416 peer reviewers. 4 2 (0.6) 0 (0.0)
A total of 315 citations were made in peer reviews, of which 49 (16%) 5 1 (0.3) 2 (0.4)
were general, and 266 (84%) were specic citations. Table 1 shows the 6 0 (0.0) 0 (0.0)
7 0 (0.0) 1 (0.2)
number of self-citations per peer reviews for both journals.
8 0 (0.0) 0 (0.0)
9 0 (0.0) 1 (0.2)
10 0 (0.0) 0 (0.0)
11 1 (0.3) 0 (0.0)
1
There is not a ow diagram for BMC Psychiatry because all research articles published
Total 334 445
in the journal in 2012 were included.
564 A.W. Levis et al. / Journal of Psychosomatic Research 79 (2015) 561565
Table 2
Summary of peer reviewer citation results.
Total number of reviews included
Total number of citations
Number of self-citations (%)
Number of self-citations including a rationale (% of self-citations)
Number of self-citations with no rationale (% of self-citations)
Number of citations to others with no rationale (% of citations to others)
a
p-Values are shown for tests of differences between the two journals in proportions of self-citations among total citations, and proportions of self-citations with no rationale among all
self-citations
the Journal of Psychosomatic Research (71/225, 32%; 60/214, 28% in sen-
sitivity analysis, excluding one review with 11 self-citations from the
Journal of Psychosomatic Research) versus BMC Psychiatry (90/315, 29%,
p = .50 in main analysis; p = .92 in sensitivity analysis). Similarly,
there was no signicant difference in proportion of self-citations that
did not include a rationale in the Journal of Psychosomatic Research
(15/71, 21%; 4/60, 7% in sensitivity analysis) versus BMC Psychiatry
(12/90, 13%, p = .21; p = .28 in sensitivity analysis).
The proportion of self-citations without a rationale was signicantly
higher than the proportion of citations to others' work without a rationale
in both the Journal of Psychosomatic Research (self-citations = 15/71, 21%;
other citations = 5/154, 3%; p b .001) and BMC Psychiatry (self-citations =
12/90, 13%; other citations = 9/225, 4%; p b .005). There was no signi-
cant difference between the two journals in the risk ratios of not including
a rationale in self-citations versus citations to others (p = .31; p = .53 in
sensitivity analysis).
Discussion
The main ndings of this study were that there were no signicant
or substantive differences between a journal that uses single-blind re-
view, the Journal of Psychosomatic Research, and a generally similar jour-
nal that uses open review, BMC Psychiatry, in (1) the proportion of all
citations made by peer reviewers that were self-citations, and (2) the
proportion of self-citations that included a rationale. In addition, there
was no signicant difference between the journals in the ratio of the
proportion of reviewer self-citations without a rationale to the propor-
tion of citations to others' work without a rationale.
The decision of whether a journal should employ blind or open
review is an important one and should reect a consideration of the
advantages and disadvantages of each method. Single-blind review
may allow reviewers to be more appropriately critical since reviewers
do not fear negative judgment of inuential authors, and important
relationships of investigators in the same eld will not be affected by
criticism in the review process [2,12,13]. On the other hand, blind peer
review has been criticized because of the secrecy of the process and
because reviewers are not held accountable for unfair or prejudicial
assessments. In single-blind review, the lack of liability for their own be-
havior may lead some reviewers to be excessively negative or rude in
reviews of rivals in a reviewer's eld [2,12,14,15]. Open review is inher-
ently more transparent, increases reviewer accountability, and might
better attribute credit to peer reviewers who put substantial time and
effort into conducting reviews [2,12,14,15]. However, this same open-
ness could lead to overly positive reviews of manuscripts written by
well-known or generally respected investigators in a eld, or could
lead to opinions being withheld due to reviewers' fear or self-interest,
thereby reducing the quality of reviews [2,12,13]. We previously pro-
posed that another advantage of open review might be a reduction
in the tendency of some peer reviewers to inappropriately request cita-
tion of their own work [6]. Based on the results of the present study,
however, this does not appear to be the case. Our ndings are in line
with results from randomized controlled trials that have shown that
peer reviews in an open review context are no different in quality
from reviews in which reviewers remain anonymous [12,16,17].
Journal of Psychosomatic Research BMC Psychiatry p-Valuea
334 445
225 315
71 (32) 90 (29) .50
56 (79) 78 (87)
15 (21) 12 (13) .21
5 (3) 9 (4)
There are important limitations to the present study. First, we did
not conduct a randomized controlled trial, in which reviewers would
be randomized to a blind review group or to an open review group.
Thus, in the present study, any difference or lack thereof in the two
sets of reviews may have been due to unique characteristics of the
Journal of Psychosomatic Research and BMC Psychiatry that do not relate
to whether they use single-blind versus open peer review. Neither
journal addresses self-citations in their instructions for peer reviewers
[6,18], but the journals do have somewhat different content scopes,
different pools of peer-reviewers, as well as, undoubtedly, some differ-
ences in the processes that reviewers must undergo to complete a
review. Each of these factors could potentially impact the pattern of ci-
tations made in reviews for the journals. Second, we included reviews of
manuscripts submitted in 2012 to the Journal of Psychosomatic Research,
but articles published in 2012 in BMC Psychiatry, so there was a small
offset in the timing of manuscript submission and publication. Third,
the comparisons between journals of self-citation proportions were
based on relatively small numbers. Only 15 self-citations without a
rationale were observed in the Journal of Psychosomatic Research (or 4
in the sensitivity analysis), and 12 were observed in BMC Psychiatry.
Fourth, it is not clear to what degree our results would generalize to
other psychiatry journals, including journals with higher or lower im-
pact factors. It may be that higher impact factor is associated with higher
quality peer reviewers, or more stringent review procedures that
would limit the number of self-citations. Fifth, only reviews for pub-
lished articles were readily accessible through the BMC Psychiatry
website. Thus, we could only compare reviews of manuscripts that
were eventually published and could not examine potential differences
in manuscripts that were not published. Sixth, we do not have data that
would help us understand why peer reviewers may decide to cite their
own work or not. Thus, we do not know why there was not a difference
between blind and open peer review in self-citation patterns. It is plau-
sible that reviewers are not familiar with guidelines for peer reviewers,
such as the COPE guidelines [7], and do not know that unnecessary self-
citation goes against recommendations in these guidelines, but we do
not know if this is the case. Seventh, as we noted in our previous
study [6], our methods for determining whether a rationale was provid-
ed for a citation were very conservative. We considered any reason or
explanation as a rationale, regardless of whether it was brief or seemed
redundant at rst glance (for examples, see the Appendix A). Ideally, we
would be able to ascertain whether a citation was actually justied and
appropriate, but the ability to do this depends on specic content
knowledge for each article that typically goes beyond the knowledge
of an editor and, in this case, the members of our research team. Eighth,
we did not record agreement between investigators who extracted
data. Finally, there is not an agreed upon term for labeling potentially
coercive peer reviewer recommendations to cite the reviewer's own
work. We used the term coercive self-citation based on its previous
use in a study that described inappropriate requests by editors to add
citations to the editor's own journal without any rationale [19].
In sum, superuous or coercive peer reviewer self-citation can
improperly highlight the work of the peer reviewer and can negatively
impact the ability of authors to work efciently and effectively. We previ-
ously hypothesized that open peer review may reduce the likelihood that
 A.W. Levis et al. / Journal of Psychosomatic Research 79 (2015) 561565 565

peer reviewers will inappropriately request that authors cite their own Appendix. A (continued)
(continued)
work. However, the present study found that there were no signicant With rationale Without rationale
or substantive differences in the pattern of self-citations made by re-
non-participation rates may inuence
viewers in reasonably similar journals that use single-blind versus open results, and I would like to see a
peer review. discussion of this.

Conicts of interest The introduction neglects a number of

studies that have found an increased risk
CVD risk among youth with bipolar
All authors have completed the Unied Competing Interest form at disorder
http://www.icmje.org/coi_disclosure.pdf and declare that no authors
have any conict of interest disclosures for the past 3-year reporting Some discussion of other studies that have
period. examined the role of personality in cancer
outcomes would help to place the
framework within which this study was
Acknowledgments carried out

Dr. Thombs was supported by an Investigator Salary Award from the

Arthritis Society (INS-13-001). There was no specic funding for this References
study, and no funders had any role in the study design; in the collection, [1] T Jefferson, M Rudin, S Brodney Folse, F Davidoff, Editorial peer review for improving
analysis, and interpretation of data; in the writing of the manuscript; or the quality of reports of biomedical studies, Cochrane Database Syst. Rev. 18 (2007)
in the decision to submit the manuscript for publication. MR000016.
[2] R Smith, Opening up BMJ peer review: a beginning that should lead to complete
transparency, BMJ 318 (1999) 4.
Appendix A. Examples of Citations With and Without Reviewer [3] M Atkinson, Peer reviewculture, Sci. Eng. Ethics 7 (2001) 193204.
Rationales (Originally published in Thombs et al., Journal of [4] DB Resnik, C Gutierrez-Ford, S Peddada, Perceptions of ethical problems with
scientic journal peer review: an exploratory study, Sci. Eng. Ethics 14 (2008)
Psychosomatic Research, 2015) 305310.
[5] B Thombs, I Razykov, A solution to inappropriate self-citation via peer review, CMAJ
184 (2012) 1864.
[6] BD Thombs, AW Levis, I Razykov, A Syamchandra, AF Leentjens, JL Levenson, et al.,
Potentially coercive self-citation by peer reviewers: a cross-sectional study,
With rationale Without rationale J. Psychosom. Res. 78 (2015) 16.
[7] Committee on Publication Ethics, COPE Ethical guidelines for peer reviewers, www.
The authors have not included more The authors have not included some publicationethics.org/resources/guidelines (Accessed July 1, 2015).
recent citations that are relevant relevant citations [8] CJ Lee, CR Sugimoto, G Zhang, B Cronin, Bias in peer review, J. Am. Soc. Inf. Sci.
Technol. 64 (2013) 217.
Other studies have used different cut Please cite. [9] BMC Psychiatry, www.biomedcentral.com/bmcpsychiatry/ (Accessed July 1, 2015).
points. [10] DG Altman, JM Bland, Interaction revisited: the difference between two estimates,
Prevalence rates should be included Other relevant citations include. BMJ 326 (2003) 219.
[11] R Core Team, R: a language and environment for statistical computing, R Foundation
The citations in the manuscript focus The following citations were not for Statistical Computing, Vienna, Austria, 2012 (www.R-project.org/).
largely on North American sources, but included [12] S van Rooyen, T Delamothe, SJW Evans, Effect on peer review of telling reviewers
that their signed reviews might be posted on the web: randomised controlled
should also include European studies
trial, BMJ 341 (2010) c5729.
[13] K Khan, Is open peer review the fairest system? No BMJ 341 (2010) c6425.
The study ndings should be discussed in The authors could include these [14] T Groves, Is open peer review the fairest system? Yes BMJ 341 (2010) c6424.
light of a recent study by references [15] EC Moylan, S Harold, C O'Neill, MK Kowalczuk, Open, single-blind, double-blind:
which peer review process do you prefer? BMC Pharmacol. Toxicol. 15 (2014) 55.
The authors may consider 4 trajectories, It appears that the authors are not [16] S van Rooyen, F Godlee, S Evans, N Black, R Smith, Effect of open peer review on
as reported by familiar with another study on this quality of reviews and on reviewers' recommendations: a randomised trial, BMJ
topic 318 (1999) 2327.
[17] S Vinther, OH Nielsen, J Rosenberg, N Keiding, TV Schroeder, Same review quality in
The authors state that little is known open versus blinded peer review in Ugeskrift for Lger, Dan. Med. J. 59 (2012) A4479.
about frequency and associations ofbut [18] BMC Psychiatry, Guide for BMC Psychiatry reviewers, www.biomedcentral.com/
omit a study by bmcpsychiatry/about/reviewers (Accessed July 1, 2015).
There are studies on how [19] AW Wilhite, EA Fong, Coercive citation in academic publishing, Science 335 (2012)
542543.