How to Review a Paper

Derek K. Oler
derek.oler@ttu.edu

William R. Pasewark
w.pasewark@ttu.edu

Texas Tech University

December 22nd, 2012

Abstract

This paper introduces the review process to new researchers, describes the institutional
background for reviews, and provides advice to reviewers. The review process is a
critical element of the dissemination of research that ultimately provides credibility to the
body of academic knowledge. This process relies on expert, anonymous reviews by
established researchers (typically those who have already published in top journals on
similar topics), but the scarcity of expertise combined with anonymity results in agency
problems. Invitations to review should only be accepted if the researcher has the skill,
time, objectivity, and inclination to do a quality review. The objectives of a quality
review are (1) to assist the editor in his/her gatekeeping role and (2) to assist the authors
in improving their work. A quality review should consider the contribution, execution,
and conclusions of the paper. We conclude by discussing pitfalls that reviewers should
avoid.

We thank Denton Collins and Dana Hermanson for helpful suggestions and comments.

Electronic copy available at: http://ssrn.com/abstract=2194232

1. Introduction

The intent of this paper is to facilitate and improve the review process by discussing

the advantages and weaknesses inherent in the process and by providing

recommendations on how a quality review can be executed. We build on prior

commentaries and advice from accounting researchers (e.g., Moizer, 2009; Bailey et al.,

2008; Kachelmeier, 2004; Omer et al., 2004) by providing greater detail on the

mechanics of the review process and its institutional background and by providing

specific, concrete guidance on how to conduct a quality review. This paper can be

especially useful to PhD students and new faculty, but may also be helpful to seasoned

faculty as they consider future reviewing activities.

In the academic accounting discipline journal publications are the main channel for

the distribution of new ideas. Journal publications are also the most important factor in

tenure decisions at top research schools, and even at schools more oriented towards

teaching, publications play a significant role. Academic journals generally require peer

reviews of submissions before they are published, making the review process one of the

most important tasks in the academic profession.

Waymire (2012) argues that developing reviewer talent benefits the discipline by

encouraging greater innovation by researchers. Yet, in spite of the singular importance of

reviews, many journals provide only brief guidance to their reviewers (and others provide

none at all).1 The purpose of this paper is to introduce new researchers to the review

process by describing the institutional background of the review process and providing

advice on how to do a quality review.

1
Generic review guidance does exist, for example, Pyrczak (1999).

Electronic copy available at: http://ssrn.com/abstract=2194232

 Reviews play a critical role because they (1) assist the editor in fulfilling a

gatekeeping role, and (2) assist the authors in improving their research. However, the

review process is subject to considerable agency problems because reviewers are

anonymous to the authors and because there is often a shortage of qualified reviewers

(Gilmore et al., 2006).2 Accordingly, the review process is vulnerable to failure because

reviewers can shirk with little or no immediate negative consequences.3 In spite of these

difficulties, there is no obvious alternative to the review process, and at present no major

journal has proposed a substitute for the traditional review process. Accounting

academics have a vested interest in ensuring that the review process works as best as it

possibly can. Accordingly, with this paper, we seek to improve the way we build

knowledge by facilitating one of our critical academic processes.

In this paper we discuss how the review process can be impaired when reviewers

accept review opportunities for which they lack the skill, time, objectivity, or interest.

We also demonstrate how researchers should not use reviewing opportunities to dish

some back out or to make the reviewer look smart at the authors expense; rather, they

should be viewed as a chance to participate in honing cutting edge research.

In this paper we show how bad reviews potentially hurt the journals reputation,

frustrate editors, and discourage new research. On the other hand, good reviews enhance

the journals reputation, make the editors job easier, and encourage innovative and

interesting research. In our experience, even when the editorial decision was

unfavorable, receiving a thoughtful review was encouraging and helpful.

2
Agency problems refers to problems from individuals maximizing their own utility at the expense of
others. For example, a manager may maximize his personal utility by surfing the web during work, and
this hurts shareholders who bear the cost of his salary.
3
As we discuss later, there are significant longer-term reputational consequences to providing a poor
review.

3
 We attempt to provide new reviewers with direction on providing review comments

regarding: (1) the contribution, (2) the motivation and literature review, (3) hypotheses

and theoretical support, (4) data, analysis, and presentation, and (5) the conclusions

drawn. We recommend beginning a review with a quick overview, focusing first on the

abstract and introduction (where the contribution and motivation can be broadly

assessed), and then a detailed read of the text and tables. Once the reviewer collects

his/her initial thoughts, often on a scratch pad or a rough electronic draft, we recommend

setting the review aside for a day or two before finalizing the review and returning it to

the editor.

The final portion of this paper identifies pitfalls that the reviewer should try to avoid.

For archival research, these include unsupported assertions of bias, which often manifests

as a correlated omitted variable. There will always be some sort of bias in the data or

analysis (and often multiple forms of bias), and potential bias should be addressed by

considering (1) whether the bias works for or against the hypotheses, and (2) how strong

of an effect that bias might have. Experimental work may also sometimes face the

criticism that the (rarely helpful) setting is not realistic.

2. Background

Ellison (2011) contends that journals have two functions, the dissemination of

knowledge and quality certification.4 The publication process involves authors, who

produce papers and are ultimately responsible for their content; editors, who make the

ultimate decision on papers that are accepted for the journal; and reviewers, who help the

4
Ellison (2011) also notes the review process may be in decline because of increased use of electronic
media such as Social Sciences Research Network (SSRN). Thus, authors (especially established authors)
feel less need to utilize journals to disseminate their work.

4
editor make his decision. Reviewers might not catch all of a papers flaws (Spiegal,

2012), but they still play a vital role in the quality assurance aspect of the journal.

Informal faculty and PhD student discussions sometimes gravitate to a critique of the

review process, especially if a participant was recently on the receiving end. As the top

accounting academic journals generally have acceptance rates of around 10% (or less),

the outcome of most submissions is, unfortunately, negative, and it is easy to blame the

reviewer(s) when this happens. However, when the sting of rejection has passed, it is

useful to consider the broader picture of why we do reviews in the first placeand why

the review process continues to be utilized in spite of apparent shortcomings.

Examinations (and critiques) of the review process have been published in many

fields. Peters and Ceci (1982), in their famous (perhaps infamous) paper, describe how

they sent twelve papers previously published in highly regarded psychology journals

back to these journals as new submissions with fictitious author names. Only three of the

twelve articles were detected as resubmitted papers; out of the nine remaining, eight were

rejected (mostly for having serious methodological flaws). While we hope that such an

expos could not happen in the academic accounting profession, the perception remains

that the process of publishing a paper in a top accounting journal is not always fair

(Moizer, 2009). This is confirmed by the survey results reported by Bailey et al. (2008).

Their results suggest that only 59.8% of respondents believe that the review process is

fair for top-tier journals, and that assistant professors (the ones under the most pressure to

publish) believe the review process is more unfair than associate and full professors.

Several papers have been published in management (e.g., Bedeian, 2003; Harrison,

2002), marketing (e.g., Bailey et al., 2012; Holbrook, 1986), finance (e.g., Bailey,

5
Hermanson, and Tomkins, 2008), and psychology (e.g., Epstein, 1995; Peters and Ceci,

1982) describing flaws in the academic review process and how it can be improved. We

attempt to integrate the relevant points and suggestions from other fields into our

recommendations here. Prior work has also been done in accounting (e.g., Moizer, 2009;

Bailey, Hermanson, and Louwers, 2008; Omer et al., 2004; Kachelmeier, 2004; Demski

and Zimmerman, 2000; Brinn et al., 1996), and we attempt to integrate (and

acknowledge) contributions from these authors here.

3. Why Are Reviews Important?

Reviews are critical to our academic profession because they help screen journal

content, and thereby vet what is added to our body of knowledge. The reader might

imagine what would happen if reviews were not done perhaps if all academic

accounting papers were merely accepted for publication. This would be similar to

handing out gold medals to all athletes instead of only the very best. Readers would not

be able to sort through the deluge of marginal, contradictory, and inaccurate papers, and

the profession would not survive the precipitous loss of credibility. Future work would

lack a firm foundation to build on. In short, the market for research would be

compromised and the entire academic accounting discipline would suffer (Kachelmeier,

2004).

3.1 Consequences of Editorial Mistakes

Editors, who are the gatekeepers of what is accepted in academic journals, rely on

reviewers to help them determine whether to accept, reject, or consider the manuscript for

revision. The effectiveness of this feedback has implications for reputation of the

6
journal, but it also influences the continuity of fruitful streams of research within the

discipline. The acceptance of a manuscript potentially impacts both the academic and

practitioner communities. In addition, it provides a basis for future research and its

contribution will be manifested in subsequent citations.

The importance of reviewer input in the publication decision is best illustrated by

consequences of editorial mistakes. Reviewers sometimes recommend acceptance of

poor quality manuscripts.5 For example, a reviewer might feel unqualified and, at the

same time, reluctant to admit to his or her lack of expertise. As a result, the reviewer

might provide an inadequate review and recommend acceptance based on the inability to

demonstrate justifiable reasons for rejection.

Another example might be a reviewer who is unwilling to invest the time needed for a

comprehensive review. As a result, the reviewer might provide cursory feedback and a

favorable recommendation to discourage the need for additional review effort. In this

case, the editor might interpret the lack of effort by the reviewer as an acknowledgement

that the manuscript lacks flaws.

A reviewer might also recommend acceptance of a poor manuscript if he or she

suspects the manuscript is written by a reputable author (Bailey et al, 2008). This lack of

objectivity may make the reviewer reluctant to point out weaknesses in the manuscript.

These behaviors potentially result in the acceptance of a poor quality manuscript (a Type

II error). This can result in the profession using weak (or simply false) research findings

as a basis for future research and may lead subsequent researchers down unproductive

research paths. Eventually, the reputation of the journal is damaged (see Figure 1).

5
Reviewers are also referred to as referees. For consistency, we will use the term reviewer here.

7
 ______________________

Insert Figure 1
______________________

Similarly, reviewers might recommend the rejection of a high quality manuscript (a

Type I error). This editorial inefficiency is sometimes related to reluctance of the

reviewer to accept innovative methodologies or unexpected results. In other cases, the

reviewer may be unwilling or unable to perform a quality review and merely lists reasons

to reject the paper (and, as Peters and Ceci, 1982, show, there are always reasons to reject

any paper).

When this occurs, the readership is deprived of useful knowledge that can increase

the body of knowledge and provide guidance to practice. The rejection of a suitable

paper discourages future research and ultimately hurts the reputation of the journal. In all

likelihood, the manuscript will ultimately be published in a competing journal and the

impact of the research findings is delayed. In the end, editor, reviewer, and author effort

and time is wasted.

The editor also depends on reviewers to provide feedback to authors that will improve

their work. This reviewer role is important whether or not the manuscript continues in

the review process. If a revision is requested, the editor typically utilizes reviewer

suggestions to formulate expectations of the revision. If the manuscript is rejected, the

expectation is that the authors will use the feedback before submitting to another journal.

In this case, useful comments will provide new direction for collecting additional data,

redesigning an experiment, conducting different analyses, or improving a write-up.

8
3.2 The Difficulty of Obtaining Quality Reviews

The benefits of a quality review are not without cost. Referees are usually not

compensated for their work (paying reviewers more than a token amount would quickly

bankrupt academic journals).6 The fact is that qualified reviewers are a scarce resource

(Gilmore et al., 2006) and top researchers are often inundated with more requests to

review than they could complete without neglecting their own research, teaching, and

service obligations.

A quality reviewer is valuable. From an editors perspective, reviewers develop a

reputation as helpful or not helpful, brash or courteous, and timely or tardy.7 As might be

expected, individuals who perform high quality and timely reviews are likely to receive

subsequent requests. In essence, talented reviewers can be punished by more requests

for review work.

Likewise, there is the potential for bad behavior to be rewarded in the review process.

By suggesting a reject recommendation the reviewer likely frees himself from an

additional review of a resubmitted manuscript. Thus, rejecting a manuscript can reduce

the reviewers workload.

The anonymity of reviewers may also contribute to lower quality reviews. By

remaining unknown to the authors, reviewers may believe that they are free from the

6
Some journals compensate reviewers if they complete a review within a certain period of time.
Compensation can take the form of direct payments, waived future submission fees, etc. Compensating
reviewers can increase turnaround time, but journals usually charge higher submission fees to offset
compensation costs.
7
The software utilized by the American Accounting Association journals offers the editor an opportunity to
rate the reviewer effectiveness and archives the responses.

9
consequences of a poor review.8 In combination, these factors provide some reviewers

with the incentive to shirk to provide sloppy and/or inappropriate reviews.

Unfortunately, poor reviews impose costs on others. Editors that receive poor

reviews must decide whether to do the review themselves or to proceed using low quality

feedback. The first alternative is usually not possible because the editors time is limited.

The second alternative increases the chances that an unsuitable paper will be published

or, more likely, that an important paper will be rejected.

4. How the Review Process Works

The editor manages the flow of manuscripts, which includes tasks such as

corresponding with authors, assigning reviewers, advising authors regarding revision, and

working with publishing organizations to develop electronic and hard copy for

distribution. When a manuscript is received, the editor or editorial staff member usually

reviews the manuscript for compliance with journal formatting standards (sometimes

referred to as quality control). For larger journals, the editor assigns the manuscript to

an associate editor who has an expertise in the topical area.9 The associate editor is

usually expected to skim the manuscript and recommend potential reviewers. The

associate editor might recommend a desk rejection, (i.e., rejection without formal

review.) Desk rejections usually occur if the content is similar to previously published

8
The negative effects of anonymity can also work the other way, when authors submit papers that are not
proofread, not previously presented, and/or contain multiple micro-font tables, and expect an anonymous
reviewer to serve as their mentor and writing coach. It should also be noted that while the reviewer is
anonymous to the authors, he is not anonymous to the editor, who can respond by reducing future review
opportunities.
9
Titles differ depending on the journal. For example, for the Accounting Review, the senior editor
assigns the manuscript to an editor.

10
manuscripts, the analysis exhibits obvious flaws, or the subject is outside of the scope of

the journals mission.10

Referees are typically selected from a pool of authors who have published in the in

the journal (or comparable journals) on similar topics. These individuals are usually

selected in one of two ways. First, the editor may utilize members of the editorial board.

These individuals are familiar with the journal and have made a commitment to

reviewing for the journal.11 Alternatively, the editor might search for individuals not on

the editorial board, referred to as ad hoc reviewers. These individuals are typically

cited in the submitted manuscript or have demonstrated an expertise in the subject of the

manuscript.

Typically two reviewers are assigned to a manuscript. Three reviewers are

sometimes assigned when one reviewer is less experienced. In all, a submitted

manuscript is usually read by three to five individuals, two or three reviewers who

scrutinize the content, analysis, and contribution of the manuscript, an associate editor,

and/or an editor who synthesizes the conclusions of the reviewers and provides the

authors with a consensus of the recommendations and makes suggestions for the revision.

To facilitate an unbiased review the identity of the reviewer is not provided to the

authors, and the authors are also unknown to the reviewer.12 Because neither party is

aware of the identity of the other, this technique is known as a double-blind review.

10
For journals of the American Accounting Association submission fees are refunded for desk rejections
because the manuscript did not undergo formal review. While the outcome of a desk rejection, on surface,
seems harsh, it benefits the editorial process by expediting a highly probable negative outcome. The
authors are benefitted by a refunded submission fee and avoiding the wait associated with the review
process.
11
The American Accounting Association journals utilize software that facilitates a search for qualified
reviewers using key words.
12
Some have suggested reviewers not be anonymous (e.g., Epstein, 1995) while other suggest it is difficult
to achieve a truly anonymous review (Bailey, et al. 2012).

11
For journals that utilize associate editors, journal policy varies on whether the associate

editor is revealed to the authors.

Spiegal (2012) believes reviewer delays and extensive revision requests are a major

reason for delaying the review process. The reviewers are usually asked to complete the

review within four to six weeks. It is significant to note that the time taken to respond to

authors is primarily controlled by individual reviewers and that reviewers vary

significantly in the time taken to respond to journal duties. Table 1 shows median days

required for each stage of the review process for Issues in Accounting Education for a

two and one-half year period. The median turnaround time for the journal is 66 days.13

The median time for a reviewer to respond to review request is five days and the median

time to complete a review, once the reviewer has accepted, is 47 days. That means

78.8% of the total median review time relates to activities under the control of the

individual reviewer.

______________________

Insert Table 1
______________________

Wide deviations in the reviewer procurement and reviewer evaluation stages indicate

reviewers vary significantly in the time taken to complete a review. In other words,

while some reviewers are quick to respond and complete a review, others tend to

significantly delay the review process. A conscientious editor will make efforts to send

reminders of upcoming deadlines, follow up when deadlines are not met, and avoid using

reviewers who are chronically tardy.

13
First response times in other areas are typically longer. For example, economics journals average 6.5
months as an average response time (Ellison 2002).

12
 When all reviews are received, the editor and/or the associate editor summarizes the

comments and make a recommendation for acceptance, revision, or rejection. If the

recommendations of the reviewers are mixed it is the responsibility of the editor to make

a final recommendation. If a revision is requested, the most important role of the editor is

to clarify the major requests of the reviewers. In some cases, the editor may ask a

reviewer to reconsider a request that does not seem logical or that is contrary to another

suggestion.

In some cases, a reviewer may make a request that requires significant effort, for

example changing the purpose of the research or collecting significant additional data.

Accordingly, the editor must decide whether to relax the reviewer request or reject the

paper because the required effort justifies a new submission, rather than a revision.

Most journals have the policy of sending revisions back to the same reviewers. As

such, when a revision is requested, the reviewer should expect to expend additional effort

in the review process. The median number of days to complete a revision for Issues in

Accounting Education is 108 days. This is plenty of time for the reviewer to lose

familiarity with the original content and suggested revisions. Consequently, it is good

practice for reviewers to reread the original manuscript and the previous review before

considering the revision.

5. How to Review a Paper

To reiterate, a review has two main objectives (1) to assist the editor in making a

decision on whether or not to publish the paper, and (2) to assist the authors in improving

their work. This section examines issues to consider before accepting a review

13
assignment and guidelines for conducting a thorough review. This section also relates

pitfalls that tempt reviewers.

5.1 Before Accepting a Review Assignment

The invitation to review a paper suggests that the manuscript has already passed the

initial screen(s) from the editor (and possibly an associate editor). However, it is unlikely

the manuscript has been read in detail before it reaches the reviewer. Before accepting an

invitation to review, a researcher should consider whether he or she has the skills, time,

objectivity, and inclination to review the paper.

Skills Invited reviewers should first evaluate their skill in reviewing a paper on two

bases: subject matter and, if applicable, methodology. The invited researcher should

read the abstract and skim the tables (and the rest of the paper, if necessary) and assess

whether he or she is familiar enough with the literature and the relevant statistical tools to

adequately evaluate the paper. If the potential reviewer does not have the skills to do the

review, the editor should be made aware. The editor may assign the paper to a different

reviewer, or may ask an additional reviewer to join the process.

Time Editors want efficient processing of manuscripts because turnaround time has

implications for the journal reputation. Quick turnaround is also important because

manuscripts utilize data that may become stale or the content may address issues that

lose importance as time passes (such as pending financial accounting pronouncements or

tax legislation).

As noted earlier, three-quarters of the review time is typically related to the reviewer.

As such, a reviewer should try to respond as quickly as possible to a reviewing invitation,

and make a realistic evaluation of his or her ability to complete the review in a timely

14
manner. Unfortunately, reviewers have a tendency to procrastinate. Issues in Accounting

Education asks reviewers to complete a review within five weeks. Historically,

approximately 56% of reviewers complete the review on time. Of these, approximately

half will complete the review in the last week before the review is due (see Table 2).

______________________

Insert Table 2
______________________

Reviewers should also be sensitive that a manuscript may be written by a person who

is preparing a dossier for promotion or tenure. While a solid research record is ultimately

the responsibility of the author, it is unfair to put candidates for promotion at a

disadvantage because of inefficient manuscript processing.

Effective editors are first concerned with content, but are also required to be good

process managers. Most editors send notices reminding reviewers of upcoming

deadlines. In addition, they usually send stern correspondence when reviews are

overdue. Editors and authors appreciate reviewers who respect deadlines.

Objectivity For most journals, the review process is perceived as independent only

under double-blind conditions. In other words, the author names or institutions should

have no influence on how the value of the manuscript is assessed. Potential reviewers

should ask themselves whether they have seen the paper presented or if they know the

authors. Many journals ask reviewers to refrain from searching for papers online (such as

SSRN) to determine author identity. A potential problem with of objectivity should be

discussed with the editor.

15
 Inclinaton In other situations, the researcher simply may not have the inclination to

review the paperfor example, the topic simply might not be interesting to the

researcher, or the researcher might have recently received multiple requests for many

reviews and is burned out. In these cases, it is also appropriate to refuse the offer. Some

researchers may feel that refusing the offer will hurt their reputations, but our

conversations with editors suggests that it is far more damaging when a reviewer accepts

an assignment, postpones a response, and then ultimately admits not being interested in

doing the review (or does a sloppy job).

5.2 The Review Task

After agreeing to do the review, a conscientious reviewer will schedule time to

complete the job well before the deadline. In reviewing the paper, we suggest first

reading the title, abstract, and introduction, and then doing a quick overview of the

tables.14 Then, once the papers motivation, contribution, and method of analysis is clear,

the reviewer should do a detailed read of the entire paper.

In the initial overview, the manuscript should be evaluated for consistency with the

mission of the journal, interest to the readership, general adherence to the journal format,

and writing effectiveness.15 In most cases, the editor or associate editor will have

screened for these qualities. However, significant deficiencies may warrant a request that

the editor reconsider a desk rejection.

14
Recommendations differ here Magro looks at the tables first and attempts to infer the hypothesis that
are being tested before reading the entire paper; Mills appears to begin with the title, abstract, and
introduction (Omer et al., 2004). Given the variance in practices by experts, we recommend that new
researchers consider various recommendations in developing their own preferences.
15
However, in some cases authors should consider whether it is worthwhile to comply with more obtuse
requirements before the paper is accepted. For example, some journals require an accepted manuscript in
11-point font, but this is more difficult for reviewers to read.

16
 In our opinion, a reviewer is not obligated to invest hours on a paper that is poorly

motivated, laden with errors, or rife with grammatical mistakes. However, reviewers

should be honest in their assessment: if the paper was insufficiently motivated, they

should say so, and not go hunting for additional, often contrived, reasons to reject the

paper. Of course, nothing prevents the reviewer from continuing to read through the

paper and providing additional comments and suggestions, especially if it is clear that

authors could use the help or if the motivation could easily be improved and is coupled

with competent analysis.

The approach taken by a reviewer should differ depending on whether the research is

empirical, analytical, or a commentary. However, the following is a general approach a

reviewer might consider. The appendix is a checklist of items that can be useful in

evaluating a manuscript.

Title The title is the first item typically viewed by the reader and it should serve as a

tool for determining whether the manuscript will be of interest to the reader. A good title

should concisely, but adequately, describe the manuscript content.

Abstract The abstract describes the purpose of the manuscript and highlights the key

take-aways from the paper. In addition, the abstract should provide brief information

regarding the methodology and findings. The abstract should also summarize the

conclusions and may also note implications for practitioners and future research.

Motivation and Literature Review The paper should identify a specific and

important problem, and should provide a current and complete overview of the relevant

prior literature. The prior work cited should provide an adequate foundation for the

current papers research questions or hypotheses.

17
 The motivation of the paper must be spelled out clearly and persuasively it is not

sufficient for the authors to simply assume the reviewer will find the paper obviously

interesting. Moreover, motivation must be carefully linked to the prior literature. The

paper should explain the relevant facts from prior work, how those facts relate to the

current paper, and what the current paper does to build on those facts. Merely providing

a laundry list of prior work without any effort to link prior work to the current paper is

inadequate.

A well-crafted motivation will convince the reader to continue reading. When

motivation is lacking a good reviewer will point this out and, where possible, suggest

improvements.

Hypothesis Development While papers (and journals) vary in their use of formally

stated hypotheses, quality research always includes theoretical backing for the analysis.

In more recent work, this is often done simultaneously while describing the contribution

and motivation of the paper. A quality review will consider the strength and

appropriateness of the theoretical development.

Good theoretical development provides the framework within which the papers

results can be understood. Generally, top journals are not receptive to studies that jump

directly to empirical results without explaining their context, because this can often result

in data mining (an ex-post, ad-hoc fitting of a story to the observed results). Tension

between competing explanations is also often needed that is, a paper should try to find

some middle ground between knowing exactly what the results will be prior to the

investigation and having no idea what the results could be and what they could mean.

For example, a paper investigating whether the market misprices accruals in a particular

18
industry would not be interesting because we already know the market misprices accruals

in general. However, if the researcher could point out plausible reasons why the market

might not misprice accruals in a particular setting, that could introduce sufficient tension

to make the paper interesting.

Data and/or Subjects The reviewer should consider whether the data is appropriate

and free from bias. Where bias exists, the reviewer should make an evaluation of the

potential impact of the bias and whether the bias favors or opposes the expected results.

When reviewing an archival paper, the reviewer should consider the data sources and

whether there are known problems. Data sources that are familiar to researchers, such as

CRSP and Compustat, generally require little explanation, but data that is unique or has

known problems may require more description (and possibly more discussion on how the

authors dealt with any problems).

In an experiment, the reviewer should consider the representativeness of the subjects;

for example, whether first-year accounting students are appropriate subjects when testing

for the effect of cognitive biases in analytical review. Also, in an experiment, the

manipulation should be clear, and the experimental results should not boil down to

obvious findings such as the subjects prefer more money over less.

Analysis and Methodology The study should employ statistical tools that are both

appropriate and free from bias. Archival work should use the best tools available to

overcome known problems (especially for studies examining long-window returns). For

example, papers like Gow, Ormazabal, and Taylor (2010) discuss methodologies to

control for cross-sectional and time-series dependence. Lennox, Francis, and Wang

(2012) discuss self-selection models, and Beaver, McNichols, and Price (2007) discuss

19
methods of dealing with delisted firms. Papers that dont use the most up-to-date tools

dont deserve automatic rejection; in our experience, results under simple OLS typically

hold up under more exotic approaches, but the reviewer should make recommendations

to improve the analysis when needed.

Results and Contribution The results of the paper should be connected to

hypotheses. Are the hypotheses supported? Do the results suggest changes to established

theory? The authors should highlight the important aspects of their results (and not

assume that these are obvious). Results should be clear; most journals now require that

tables be able to stand on their own, meaning that a reader should be able to understand

them without having thoroughly read the paper. This means identifying the dependent

variable and providing definitions for all variables used (or referring the reader to another

table or appendix that is close at hand).

The results of the paper (and their discussion) should lead smoothly back into the

main contribution of the paper, and should not lead the reader down meandering side

issues. The contribution of the paper should be sufficient to warrant publication in the

journal. This question is often considered by the editor before the paper is sent to the

reviewer(s), but a conscientious reviewer will also address this big picture question.

Contribution has two elements: (i) does the paper, by itself, make a sufficient

contribution to our understanding? and, (ii) does the paper make a sufficient marginal

contribution relative to other work that is already published, or is found in working

papers already in circulation? The answer to the first question must always be

affirmative to warrant proceeding: if no other paper has investigated the topic, the reason

might be that the topic is simply not interesting or important. If the contribution of the

20
paper is wanting, a good reviewer should recommend where the contribution should be

improved or perhaps journals where the paper might be more appropriate.

Conclusions The conclusion of the study should be supported by the analysis and

results. In some cases, authors may overreach (or, less frequently, undersell their

findings). The authors should discuss important limitations of the study, avoiding

mention of obvious points, such as a generic future results may be different. Finally,

the authors should discuss legitimate directions for future research.

5.3 Preparing a Reviewer Report

A quality review report begins with a succinct overview of the paper and its findings;

i.e., it will identify the papers incremental contribution (Kachalmeier, 2004). The report

should then introduce the most important elements of the paper both positive and

negative. For example, a paper may address a particularly important question and be well

motivated, but might suffer from a fatal bias in the data used. After laying out the most

significant issues of the paper, the report should list minor issues in order from most

significant to least significant.

Whenever possible, reviewers should provide suggestions how to overcome problems

noted. If a revision is requested, the associate editor or editor will likely utilize these

suggestions when responding to the author. If the manuscript is ultimately rejected, the

reviewers suggestions should reflect how the study might by reconducted to achieve

more publishable results. If practical, the reviewer may want to offer suggestions for

publication in alternative journals. However, as Demski and Zimmerman (2000) point

out, the reviewer is not an anonymous co-author; the authors are not obligated to take the

reviewers advice (although they should explain why they dont in their response), and

21
reviewers are not obligated to solve the papers problems (although some suggestions are

often appreciated). Further, the reviewer should distinguish between correctable and

uncorrectable flaws (Moizer, 2009).

Most journals ask reviewers to express their recommendation for a manuscript. The

software utilized by the journals of the American Accounting Association allows the

reviewer to choose between five recommendations (although these may be altered by

individual editors):

1. accept for publication with no changes

2. accept for publication subject to minor improvements
3. revise utilizing review comments before a final decision can be made
4. significantly revise before a final decision can be made
5. reject without further consideration

In general, reviewers should not put their recommendation in their review report, as

this potentially preempts the editor (who has the ultimate responsibility to make these

calls). The reviewer should make a recommendation to the editor outside of the review

report.

In the first round, if the reviewer provides a positive report and the editor concurs, the

paper typically will receive a revise and resubmit recommendation.16 The authors,

following the guidance from the editor, should provide a detailed memo discussing how

they dealt with each review point (although the authors need not follow every suggestion,

they should explain their decision when they do not). Most journals now discourage

multiple-round sessions and prefer to make an up-or-down decision after one or two

rounds. In subsequent rounds, we recommend that reviewers should be careful not to

move the goalposts for authors. That is, reviewers should avoid the impulse to produce

16
Journals rarely accept a paper for publication without requiring at least some changes.

22
new, significant recommendations beyond their first review, but should focus on whether

their original review points were adequately addressed.17

In some cases the authors may fail to address the editor and reviewer comments

(perhaps because the authors might not have fully understood them), and do not make

progress in their revision. When this happens it should be clear to the editor that their

earlier review points were not adequately addressed. In rare cases the reviewer may

realize he made a mistake and that his recommendations appear to have diminished the

value of the paper; in these cases, we think complete honesty is appropriate (rather than a

somewhat dishonest revision of history where the reviewer pretends he didnt make the

recommendation in the first place and is critical of the change in the revision). Reviewers

should also pay attention to the comments from other reviewers and from the editor

(Moizer, 2009).

5.4 Reviewing Pitfalls

It is important to recognize that a review is not an opportunity to get revenge for past

slights from poor reviews (dishing some back out), nor a game in which the reviewer

merely lists all possible reasons to reject the paper (perhaps thinking that such behavior

will impress the editor by showcasing the intelligence of the reviewer).18

There are a number of valid criticisms that can be made of any paper. There are also

some cheap shots that seem to serve as substitutes for a careful reading and thoughtful

analysis of the paper.

17
There are likely many horror stories of multiple-round submissions; we are personally familiar with
one paper going five rounds, where an incompetent reviewer eventually cycled back to old review points
that he had earlier acknowledged as resolved.
18
Bailey et al. (2008) report that behaviors like getting even, and even delaying reviews for selfish
reasons, rank among the most serious perceived threats to the review process.

23
 Research Bias They are many variants of the bias problem, but most represent a

misinterpretation (or more accurately, an overstatement) of the problem. All data,

experimental or archival, is subject to error and bias. The key questions are (1) does the

bias work for or against the results, and (2) is the bias significant enough to warrant

concern? Bias in the results or analysis could be sufficient to warrant disclosure and

discussion, a robustness check, or perhaps even rejection if the papers results are

misleading (hopefully with suggestions as to how the bias could be overcome or at least

mitigated).

One potentially significant form of bias is a correlated omitted variable. However,

merely stating that the possibility exists is unhelpful, because all studies are subject to

correlated omitted variables. There may be cases where the authors missed an important

variable, and if so the reviewer should point this out and consider the effect that the

omission might have. For example, Richardson, Sloan, Soliman, and Tuna (2005)

discuss the issue of a correlated omitted variable problem (see their footnote 3), and

Dechow, Hutton, Kim, and Sloan (2011) provide more insights on the effects of a

correlated omitted variable in detecting earnings management.

Another form of bias is the possibility of self-selection. Self-selection is present

whenever the phenomenon studied includes management choice; for example, accruals,

earnings manipulation, discontinued operations, issuing equity, and pursuing an

acquisition. The presence of self-selection rarely invalidates the results, but it can affect

the interpretation of the results. Consider the following sequence that has been studied

by Xie (2001):

Discretionary Accruals Abnormal Future Returns

24
 Xie (2001) finds that discretionary accruals have predictive power over future returns,

building on Sloans (1996) hypothesis that the market fails to fully consider the

differential persistence of the accruals portion of income versus the cash portion of

income. However, the above relationship reflects only the observable factors involved.

More fully, the relationship is:

Management Choices Discretionary Accruals Abnormal Future Returns

Moreover, management choices are not randomly distributed either: they are

functions of management incentives, which are in turn affected by managements ethical

capacity and cultural factors, and these additional factors are much more difficult to

objectively measure. This information in no way invalidates the relationship between

discretionary accruals and future returns that is well documented in the literature.

In some cases self-selection can affect the interpretation of results. For example, it

may be that whatever causes discretionary accruals can also affect future returns, so one

cannot be certain that discretionary accruals cause future returnsthey are merely

associated. This is a weakness common to all archival research: causality is much harder

to establish than association. This inherent weakness can often lead to interesting

research opportunities for experimentalists, who have tools better suited to determining

causality. Self-selection issues can also possibly open the door to asking (and answering)

additional interesting questions, such as what motivates firms to acquire other

companies in the first place? or why would managers want to inflate earnings in some

settings and deflate them in others?

25
 Sweeping Generalizations Reviewers may be tempted to make broad statements of

criticism without strong evidence to support these claims. Some of the most common

claims in this regard are that a paper is not interesting or lacks contribution.

While these claims may be true, neither stands on its own. For example, the claim of

uninteresting should be accompanied by examples of what the readership is expecting

and how they typically use journal content. Then the reviewer should state why the

current article fails to meet those expectations.

When unsupported, the criticism of a lack of contribution can be particularly

damaging because the criticism is often characterized as a fatal flaw that justifies

rejection. The criticism of lack of contribution should also be supported. For example,

the reviewer might identify previous studies that already address the same issue.

Alternatively, the reviewer might indicate why the results would be of little consequence

to practitioners or future researchers.

A Need for Open-Mindedness A final cheap shot is the improper assertion that broad

interpretations of findings from prior work means that future research cannot make a

contribution. This is more an indication of a reviewer who is simply not interested in

learning new things. For example, a reviewer could observe that a growing body of

research finds that the market is not completely efficient with regard to incorporating

financial statement fundamentals into market prices, and therefore research showing the

market fails to understand the implications of changes in pension assumptions does not

make a marginal contribution. This is similar to claiming that the study of disease is not

useful because we already know that people die. A similar approach is the assertion that

26
I just dont buy it, without any explanation of why the results of the paper might be

wrong or what might be missing.

These pitfalls can be often be avoided by a thoughtful reviewer reflecting on what he

is writing. Sometimes it can be useful to apply the criticism one is contemplating to

already published work. For example, if the reviewer believes the paper he is reviewing

has a critical flaw, would that critical flaw also suggest that papers like Fairfield,

Whisenant, and Yohn (2003) or Feltham and Ohlson (1995) should have not been

published? If so, it might be the criticism and not the paper that is at fault.19

6. Summary and Conclusions

Reviews are an essential part of the academic process, and while sloppy reviews hurt

the authors, the journal, and ultimately the profession, good reviews can encourage the

authors, support the journal, and improve the profession. Accordingly, we provide an

overview of the review process and discuss its inherent weaknesses, and we provide

specific guidance on how to do a good review. Finally, we discuss pitfalls (cheap

shots) that a reviewer should try to avoid.

The observations and recommendations in this paper are based on our opinion,

although this has been shaped by our personal experiences and those of our colleagues.

Ultimately, a good review reflects a conscientious effort by the researcher. In some

ways, the review process is affected by the tragedy of the commons: reviewers may

individually benefit from shirking, while the benefits of a good review accrue to the

19
We have purposely selected these papers because they were rejected by the journals they were initially
submitted to, as was Ball and Brown (1968).

27
profession as a whole. Thus, the quality of the peer-review system depends on the skills

and integrity of reviewers.

We do not wish to imply that responsibility for the review process (and its outcome)

lies principally with the reviewer, or even the editor. Although these individuals

certainly have a strong impact, we believe the authors have the greatest influence over

whether their paper is accepted or rejected. Editors and reviewers must work with what

they are given, and even if they make a mistake in rejecting a paper that should have been

accepted, there are many other competing accounting journals that will be happy to

publish good research. Sloppy work will, in general, receive negative reviews; authors

who would prefer good reviews followed by publication should avail themselves of

Zimmermans (1989) solid advice.

In our experience, reviews from accounting journals are generally of better quality

than reviews from other business academic journals. This is a reflection of the

collegiality of the academic accounting in general. In recent years the profession has

received calls for more diversity in accounting research, and calls for improvements to

the review process (Waymire, 2012). Recent research suggests that concentration (the

propensity of publications in top journals to be dominated by a few elite schools) has

decreased in top accounting journals (Oler, Oler, Skousen, and Talakai, 2012), suggesting

that diversity is indeed increasing. We believe that quality reviews will help this positive

trend continue into the future.

28
References

Bailey, Charles D., Joe F. Hair, Dana R. Hermanson, and Victoria L. Crittenden, 2012,
Marketing academics perceptions of the peer review process, Marketing Education
Review 22(3), 265-280.

Bailey, Charles D., Dana R. Hermanson, and Timothy J. Louwers, 2008, An examination
of the peer review process in accounting journals, Journal of Accounting Education
26, 55-72.

Bailey, Charles D., Dana R. Hermanson, and James G. Tomkins, 2008, The peer review
process in finance journals, Journal of Financial Education 34(2), 1-27.

Ball, Ray and Paul Brown, 1968, An empirical evaluation of accounting income numbers,
Journal of Accounting Research Autumn, 159-178.

Beaver, William, Maureen McNichols, and Richard Price, 2007, Delisting returns and
their effect on accounting-based market anomalies, Journal of Accounting and
Economics 43, 341-368.

Bedeian, Arthur G., 2003, The manuscript review process: the proper roles of authors,
referees, and editors, Journal of Management Inquiry 12(4), 331-338.

Dechow, Patricia M., Amy P. Hutton, Jung Hoon Kim, and Richard G. Sloan, 2011,
Detecting earnings management: A new approach, Working Paper, Berkley.

Demski, Joel S. and Jerold L. Zimmerman, 2000, On research vs. teaching: A long-
term perspective, Accounting Horizons September, 343-352.

Ellison, Glen, 2011, Is peer review in decline? Economic Inquiry, 49(3), 635-657.

Ellison, Glen 2002, The slowdown of the economics publishing process, Journal of
Political Economy 100(5), 947-993.

Epstein, Seymour, 1995, What can be done to improve the review process, American
Psychologist October, 883-885.

Fairfield, Patricia, Scott Whisenant, and Teri Yohn, 2003, Accrued earnings and growth:
Implications for future profitability and mispricing, The Accounting Review 78, 353-
371.

Feltham, Gerald A., and James A. Ohlson, 1995, Valuation and clean surplus accounting
for operating and financial activities, Contemporary Accounting Research 11, 689-
731.

29
Gilmore, Audrey, David Carson, and Chad Perry, 2006, Academic publishing: Best
practice for editors, guest editors, and reviewers, European Business Review 18(6),
468-478.

Gow, Ian D., Gaizka Ormazabal, and Daniel J. Taylor, 2010, Correcting for cross-
sectional and time-series dependence in accounting research, The Accounting Review
85, 583-612.

Harrison, David, 2002, Obligations and obfuscations in the review process, Academy of
Management Journal 48(6), 1079-1084.

Holbrook, Morris B., 1986, A note on sadomasochism in the review process: I hate when
that happens, Journal of Marketing July, 104-108.

Kachelmeier, Steven J., 2004, Reviewing the review process, Journal of the American
Taxation Association 26, 143-154.

Lennox, Clive S., Jere R. Francis, and Zitian Wang, 2012, Selection models in accounting
research, The Accounting Review 87, 589-616.

Moizer, Peter, 2009, Publishing in accounting journals: A fair game? Accounting,

Organizations, and Society 34, 285-304.

Omer, Thomas C., Susan L. Porter, Robert J. Yetman, Anne M. Magro, Lillian F. Mills,
Richard C. Sansing, and Benjamin C. Ayers, 2004, A discussion with reviewers:
Insights from the midyear ATA meetings, Journal of the American Taxation
Association 26, 135-141.

Oler, Derek K., Mitchell J. Oler, Christopher J. Skousen, and Jayson Talakai, 2012, Has
concentration in the top six accounting journals changed over time? Working Paper,
Texas Tech University.

Peters, Douglas P., and Stephen J. Ceci, 1982, Peer-review practices of psychological
journals: the fate of published articles, submitted again, Behavioral and Brain
Sciences 5, 187-195.

Pyrczak, Fred, 2005, Evaluating Research in Academic Journals: A Practical Guide to

Realistic Evaluation, 3rd edition, Pyrczak Publishing, Glendale, CA.

Richardson, Scott A., Richard G. Sloan, Mark Soliman, and Irem Tuna, 2005, Accruals
reliability, earnings persistence, and stock prices, Journal of Accounting and
Economics 39, 437-485.

Sloan, Richard G., 1996, Do stock prices fully impound information in accruals and cash
flows about future earnings? The Accounting Review 71, 289-315.

30
Spiegel, Matthew, 2012. Reviewing less progressing more, The Review of Financial
Studies 25(5) 1331-1338.

Waymire, Gregory, 2012, Seeds of Innovation: Can We Alter the Status Quo?
Accounting Education News 40:1, 3-4.

Xie, Hong, 2001, The mispricing of abnormal accruals, The Accounting Review 76, 357-
373.

Zimmerman, Jerold L., 1989, Improving a manuscripts readability and likelihood of

publication, Issues in Accounting Education 4, 458-466.

31
_______________________________________________________________________

TABLE 1
Median Duration per Stage of a First Round Review

Stage Median Days Percent Std Dev

Quality Control 1 1.5% 3.2
Reviewer Procurement 5 7.6% 7.2
Reviewer Evaluation 47 71.2% 18.9
Associate Editor Evaluation 9 13.6% 15.2
Editor Decision 4 6.1% 23.2
Total Days for First Round Decision 66 100.0% 40.59

Author Preparation of Revision 108

Stage Activities
Quality Control Assistant editor inspects uploaded documents for completeness and
conformity to journal format
Editor makes initial determination of appropriateness for journal (makes
desk rejection if necessary
Editor assigns associate editor and selects potential reviewers
Reviewer Procurement Requests to review are sent
Reviewers accept or decline request to review
Reviewer Evaluation Reviewers evaluate manuscript
Reviwer report and recommendation is prepared
Associate Editor Associate editor evaluates manuscript and reviews comments of
Evaluation reviewers
Associated editor determines whether there is consensus among readers
and makes a recommendation to the editor
Editor Decision Editor makes first round decision
Authors are notified of first round decision

Based on 588 manuscripts submitted to Issues in Accounting Education between January 1, 2010 to July 26, 2012.

_______________________________________________________________________

32
_______________________________________________________________________

TABLE 2
Average Number of Days to Complete a First Round Review

Reviewers
Days to Review Number Percent Cumulative Percent
0 to 7 3 1.95% 3 1.95%
8 to 14 6 3.90% 9 5.84%
15 to 21 15 9.74% 24 15.58%
22 to 28 22 14.29% 46 29.87%
29 to 35 40 25.97% 86 55.84%
36 to 42 30 19.48% 116 75.32%
43 to 49 18 11.69% 134 87.01%
50 to 56 7 4.55% 141 91.56%
57 to 63 6 3.90% 147 95.45%
64 to 72 3 1.95% 150 97.40%
>72 4 2.60% 154 100.00%
Total 154 100.00%

45

40

35

30

25

20

15

10

0
0to7 8to14 15to21 22to28 29to35 36to42 43to49 50to56 57to63 64to72 >72
AverageDaystoCompleteaReview

Based on the performance of 154 reviewers for Issues in Accounting Education from January 1, 2010 to July 26, 2012.
__________________________________________________________________

33
_______________________________________________________________________

FIGURE 1
Hazards of a Poor Review Process

ReviewDecision
Manuscript
Accept Reject
Quality
CorrectDecision TypeIError
Good Readership becomes aware of Readership is deprived of
useful knowledge useful knowledge
Professional practice benefits Professionals are unaware of
from utilizing results potential means of improving
Researchers are provided a practice
foundation for future work Future research utilizes
Reputation of the journal is research findings
enhanced

TypeIIError
Body of knowledge
incorporates faulty findings
Professional practice is
misguided by utilizing faulting
Poor findings
Researchers potentially pursue
an unfruitful research path
Reputation of the journal
suffers
CorrectDecision
The readership is spared from
uninteresting or misleading
research.
Authors gain by utilizing
review results in future research

_______________________________________________________________________

34
_______________________________________________________________________

APPENDIX
Considerations in Reviewing a Manuscript

Overall
Is the manuscript content consistent with the mission of the journal?
Will the subject matter be of interest to the journal audience?
Does the manuscript adhere to the prescribed journal format?
Is the quality of the writing effective?

Title
Does the title adequately describe the subject matter?
If applicable, does the title identify the participants and findings?
Is the title specific and concise?

Abstract
Is the purpose manuscript adequately identified?
If applicable, are the participants, methodology, and findings identified?
Is the usefulness of the results to practitioners and future researchers identified?

Motivation / Literature Review

Is a specific problem identified and supported as important?
Have the conclusions of past research been accurately represented?
Is the literature review adequately extensive and current?
Are the research questions or hypotheses developed from the theory represented in
the literature review?
Is the identified literature pertinent (rather than a laundry list of prior with without
proper links to the proposed study)?
Is the motivation for the study clear by the end of the literature review?

Hypothesis Development
Are the hypotheses or research questions based on theory from previous work?
Does the study attempt to anticipate findings based on the findings of previous
research?

35
Sample / Subjects / Data
Are the data from an appropriate source?
Is the data source reliable?
Are the data or participants free from bias? If bias exists, does it potentially affect
the results?
Is the sample randomized and stratified?
Are non-responses or eliminations adequately justified?
Are the sample demographics adequately described?
Is the sample size adequate?
If human participants were utilized, was adequate consent obtained?

Instrument
Has the full instrument been provided for observation?
Is the question format appropriate?
Do the questions demonstrate adequate content validity?
Does the instrument contain sensitive inquires that potentially bias results?

Experiment
Are participants selected in a random manner?
Are treatments alternated?
Are administrators properly trained and unbiased?

Methodology and Analysis

Are proper statistical methods utilized?
Has statistical significance been properly established?
Are the tables describing the results comprehensive and descriptive?
Do the statistical results properly relate to the research questions or hypotheses?

Results and Contribution

Are the results related to the purpose of the study?
Are the results sufficiently generalizable to the external environment?
Are the results described in perspective to previous studies?

Conclusions
Are the conclusions supported by the analysis?
Are the limitations of the study discussed?
Are implications for practitioners and future research adequately addressed?

_______________________________________________________________________
Note: These considerations are not intended as all inclusive.

36