465483

13
TAP23210.1177/0959354312465483Theory & PsychologyHager

Article

Theory & Psychology

The statistical theories of 23(2) 251­–270
© The Author(s) 2013

Fisher and of Neyman and Reprints and permissions:

sagepub.co.uk/journalsPermissions.nav

Pearson: A methodological DOI: 10.1177/0959354312465483

tap.sagepub.com

perspective

Willi Hager
Georg-Elias-Mueller-Institute for Psychology

Abstract
Most of the debates around statistical testing suffer from a failure to identify clearly the features
specific to the theories invented by Fisher and by Neyman and Pearson. These features are
outlined. The hybrids of Fisher’s and Neyman–Pearson’s theory are briefly addressed. The lack
of random sampling and its consequences for statistical inference are also highlighted, leading
to the recommendation to dispense with inferences and perform approximate randomization
tests instead. A possible scheme for the appraisal of substantive hypotheses is offered, the
corroboration of which is a necessary prerequisite for scientific explanations and predictions.
The scheme is partly based on the Neyman–Pearson theory. This theory, though not perfect,
is superior to its competitors, especially when examining substantive hypotheses. The many
statistical and extra-statistical decisions prior to experimentation and the inevitable subjectivity
of our research endeavors are emphasized. If feasible, statistical problems should be discussed
from an extra-statistical methodological/epistemological viewpoint.

Keywords
approximate randomization tests, lack of random samples and its consequences, statistical
testing theories, statistical tests, substantive hypotheses and the NPT

This article will deal mainly with three questions: What did Fisher, on the one hand, and
Neyman and Pearson, on the other, express in their theories? Can the lack of random sam-
ples be incorporated into the theories? Of what help can statistical tests be in examining
substantive (non-statistical) hypotheses?

Corresponding author:
Willi Hager, Institute for Psychology, University of Göttingen, Gosslerstr. 14, D-37073 Göttingen, Germany.
Email: whager@uni-goettingen.de

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

252 Theory & Psychology 23(2)

Statistical testing
Fisher (1956) formulated as the goal of any statistical test: “The statistical tests … are …
in use to distinguish real effects of importance to a research programme from such …
effects as … appeared in consequence of errors of random sampling, or of uncontrolled
variability, of any sort” (p. 75). Statistical tests refer to statistical hypotheses, which are
assertions about (theoretical) values of a random variable and/or about its distribution
(cf. Hays, 1994, p. 269). They inevitably test their particular null hypothesis (H0), stating
that the variation in the data is solely due to chance, and this may mean more specifi-
cally: H0: µ 1 – µ2 = 0 with µ1 and µ2 being two (theoretical) means. Any statistical test is
about p(Data/H0): that is, the probability of the data under the H0 assumed valid.
Statistical tests do not refer to different modes of interpreting their results. To fill
this lacuna, R. A. Fisher developed his Theory of Significance Testing (FST) in the
early 1920s, and in the late 1920s Neyman and Pearson began to develop their
Theory of Testing Statistical Hypotheses (NPT)—two very different theories, as it
turned out.

The two statistical testing theories

To understand the differences between the two testing theories, the original sources
should be consulted. Amongst others, Halpin and Stam (2006), Hubbard (2004), and
Huberty (1987, 1993) give many original quotations, compiled and supplemented in
Table 1. Only some points will be addressed subsequently.
Fisher (1956) claimed: “Inductive inference is the only process known to us by which
essentially new knowledge comes into the world” (p. 6). And: “[Inductive] inferences we
recognize to be uncertain inferences, but it does not follow from this that they are not
mathematically rigorous inferences” (Fisher, 1935, p. 39). (The necessary populations
“have no objective reality” and are mere inventions of the statistician; Fisher, 1956, p.
77.) In contrast, Neyman (1942) stated that “there is a considerable amount of reasoning
involved [in the theory of testing hypotheses]. As usual, however, the reasoning is deduc-
tive” (p. 301). Neyman and Pearson (1933a) thought that “no test based on the theory of
probability can by itself provide any valuable evidence of the truth or falsehood of … [a
statistical] hypothesis” (pp. 290–291). For Neyman and Pearson (1933a), a Neyman–
Pearson test is a “rule of (inductive) behaviour,” as opposed to Fisher’s “inductive infer-
ences”: “If a rule … unambiguously prescribes the selection of action for each possible
[experimental] outcome …, then it is a rule of inductive behavior” (Neyman, 1950, p. 10;
1957, 1976). Following Popper (1934/1992), this is a “methodological rule” (pp. 53–56)
which may serve as a convention. Whereas Neyman and Pearson emphasized the deci-
sions necessary with their theory (Neyman, 1942, p. 303), Fisher (1956) rejected deci-
sions as unscientific, because “[they] are final” (pp. 98–103), while the results of
significance tests are “provisional, and capable, not only of confirmation, but of revi-
sion” (p. 99). Pearson (1955) replied that “the tests themselves give no final verdict, but
as tools help the worker … to form his final or provisional decision” (p. 206). Fisher
(1966, pp. 44–45) emphasized that randomization is a necessary pre-requisite for a valid
interpretation of the outcome of a statistical test; Neyman acknowledged the importance

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 253

of this concept in 1950 (pp. 292–294; 1967; see also Cook & Campbell, 1979; Wilkinson
& The Task Force on Statistical Inference, 1999, p. 595, for strategies with non-random
assignment).
In the NPT two kinds of error can occur with certain probabilities (Neyman, 1950, pp.
261–264): α = p(AH1/H0 valid; Type I error) and β = p(AH0/H1 valid; Type II error; A:
acceptance), the complement of β is called power (1 – β), the probability of correctly
accepting the H1. Although ideally both hypotheses should be considered as equally
important, prefixing unequal error probabilities means to consider both hypotheses as
unequally important (Neyman, 1942, p. 304). Together with the sample size and the non-
centrality parameters (NCPs), assessing the magnitude of a relationship, there are four
related determinants, which can be subjected to power analysis to control α and β by
determining the sample size. For its execution an exact H1,crit is necessary, which is
selected by choosing NCPcrit. Power analysis then rests on special graphs or tables; the
concept of power is NPT-specific. Despite the caveat of Wilkinson and the Task Force on
Statistical Inference (1999, p. 596), other forms of power analysis are possible—it makes
no sense to use the sample size as a “dependent variable” when it is fixed. Fisher’s (1966)
counterpart to power is “precision”: “The value of the experiment is increased whenever
it permits the [H0] to be more readily disproved” (p. 23). “Precision” can be associated
with the NPT without causing inconsistencies—like all methods of Experimental Design,
which are not FST-theoretical, but specific to Fisher’s Theory of Experimental Design.
For Fisher (1966), “the [H0] is never proved or established, but is probably disproved”
(p. 16), and it is not “accepted” (p. 17). Wilkinson et al. (1999, p. 599) and many others
follow this verdict. Neyman (1942) states: “If [the test statistic falls into the region of
rejection], the … H0 will be rejected, if it does not, it will be accepted” (p. 303)—an “act
of will or a decision” (Neyman, 1957, p. 10; cf. Frick, 1995), but this decision must rest
on a comparatively small β value: If H1 is only accepted if α is small, then accepting H0
is only justified with small a priori β; often α < β ≤ .20 (.30); Neyman and Pearson did
not mention this important point.
Some authors take issue with the frequentist definitions of the two error probabilities
α and β, being “ill-defined long-run error rates,” according to Oakes (1986, p. 145). The
NPT is based on an axiomatic theory of probability (Neyman, 1952, pp. 24–27) in
which “probability” is defined as an “abstract concept,” and the usual empirical coun-
terparts are relative frequencies of events (Neyman, 1952, pp. 26–27; 1977). The short-
coming of the “ill-defined error rates” can be overcome by invoking an auxiliary
hypothesis1 without empirical content, stating that the error probabilities are nearly
equal to the ones fixed prior to experimentation—an assumption researchers usually,
but tacitly, accept (but see Hacking, 1965).
A Neyman–Pearson test leads to a categorical decision concerning the acceptance
of the H0 or the opposed H1: “The outcome of the test reduces to either accepting or
rejecting the hypothesis tested” (Neyman, 1950, p. 260). However, this decision only
tells us whether a relationship exists in the data or (approximately) not. If it does, its
magnitude has to be assessed, “because science is inevitably about magnitudes”
(Cohen, 1990, p. 1309). This magnitude is expressed by effect sizes (ESs), which are
functionally related to the NCPs (Winer, Brown, & Michels, 1991, p. 127), so that they
can be associated with the NPT without inconsistencies; they are—as opposed to the

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

254 Theory & Psychology 23(2)

Table 1.  An overview of the main features of the statistical theories of R.A. Fisher (FST) and
of J. Neyman and E.S. Pearson (NPT).

Fisher’s Theory of Significance Testing Neyman and Pearson’s Theory of Testing

(FST) Statistical Hypotheses (NPT)
Statistical tests serve to distinguish systematic from unsystematic influences (Fisher);
any statistical test gives the probability p(Data/H0).
Main characteristics of the theories and underlying philosophies
Theory of inductive (statistical) inferences. Theory of decisions on statistical
hypotheses. No inferences intended.
Significance tests, some of which have been Statistical hypotheses tests, none of which
invented by Fisher (e.g., ANOVA). were created by Neyman and Pearson.
Inductive—from the particular to the Deductive—from the general to the
general, but with deductive aspects. particular.
Disproving H0 enables inductive inferences. Hypothesis tests as a “rule of inductive
behavior.”
FST as an objective method of inductive NPT gives guidelines for making optimal
inference, possibly subject to revision. decisions about statistical hypotheses.
(Fisher rejected the notion of decisions in the The decisions are either final or provisional
sciences, because they are “inevitably final.”) and the basis for subsequent “inductive
behavior.”
FST gives guidelines for interpreting the Hypothesis tests for keeping the long-run
strength of evidence against H0. frequencies of errors low.
Significance tests as means of learning from Hypothesis tests as a means of learning
the data (Fisher). from the data (Pearson).
Rejection of Bayesian thinking.
Logical syllogisms not part of the two theories.
Evidential/epistemic testing paradigm; Behavioral testing paradigm; non-positivist
positivist view. view.
Agronomy; later different contents, Industrial quality control; later different
showing the wide application of the FST. contents, showing the wide applicability of
the NPT.
Every experiment gives the data a “chance Every experiment as a basis for the decision
of disproving the null hypothesis” (Fisher, about H0 and H1.
1966, p. 16).
Data as “objective observational facts,” but (Experimental data as potentially faulty.)
distorted by errors in measurement and
random sampling.
(–) Statistical hypotheses as possibly false; the truth
or falsehood cannot be detected for sure; one
subsequently acts, as if they were true.
Fisher (1950) referred to “scientific” Recommendations concerning the
hypotheses, but these seem to be connections between the substantive
complex statistical hypotheses rather and the statistical hypotheses (Neyman).
than “substantive” ones. No proposals Alternative hypotheses can be derived as
according the connection between the conforming to the substantive hypothesis.
substantive and the statistical hypotheses.
No possibility of deriving an H1 as
conforming to a substantive hypothesis.

(Continued)

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 255
Table 1. (Continued)
Fisher’s Theory of Significance Testing
(FST)
Populations: not theory specific
Populations merely imagined by the
statistician as being infinite, assumed
normal. Actual sample regarded as
“random sample” from this population.
Necessary for Fisher’s inductive inferences
and for the analytical derivation of sampling
distributions of parametric statistical tests.
Testing: theory specific
Choose appropriate random variables and specify their distributions.
Apply methods of Experimental Design
for controlling experimental errors and
enhancing precision, if possible.
Experimental design and significance tests
are “two sides of the same thing” (Fisher).
Randomize to reduce errors in the
experimental data and to ensure
interpretability of a significance test.
State H0.
(–)
Specify the test statistic (TS) and the referent sampling distribution under the H0.
State how you cope with the statistical
assumptions.
(A p value can be prefixed, since Fisher
sometimes proposed p ≤ .05 or p ≤ .01 as
conventions.)a The p value is a property of
the data and a random variable (Fisher’s
“significance level”).
(–)
(–)
(–)
(–)
(–)
Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014
Neyman and Pearson’s Theory of Testing
Statistical Hypotheses (NPT)
Normal distribution; no objective reality
(Pearson).
Necessary for the analytical derivation
of sampling distributions of parametric
statistical tests. Basis for repeating the
experiment many times (“long run”).
Apply methods of Experimental Design
for controlling experimental errors and
enhancing precision and power, if possible
(Neyman, 1967; Pearson, 1962).
Experimental design and hypotheses tests
are different themes.
Randomize to reduce errors in the
experimental data (Neyman, 1967) and
ensure interpretability of an NP test.
State the statistical hypothesis which is
derived from the substantive one.
State the complementary statistical
hypothesis whereby you get H0 and H1.
State how you cope with the statistical
assumptions: Neyman: “Postulate the
assumptions to hold” (Neyman, 1942,
p. 311).
Specify a (conventional) value of α with the
option to choose larger values than α = .05;
α is a property of the test.
Specify the non-central distributions under
the H1,s.
Specify a value of β; β ≤ .20 (.30).
Specify a value of the non-centrality
parameter (NCPcrit) (or of the effect size).
Perform a power analysis to control α and
β relative to NCPcrit.
The result is the sample size. (Other forms
of power analysis are possible; Neyman,
1977).
(Continued)
256 Theory & Psychology 23(2)

Table 1. (Continued)

Fisher’s Theory of Significance Testing Neyman and Pearson’s Theory of Testing

(FST) Statistical Hypotheses (NPT)
(–) Determine the region of rejection.
Collect data.
Calculate the value of TS (optional). Calculate the value of TS.
Determine the exact p value for the data (–)
or for TS.
If H0 is rejected (p ≤ .05), the data show a “Rule of inductive behavior:” If TS < TScrit
“statistically significant” result; inferences and if β a priori is sufficiently small, H0 is
are possible. accepted; NCP ≈ 0.
If p > .05, H0 is not rejected; the data “Rule of inductive behavior:” Accept H1 if
failed to produce a “statistically significant” TS ≥ TScrit.
result.
The smaller the p value, the stronger is the (Neyman and Pearson gave no hint
evidence against H0 (and for a “research concerning the empirical value of the NCP,
program,” Fisher, 1956, p. 75). (Fisher did when H1 is accepted.)
not specify, what he meant by “research
program.” He only was interested in the
“strength of evidence” if p ≤ .05.)
(–) Neyman and Pearson did not deal with the
question how to evaluate the substantive
hypothesis relative to accepting the H0 or
the H1.
Note. Chow (1996, p. 21) gives a similar, but partly erroneous table [e.g., NPT is about the Bayesian
probability p(H0/Data)]. One-sided testing is assumed.
aFisher’s opinion regarding his p value is ambiguous (1950, pp. 80, 93; 1966, pp. 13, 25, 187).

NCPs—free from the sample size: d = (M1 – M2)/se (M: sample mean; s: standard
deviation; Cohen, 1988, pp. 21–22). Power analysis based on (theoretical) ESs can be
performed for many tests (e.g., Cohen, 1988; Kraemer & Thiemann, 1987). But in
research practice they are very rare and this leads to poor empirical investigations with
an average power of about .50 (Cohen, 1994, p. 1000).
Table 1 shows many differences between Fisher’s significance testing and the testing
of statistical hypotheses of Neyman and Pearson, based on tables of Halpin and Stam
(2006), of Huberty (1987, 1993), of Hubbard (2004), and on Fisher (1935, 1947, 1950,
1955, 1956, 1966), on Neyman (1942, 1950, 1952, 1957, 1971, 1976), on Neyman and
Pearson (1928, 1933a, 1933b), and on Pearson (1955, 1962), supplemented by some
features partly taken from Chow (1996), Mayo (1996), and Wilkinson et al. (1999).

The hybrids
Table 1 shows how the tests should be prepared and how the results should be
interpreted—in the ideal case. It is well known, however, that in research practice mix-
tures of the distinct theories prevail (Hubbard, 2004), called the “hybrid models” by
Halpin and Stam (2006; see Spielman, 1973). The FST constitutes the essential part of

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 257

the analysis, supplemented by one or more concept/s of the NPT and partly accompanied
by Bayes thinking, giving p(H0/Data) (Gigerenzer & Murray, 1987, pp. 24–25; Pruzek,
1997).2 Scanning several statistical articles, partly of well-known authors, revealed
hybrid terminology in more than 40% of them: for example, “significance test as accept–
reject rule.” Although the authors differ according to the details, especially Gigerenzer
and Murray (1987) and Halpin and Stam (2006), they agree that the hybrids constitute an
“incoherent mishmash” of both statistical theories (Gigerenzer, 1993, p. 314), which are
“incommensurate” (Hubbard, 2004, p. 319) and “incompatible” (Gigerenzer & Murray,
1987, p. 28). (The latter authors speak of the “hybrid theory” [pp. 21, 23], but a system
of conjectures must be non-contradictory and logically consistent to qualify as a “the-
ory.” Any theory has its own basic constructs and ideas, and melting incompatible con-
structs from different theories constitutes one of the greatest sins scientists can commit;
see Neyman, 1976, for an example.) Gigerenzer and Murray (1987, pp. 22–24) see the
origins of this hybridization in the early textbooks, in which “the hybrid was … pre-
sented as the only truth, … as the single solution to inductive inference” (p. 21; Halpin
& Stam, 2006; Huberty, 1993). The “institutionalization of the hybrid/s” was attained, in
part, by a “neglect of … controversial issues,” and by an “anonymous presentation of the
[constituents of the] hybrid/s” (Gigerenzer & Murray, 1987, p. 23; Halpin & Stam, 2006,
p. 640). A further reason for this hybridization may be that both parties (necessarily)
referred to the same tests. Overall, the hybrids should be banned (e.g., Cortina & Dunlap,
1997; Gigerenzer, 1993, p. 332; Huberty & Pike, 1999, p. 17; for further insightful
sources see Cowles, 1989; Halpin & Stam, 2006; Mayo, 1996; Oakes, 1986).

On the problem of the exact H0

This problem occurs under both theories, but is discussed here in NPT terms. The H0 are
necessarily exact in order to derive the sampling distributions analytically. This H0,
though, is never correct, since the density of an exact value is zero (e.g., Nickerson,
2000, pp. 275–276; Rindskopf, 1997, pp. 320–322, give an overview of the literature;
see also Harlow, Mulaik, & Steiger, 1997). But the consequence of the a priori choice of
α is that a set of small non-null hypotheses is inevitably declared compatible with the
exact H0, so that the H0 always is composite, “composite” here meaning a set of H0,r.
Thus, retaining “H0” means accepting the whole set H0,r, including H0 and the small non-null
hypotheses, the correct interpretation being “an approximate null difference/correlation”
(Chow, 1996). Each non-null hypothesis is associated with its own non-null effect, ES0,r;
the maximum non-null effect ES0,max varies with α and n. Consider an example with H0:
µ1 – µ2 ≤ 0; H1: µ1 – µ2 > 0; n = 21, α = .05, and tcrit(.05;40) = tcrit,1,min = 1.68385. The ES used
above was: d = (M1 – M2)/se = (2/n)1/2*t. The maximum non-null effect is: d0,max =
(2/n)1/2*(1.68385 – 0.00001) = 0.51964; t0,max = 1.68384; the minimum ES under the H1,s
is: dcrit,1,min = (2/n)1/2*tcrit,1,min = 0.51965. Values d0,r ≤ 0.51964 lie in the region of accept-
ance and usually are more likely under the H0,r than under the H1,s; they are interpreted as
associated with the H0,r (Rindskopf, 1997, pp. 320–322). Values d1,s ≥ 0.51965 and ts ≥
1.68385 lead to accepting the set H1,s. (See also the informative example of Rosnow &
Rosenthal, 1989, pp. 1277–1278, in which an effect of d = .50 is associated with an H0
retained using a comparatively low sample size—N = 20, t = 1.118 and p > .10—and

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

258 Theory & Psychology 23(2)

with an H1 for a larger sample size, i.e., N = 80, t = 2.236 and p < .05 [t and p values
slightly modified]; see also Cohen’s values of the ESc critical for “statistical signifi-
cance” [Cohen, 1988, pp. 67, 28–39]). A variant of the (computationally laborious)
“good-enough principle” of Serlin and Lapsley (1993) is already part of any statistical
test, and this belt seems unnecessary (Chow, 1996, pp. 56–57). “The … [statistical] test
acts as a (mediocre) filter for separating effects of different magnitude” (Abelson, 1997,
p. 14). There are other points worth discussing, but space is limited. Let’s attack a further
problem.

The lack of random samples and some of its consequences

In statistical testing, another problem needs to be addressed: “All our conclusions … rest
on a process of random sampling, without it our tests of significance would be worth-
less” (Fisher, 1947, pp. 435–436). But Edgington (1995) states: “Few experiments in …
psychology … use randomly selected subjects, and those [experimenters] who do usu-
ally concern populations so specific as to be of little interest” (pp. 7, 335–336). Instead,
“convenience samples” with participants at hand are used (Hunter & May, 2003;
Reichardt & Gollob, 1999), which—though not necessarily explicitly—are interpreted
as random samples from the target populations—an auxiliary hypothesis not testable,
which serves as a convention (Westermann, 2000, p. 351); the target populations are
never infinite (Reichardt & Gollob, 1999, p. 117). But Hays (1994) warns: “Unless the
assumption of random sampling is at least reasonable, the probability results of inferen-
tial methods mean very little, and these methods might as well be omitted” (p. 227).
Frick (1998) reasons: “The assumption of random sampling from a population is unjusti-
fied, unnecessary, inaccurate, and does not serve psychology well” (p. 527).
If random samples are lacking, “it is important to see whether there is any justification
for … hypothesis testing procedures” (Edgington, 1966, p. 485). Fisher (1966, pp. 44–
48) was the first to consider this problem, and later it was shown “that … [randomiza-
tion] tests could be based on random assignment [or permutation] alone, stressing their
freedom from the assumption of random sampling” (Edgington, 1995, p. 18). (See for
the procedure to generate the randomization sampling distributions by permuting the
data at hand, Edgington, 1995—in the same way the sampling distributions for the rank
tests are generated.) Generally, the randomization critical values are close to the ones
obtained from the respective parametric sampling distributions (e.g., Anderson & Ter
Braak, 2003; Edgington; 1995, pp. 39–44; Hunter & May, 2003; Scheffé, 1959, pp. 291–
330; Still & White, 1981), so that the tabled distributions can be used instead of the
randomization ones (Edgington, 1995, pp. 338–340): “The [parametric] test is used as an
approximation to a randomization test,” and it is called an “approximate randomization
test” (Edgington, 1969). (In contrast to “randomization tests,” “permutation tests” rest
on random sampling from unspecified populations; Edgington, 1995, pp. 337–338).
Randomization tests, free from assumptions concerning populations, test a “wider
hypothesis” (Fisher, 1966, pp. 44–49) than, for example, the t test on means from normal
distributions: that is, the H0 that the underlying distributions are equal or the “identity of
treatment effects” (Edgington, 1995, p. 339). Rejecting this H0 means that the distribu-
tions are unequal. To test hypotheses about, say, expected means, the randomization tests

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 259

need to be associated with the assumption that the underlying distributions are at least
approximately equal in form, which seems widely accepted in empirical research. Based on
this assumption, the importance of findings such as those of Boik (1987) is de-emphasized.
(Boik refers to the non-robustness of a randomization test in the case of heterogeneous
variances—a finding that also holds for normal theory tests.) With this new justification,
statistical tests are mere binary decision rules of accept–reject a statistical hypothesis,
apparently conforming to the NPT, but not to the FST (see Reichardt & Gollob, 1999, for
another approach concerning randomization tests).
Without random sampling, “statistical inferences about populations are … irrelevant”
(Edgington, 1995, p. ix; see also Hays, 1994, p. 227), but generalizability is only seldom
the research goal (Frick, 1998; Hunter & May, 2003, p. 180). With randomization tests,
generalizations are impossible and the results are valid only for the one experiment
(Edgington, 1995, p. 339; Westermann, 2000, pp. 431–433). Oakes (1986) stated: “It is
not unreasonable to conclude, as Popper did with induction, that statistical inference is a
myth” (p. 145), but a myth statisticians and also psychological researchers adhere to
invariably. Parameter estimations need populations, which are absent, and without
parameters confidence intervals are no longer needed (cf. Cortina & Dunlap, 1997, p.
170). Additionally, “it is rare for psychologists to need estimates of parameters; we are
more typically interested whether a causal relationship exists between independent and
dependent variables” (Killeen, 2005, p. 345; see also Cortina & Dunlap, 1997, p. 171;
see for alternative approaches, e.g., Davison & Hinkley, 2006; Efron & Tibshirani, 1993;
Thompson, 1993).
Amongst others, Anderson and Ter Braak (2003), Still and White (1981), and
Westermann (2000, pp. 374–375) dealt with the power and reasoned that it is about equal
to that of the corresponding parametric tests. Therefore power analysis for approximate
randomization tests can be performed in the same manner as for parametric tests.

Substantive and statistical hypotheses

In this part, a possible scheme for examining substantive hypotheses (SuH) by statistical
ones will be presented; a subject which is discussed only rarely in the statistical literature
(e.g., Fisher, 1966; Neyman, 1950, 1957, 1976; Pearson, 1962). Neyman (1950) stated
that when planning an experiment, one usually has in mind the appraisal of an SuH,
“which is not a statistical hypothesis H. H is never identical with the [nonstatistical
SuH]; since… the [SuH] does not mention any random variables” (p. 290). What about
psychologists? “We must carefully distinguish substantive theory [and SuH] from statis-
tical hypothesis” (Meehl, 1978, p. 824). “[An] H1 is neither the substantive hypothesis
itself nor a paraphrase of the substantive hypothesis” (Chow, 1996, p. 70; cf. Cortina &
Dunlap, 1997). Also Fisher (1956, pp. 46–50) addresses “scientific hypotheses,” which
“differ from the simple (statistical) hypotheses such as ‘the random distribution of the
stars’, in that they allow of one or more parameters,” so that Fisher’s “scientific hypoth-
eses” are better described as complex statistical hypotheses. Moreover, he gave no hint
of how to connect his “scientific” hypotheses with substantive ones, but he explained in
which way his scientific hypotheses can be subjected to a test of significance.
What is the difference between the two kinds of hypotheses?

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

260 Theory & Psychology 23(2)

Although any competent psychologist knows the difference between an SuH and a
statistical hypothesis (Meehl, 1967, p. 107), in practice this fact is often neglected, as
may be seen in the many cases where an SuH is equated to an H1 (and its artificial nega-
tion to an H0; an SuH can only be negated by another SuH some researchers adhere to;
Neyman, 1950, p. 290; Popper, 1934/1992, p. 87). Statistical hypotheses refer solely to
random variables (see above), whereas substantive hypotheses typically are conjectures
about unobservable hypothetical entities, states, processes, and the relationship/s between
the concepts (Popper, 1934/1992). These differences mean that substantive theory/SuH
appraisal is completely different from statistical testing.
It is not the aim of science to formulate SuH or to generalize (Popper, 1934/1992).
Instead, one of the most important aims of science is to explain the domain-specific phe-
nomena, and to control and to predict behavior (e.g., Popper, 1934/1992, pp. 59–62).
These aims can only be reached by corroborated3 SuH and theories, and corroboration is
as important as discorroboration. Theories are tools for generating SuH, and any theory
leads to more than one SuH (Chow, 1996, p. 49). Some SuH not inspired by a theory are
mere inventions, possibly based on some data (Popper, 1934/1992, p. 32). In this case,
they must be examined with new experiments (Mayo, 1996). Research consists in
attempts to apply the SuH to one experimental situation after the other, the SuH being
evaluated separately for each experiment. This process is mostly deductive, as is the
NPT; also Fisher (1950, p. 8) uses deductive arguments in the present context. Most
importantly, the SuH determines the adequate analysis and not the Experimental Design:
“In research it is better to ask, ‘What is the best way to test my [substantive …] hypoth-
esis’ rather than ‘Which statistical test is appropriate for these data?’” (Gonzalez, 1994,
p. 326). What about the linkage between the hypotheses?
“A statistical hypothesis H [concerning certain …] random variables, is … formulated
so as to be intimately related to the … [SuH]” (Neyman, 1950, p. 290), with the linkage
being “frequently loose.” Chow (1996, pp. 46–51) prefers an “implicative relationship,”
whereas Meehl (1967, p. 104) proposed “to derive” statistical hypotheses “in a rather
loose sense” (“deductive derivability”; Meehl, 1997, p. 398); Popper (1934/1992, pp.
76–77) and others (e.g., Chow, 1996, pp. 71–72) discuss or use the logical “modus tol-
lens” (Cohen, 1994, p. 998; Falk & Greenbaum, 1995, p. 75). Meehl (1997, pp. 398–400)
shows why this syllogism is inappropriate in appraising SuH, as it is with statistical test-
ing (Cohen, 1994, p. 998; Cortina & Dunlap, 1997, p. 170). Therefore, this syllogism is
replaced by decisions, so that the burden of the “proper” decisions is put on the research-
er’s competent shoulders.
The derivation of the substantive prediction must consider many auxiliary hypotheses
(Meehl, 1978, 1997, p. 398), the “ceteris paribus clause” (CPC, i.e., “all other things
being about equal,” Meehl, 1997, p. 398), the applicability of the theory/SuH, the experi-
mental design, the a priori criteria for “corroborating” or “discorroborating” the SuH
(Lakatos, 1978; Popper, 1934/1992), most of the points of Wilkinson et al. (1999), and so
on. In short: The concrete experimental situation with all details must be specified, form-
ing the set CSES (Completely Specified Experimental Situation), filling the subsequent
lacuna: “We left in our mathematical model a gap for the exercise of a more intuitive
process of personal judgment in such matters as … the appropriate … (error probability
α …), the magnitude of worthwhile effects and the balance of utilities” (Pearson, 1962,

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 261

p. 396). Most decisions are extra-statistical, inevitably subjective, and sometimes arbi-
trary—science is a wo/man-made endeavor. Thompson (1993) states: “Like it or not,
empirical science is inescapably a subjective business” (p. 365; cf. Frick, 1999, p. 187), so
that the “fight against subjectivity” (Gigerenzer, 1987, p. 11) can never be won—despite
statistical tests. We must learn to accept that subjective and arbitrary factors play a deci-
sive role in appraisals of SuH, and that our data are impregnated by theories and hypoth-
eses, and they are fallible (Popper, 1934/1992, pp. 59, 107; see also Hacking, 1965, p. 84;
Lakatos, 1978, pp. 29–30; Polkinghorne, 1983, p. 125). And: Different research strategies
and different types of analysis will possibly lead to divergent results.
Direct derivations from the SuH are impossible because of the theoretical variables
involved, and to appraise an abstract SuH, the abstract concepts must be converted into
observable or empirical variables, which can be chosen according to former theory
appraisals. After enriching the SuH with the set CSES, a substantive prediction (SuP)
referring to observable variables solely is derived from the SuH. From the SuP, enriched
by random variables (RVs), a statistical testing theory, and statistical auxiliary hypothe-
ses (SAH), a conforming H1 is derived. In rare cases a conforming H0 can be derived
from an SuP. “The primary product of a research inquiry is one or more measures of
effect size, not p values” (Cohen, 1990, p. 1310). Although some authors claim the con-
trary,4 any theory/SuH claiming a relationship only contains information concerning the
area or interval the ES should manifest in (Frick, 1999, p. 186); the area depends on the
SuP and the CSES, and it is restricted by choosing the EScrit for power analysis. The EScrit
can be based on former applications of the theory/SuH and modified with respect to dif-
ferences in the CSES. Although the call for ESs is ubiquitious, nobody tells us what to do
with them. So, I propose to interpret the ESs as a crude possibility of operationalizing the
“degree of corroboration” (DC; Popper, 1934/1992, p. 281): DC = ES/EScrit. It is crude
because it is unknown whether there is a monotonous relationship between the ES and
the (real) “degree of corroboration” (see, for a more sophisticated alternative, Meehl,
1997). After testing, the concept of “inductive behaviour” is again applied “to assume a
particular attitude towards the … [SuH]” (Neyman, 1957, p. 10). The following exam-
ples are simplified (see Table 2 for a summary of steps).

Example 1
Paivio’s (1971, 1990) Dual Coding Theory tries to explain different amounts of learning
by the extent the two storages or codes are used. High imaginal content, as in “chair” (as
contrasted with “idea”), is postulated to activate the verbal plus the imaginal store, thus
leading to a superior performance of high imagery words because of more features
encoded. This theory is applicable to verbal learning. An SuH-P may be: “If high and low
imaginal content words are learned, then the amount of learning is CPC-P higher for
words with high imaginal content than for words with low imaginal content.” One of the
learning paradigms is chosen and becomes part of the CSES-P. The empirical independ-
ent variable is the scaled imagery of the words in the lists, assessed prior to experimenta-
tion; the empirical dependent variable is the number of correctly reproduced words; it is
at least interval scaled. The Ss are randomized. These aspects become part of the CSES-P.
The SuH-P is enriched by the CSES-P, and one of several possible SuP-P is then derived

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

262 Theory & Psychology 23(2)

Table 2.  Appraisal of a substantive hypothesis and the NPT: Some necessary complements.

Refer to a substantive theory (optional).

State the substantive hypothesis (SuH) to be appraised (obligatory).
Choose empirical variables as counterparts of the theoretical ones (obligatory).
Consider the scale type of the empirical dependent variable (obligatory).
Specify the enrichments (>–) or auxiliary hypotheses, without which no derivations are possible
(obligatory).
Apply methods of Experimental Design, especially randomization (CPC), and—if possible—
enhance power and precision (obligatory).
Completely specify the experimental situation (CSES). The side conditions must be appropriate
for the theory/SuH to be applicable (obligatory).
Derive the substantive prediction (SuP) (obligatory).
Interpret the dependent variable as random variable (and specify its distribution) (obligatory
when statistical testing is intended).
Choose summarizing random variables, such as means and variances (obligatory).
Choose a statistical testing theory; here: NPT (obligatory).
Derive the conforming statistical hypothesis from the SuP (with allowance for the scale type and
the CSES) (obligatory).
Specify the complementary statistical hypothesis (obligatory).
(The following steps are contained in Table 1.Power analysis is based on an ES.)
Subject your statistical hypotheses to an approximate randomization test (optional).
Do not blindly trust your statistical test: Inspect your data; if possible, compare it to those of a
similar experiment for the same SuH (obligatory).
Applications of “rules of inductive behavior”:
1. SuP >–/≈> StP ↔ H1:
If TS < TScrit, accept the disagreeing H0; ES ≈ 0; reject the StP.
Decision: The SuP does not hold and the SuH is discorroborated.
If TS ≥ TScrit, accept the conforming H1 and the StP.
Decide on the SuP: Compute ES and compare it to EScrit. If DC = ES/EScrit ≥ 1.00, the SuP is
considered as “true.” In the case of DC < 1.00, decide whether the difference is tolerable or
not.
If there is no reason to doubt experimental validity, the SuH is corroborated. The larger the
DC, the better corroborated is the SuH and the better are the respective explanations.
2. If SuP >–/≈> StP ↔ H0 holds (ES ≈ 0) (a rare, but possible case; see Example 2), you can
only decide on the SuH as “corroborated” or “discorroborated,” dependent on the statistical
hypothesis being derived. (The computation of the DC makes no sense here.)
Note. This table contains the most important complements for appraisal of a substantive hypothesis using
the NPT.

(∧: logical “and”; ≈>: loose derivation; >–: is enriched by; AH: auxiliary hypotheses):
[SuH-P >– (AH-P ∧ CPC-P ∧ CSES-P)] ≈> SuP-P: “If words with scaled high imagery
(MHI > 5.50; B2) and scaled low imagery (MLI < 3.50; B1) are learned, then more words
with scaled high imagery are CPC-P correctly recalled than words with scaled low
imagery.” The number of correctly reproduced words is interpreted as a RV, and means
and variances are defined as summarizing RVs, so that the expected ES is: δ = [µ2 –
µ1]/σerror > 0. As testing theory the NPT is chosen. Since the SuH-P and the SuP-P are

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 263

directional, a directional statistical hypothesis (about means) must be derived (StP: sta-
tistical prediction; “↔”: is equivalent to; RT: randomization test): [SuP-P >– (RV ∧ NPT
∧ RT ∧ SAH)] ≈> [StP-P: µ2 – µ1 > 0] ↔ H1,1, and: H0,1: µ2 – µ1 ≤ 0. With this step the
single subjects are replaced by two “hypothetical mean subjects.” (If a second factor is
added, several statistical hypotheses are derivable alternatively.) Power analysis rests
upon: α1 = .05; β1 = .05; δ1,crit = +1.00, leading to n = 22 (Cohen, 1988, p. 30), so that
tcrit(.05;42),1 = +1.682. (The independent variable usually leads to large ESs.) One gets: t1 =
+3.50; the conforming H1,1 and the StP-P are accepted; d1 = +1.0553 > δ,crit = +1.00. The
SuP-P holds and—if the experiment is judged valid—also the SuH-P; the “degree of cor-
roboration” is equal to DC-P = 1.06/1.00 = 1.06—as expected. If d1 < δ1,crit, the corrobo-
ration is restricted. A conforming outcome leads to adding the experiment to the
successful applications of the theory; a negative outcome reduces its domain of applica-
bility (Westermann, 2000). With Fisher’s approach (1950, p. 8) the derivation of an H1 is
impossible.

Example 2
The StP is introduced since often statistical hypotheses are derived which cannot exhaus-
tively be tested with one test, for example the StP-Q: “The relationship between the
independent and the means of the dependent variable, an RV, is exclusively positive lin-
ear.” The shortened derivation leads to: (SuH-Q) >–/≈> (SuP-Q) >–/≈> StP-Q ≈> [H1,2:
Δlinear > 0) ∧ (H0,3: ∑ Δ2deviations = 0)] [Δ: theoretical contrast]. Both hypotheses are indis-
pensable, because the presence of a linear trend does not preclude further trends. Power
analysis is a bit laborious here, but manageable.

Example 3
If—on the other hand—a qualitative and strictly monotonous or strictly ordered trend
(i.e., without equalities and without rank inversions) is predicted: StP-MT: µ1 < µ2 < …
< µK, the most simple procedure consists in performing a series of K – 1 t tests (Winer
et al., 1991, p. 146) on the partial hypotheses H1,t; these tests also are applicable for
more complex patterns, for example a bitonic trend with at least one equality and so
on. Δt = µk+1 –  µk > 0 suffices, since there can be only one rank order in the data. The
retention of at least one H0,t: Δt = µk+1 ≤ µk leads to rejecting the StP-MT. Despite the
cumulation of one of the error probabilities, power analysis can easily be done here
(Hager, 2004). This StP-MT very often results for qualitative independent variables:
for example, if the SuH-P is examined with K ≥ 3 experimental treatments, a test for an
ordered alternative should not be used, since they do not test against the StP-MT as H1
(Berenson, 1982). The H1 of these tests is “weaker,” since it is accepted in spite of
partial equalities and/or partial rank inversions, as long as at least one pair difference—
not necessarily between adjacent means—conforms to the prediction. Thus, these tests
are roughly similar to K(K – 1)/2 t tests, but without cumulating error probabilities.
This proposal for examining SuH contains an asymmetry: Whenever an H0 is derived,
the corroborating result is accepting this H0 which implies ES ≈ 0. So, only a dichoto-
mous decision is possible, and a qualified appraisal of the SuH cannot be achieved.

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

264 Theory & Psychology 23(2)

The proposed deductive scheme can, by the way, also be used for SuH in the techno-
logical domain (e.g., psychotherapy research).
But the commonly encountered alternative to the procedure just outlined is to choose an
Experimental Design, collect data, and to analyze them with the analysis of variance
(ANOVA) appropriate for this design, thus following Fisher (1966): “Statistical procedure
and experimental design are only two different aspects of the same whole” (p. 3). But the
Experimental Design is only one determinant of the proper statistical tests. And: ANOVAs
test bidirectional hypotheses on means or on squared multiple correlations, whereas most
SuH are directional. (Any psychotherapy claims to help the clients to lead a better life.)

Conclusions
What about the questions this article began with?
What did Fisher, on the one hand, and Neyman and Pearson, on the other, express in
their respective theories? The main features of the theories of these authors have been
compiled in Table 1 and will not be repeated here.
Can the lack of random samples be incorporated into the theories? Of what help can
statistical tests be in examining substantive hypotheses?

The case for an SuH preceding statistical testing

Progress in scientific knowledge, be it cumulative or not (Kuhn, 1970; Mayo, 1996), can
only be achieved by examining SuH, not by testing isolated statistical hypotheses
(Neyman, 1950, p. 290). Therefore, research should begin with an SuH, often inspired by
a theory. As Popper (1981) formulated it: “[P]ure observational knowledge, unadultered
by theory, would, if at all possible, be utterly barren and futile” (p. 23).

The case for statistical testing (ST) I

According to Abelson (1997), “[W]e are awash in a sea of uncertainty, caused by a flood
tide of sampling and measurement errors” (p. 13). Statistical tests enable the researchers
to divide the total variation into an unsystematic plus a systematic part and an unsystem-
atic part (Abelson, 1995, p. 9; Fisher, 1956, p. 75). The criterion for this separation is
chosen by the researcher/s and varies over different experiments; it is probabilistic, thus
taking into account the fallibility of our data.

The case for ST II

Statistical testing is the most wide-spread strategy for data analysis and constitutes a
conventional procedure. According to Westermann (2000, p. 351), procedures like this
are necessary ingredients of domain-specific paradigms in the sense of Kuhn (1970), and
without them no systematic research is possible in normal science; therefore any ban on
statistical tests would be counterproductive and useless. But: Statistical testing should
always follow an SuH.

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 265

The case for the NPT I

If there is no random sampling, approximate randomization tests are applied, whose
results are exclusively valid for the sample under study. This approach precludes gener-
alizations and is consistent with the NPT.

The case for the NPT II

The examination of SuH is error-prone, and there are two probabilities of wrong
assertions, which the NPT allows to control, thus additionally conforming to the central
aim of Experimental Design: to minimize the frequency of wrong assertions. This fact
is often overlooked by authors arguing for a ban on statistical testing (e.g., Schmidt &
Hunter, 1997; see also Harlow et al., 1997).

The case for the NPT III

According to Neyman and Pearson, the truth or falsehood of statistical and substan-
tive hypotheses always remains unknown. We only act as if the hypothesis accepted/
corroborated were true. This opinion is shared by modern philosophers like Popper
and Lakatos.

The case for the NPT IV

After deriving a substantive prediction from the SuH, the conforming statistical hypoth-
esis is derived prior to experimentation (Meehl, 1967, 1997) and its complement is
stated. Only the NPT allows derivation of an H0 or an H1 on an a priori basis, both of
which can be accepted. This is necessary when SuH are appraised.

The case for the NPT V

The criteria for deciding on an SuH must be defined prior to experimentation (Lakatos,
Popper), as is the case with the NPT.

The case for the NPT VI

The ESs can be associated with the NPT (but not with the FST). They are computed
after the NP test with its dichotomous decision to get a qualified judgment concerning
the SuH.

The case for the NPT VII

The NPT explicitly addresses the many and mainly extra-statistical decisions the
researcher has to make before data gathering.

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

266 Theory & Psychology 23(2)

The case for the NPT VIII

Meehl (1978) commented on the FST: “[T]he almost universal reliance on merely refut-
ing the null hypothesis … is a terrible mistake, is basically unsound, poor scientific strat-
egy, and one of the worst things that ever happened in the history of psychology” (p. 817).
Oakes (1986) reasons: “[T]he statements emanating from a Neyman–Pearson analysis are
deductive truths (conditional upon a variety of assumptions). The charge against the
Neyman–Pearson doctrine is one of irrelevance. … We are offered rules for decisions with
ill-defined long-run error rates” (p. 145). Of course, the NPT should not be considered as
perfect and therefore is open to justified criticisms (e.g., Spielman, 1973). The same holds
for any other theory (Kuhn, 1970). Cortina and Dunlap (1997) correctly state: “The abuses
of [statistical tests] have come about largely because of a lack of judgment or education
with respect to those using the procedure. The cure lies in improving education and, con-
sequently, judgment, not in abolishing the method” (p. 171; Falk & Greenbaum, 1995, p.
94; Huberty, 1993, pp. 331–332). Neyman and Pearson (1928) add: “The [statistical] tests
should only be regarded as tools which must be used with discretion and understanding,
and not as instruments which in themselves give the final verdict” (p. 232). “There are no
objective procedures that avoid human judgment and guarantee correct interpretations of
results” (Abelson, 1997, p. 13). Hays (1988) correctly claims:

It is sad but true: the specifity of our final conclusion is more or less bought in terms of what
we already know or can at least assume to be true [as is the case, for example, with the
auxiliary hypotheses]. If we do not know or assume anything, we cannot conclude very
much. (p. 815)

At the end of this article it seems appropriate to return to some general non-statistical
aspects in order not to conclude with statistics (recency effect!).
There is no algorithm for the appropriate examination of a substantive hypothesis via
statistical hypotheses—at best there are some heuristics as presented above. Despite
these heuristics, the researchers’ decisions depend to a high degree on their knowledge
concerning many aspects partly addressed in this article. But whatever decisions are
chosen: “A principle … to remember is, keep it simple! Concentrate on how well and
carefully you can carry out a few meaningful manipulations” (Hays, 1994, p. 585; see
also Cohen, 1990, pp. 1304–1305). “Other things being equal, the simpler the experi-
ment, the better will its execution, and the more likely will one be able to decide what
actually happened and what the results really mean” (Hays, 1994, p. 518). Without the
choice of an appropriate experimental design and without specifying the CSES, no deri-
vations are possible. Therefore, it is necessary to pay more attention to designing and
planning an experiment carefully and to invoke statistical techniques in a later phase of
designing (Gigerenzer, 1987, p. 23; Preece, 1982, p. 204).
Overall, there are important reasons to continue with statistical testing, but judi-
ciously and with sound judgment; statistical tests, though important, are merely conveni-
ent tools for appraising SuH. The basis should be the NPT—the best testing theory
available, which, though, is not perfect. I recommend routinely choosing an extra-
statistical viewpoint such as substantive hypotheses, a methodology, some philosophy or
epistemology as a general background for discussing statistical problems (Meehl, 1997;

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 267

Serlin & Lapsley, 1993)—the isolated discussion of statistical problems has rarely been
beneficial.
In view of the long duration of the discussion about statistical testing, one question
arises: What are the reasons for not inventing a superior theory to replace the NPT?

Funding
This research received no specific grant from any funding agency in the public, commercial, or
not-for-profit-sectors.

Notes
1. Some auxiliary hypotheses are testable, and some are not. The latter constitute assumptions
without empirical content and are accepted in research practice as conventions (Westermann,
2000). That randomization is effective is a non-testable auxiliary hypothesis. Choosing a
standard value for α constitutes a convention (Hays, 1994, p. 333). See on the necessity of
testable and non-testable auxiliary hypotheses in research, for example, Chow (1996), Lakatos
(1978), Meehl (1978, 1997), and Popper (1934/1992).
2. Some authors claim the researchers are interested in this probability (Cohen, 1994), but
Chalmers (1999, pp. 141–154) found no modern philosophical conceptions of science which
deal with the probability of a hypothesis, so that Bayesianism does not seem appropriate for
replacing statistical testing (Mayo, 1996)—despite its many defenders (e.g., Pruzek, 1997;
Rindskopf, 1997).
3. “Corroboration” is a term Popper (1934/1992, pp. 251–252) uses to avoid (inductive) “confir-
mation” or “verification”; it is not a “truth value” (Popper, 1934/1992, p. 275).
4. No one gives an example. I have scanned more than 150 substantive hypotheses (Hager, 2004),
and none led directly to an exact value of the ES; in seven cases precise predictions/expectations,
not necessarily connected with a SuH, were examined.

References
Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Erlbaum.
Abelson, R. P. (1997). On the surprising longevity of flogged horses: Why there is a case for sig-
nificance tests. Psychological Science, 8, 12–15.
Anderson, M. J., & Ter Braak, C. F. J. (2003). Permutation tests for multifactorial analysis of vari-
ance. Journal of Statistical Computation and Simulation, 73, 85–113.
Berenson, M. L. (1982). A comparison of several k sample tests for ordered alternatives in com-
pletely randomized designs. Psychometrika, 47, 265–280.
Boik, R. J. (1987). The Fisher–Pitman permutation test: A non-robust alternative to the normal
theory F when variances are heterogeneous. British Journal of Mathematical and Statistical
Psychology, 40, 26–42.
Chalmers, A. F. (1999). What is this thing called science? (3rd ed.). St. Lucia, Australia: University
of Queensland Press.
Chow, S. L. (1996). Statistical significance: Rationale, validity and utility. London, UK: Sage.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ:
Erlbaum.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation. Boston, MA: Houghton
Mifflin.

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

268 Theory & Psychology 23(2)

Cortina, J. M., & Dunlap, W. P. (1997). On the logic and purpose of significance testing.
Psychological Methods, 2, 161–172.
Cowles, M. (1989). Statistics in psychology: An historical perspective. Hillsdale, NJ: Erlbaum.
Davison, A. C., & Hinkley, D. (2006). Bootstrap methods and their application (8th ed.).
Cambridge, UK: Cambridge University Press.
Edgington, E. S. (1966). Statistical inference and nonrandom samples. Psychological Bulletin, 66,
485–487.
Edgington, E. S. (1969). Approximate randomization tests. The Journal of Psychology, 62,
143–149.
Edgington, E. S. (1995). Randomization tests (3rd ed.). New York, NY: Dekker.
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Andover, MA: Chapman
& Hall.
Falk, R., & Greenbaum, C. W. (1995). Significance tests die hard. Theory & Psychology, 5, 75–98.
Fisher, R. A. (1935). The logic of inductive inference. Journal of the Royal Statistical Society, 98,
39–82.
Fisher, R. A. (1947). Development of the theory of experimental design. Proceedings of the
International Statistical Conferences, 3, 434–439.
Fisher, R. A. (1950). Statistical methods for research workers (11th ed.). Edinburgh, UK: Oliver
& Boyd.
Fisher, R. A. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical
Society, B, 17, 69–78.
Fisher, R. A. (1956). Statistical methods and scientific inference. Edinburgh, UK: Oliver &
Boyd.
Fisher, R. A. (1966). The design of experiments (8th ed.). Edinburgh, UK: Oliver & Boyd.
Frick, R. W. (1995). Accepting the null hypothesis. Memory & Cognition, 23, 132–138.
Frick, R. W. (1998). Interpreting statistical testing: Process and propensity, not population and
random sampling. Behavior Research Methods, Instruments, and Computers, 30, 527–535.
Frick, R. W. (1999). Defending the statistical staus quo. Theory & Psychology, 9, 183–189.
Gigerenzer, G. (1987). Probabilistic thinking and the fight against subjectivity. In L. Krüger,
G. Gigerenzer, & M. Morgan (Eds.), The probabilistic revolution: Vol. 2. Ideas in the sciences
(pp. 11–33). Cambridge, MA: MIT Press.
Gigerenzer, G. (1993). The superego, the ego, and the id in statistical reasoning. In G. Keren &
C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological
issues (pp. 311–339). Hillsdale, NJ: Erlbaum.
Gigerenzer, G., & Murray, D. J. (1987). Cognition as intuitive statistics. Hillsdale, NJ: Erlbaum.
Gonzalez, R. (1994). The statistics ritual in psychological research. Psychological Science, 5, 321,
325–328.
Hacking, I. (1965). Logic of statistical inference. Cambridge, UK: Cambridge University Press.
Hager, W. (2004). Testplanung zur statistischen Prüfung psychologischer Hypothesen [Power anal-
ysis for the statistical examination of psychological hypotheses]. Göttingen, Germany: Hogrefe.
Halpin, P. F., & Stam, H. J. (2006). Inductive inference or inductive behavior: Fisher and Neyman–
Pearson approaches to statistical testing in psychological research (1940–1960). American
Journal of Psychology, 119, 625–653.
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance
tests? Mahwah, NJ: Erlbaum.
Hays, W. L. (1988). Statistics (4th ed.). London, UK: Holt, Rinehart & Winston.
Hays, W. L. (1994). Statistics (5th ed.). Belmont, CA: Wadsworth.
Hubbard, R. (2004). Alphabet soup: Blurring the distinction between p’s and α’s in psychological
research. Theory & Psychology, 14, 295–327.
Huberty, C. J. (1987). On statistical testing. Educational Researcher, 16(8), 4–9.

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

Hager 269

Huberty, C. J. (1993). Historical origins of statistical testing practices: The treatment of Fisher vs.
Neyman–Pearson in textbooks. Journal of Experimental Education, 61, 317–333.
Huberty, C. J., & Pike, C. J. (1999). On some history regarding statistical testing. In B. Thompson
(Ed.), Adcances in social science methodology (Vol. 5, pp. 1–22). Stamford, CT: JAI Press.
Hunter, M. A., & May, R. B. (2003). Statistical testing and null distributions: What to do when
samples are not random? Canadian Journal of Experimental Psychology, 57, 176–188.
Killeen, P. R. (2005). An alternative to null hypothesis significance tests. Psychological Science,
16, 345–353.
Kraemer, H. C., & Thiemann, S. (1987). How many subjects? Statistical power analysis in
research. Newbury Park, CA: Sage.
Kuhn, T. S. (1970). The structure of scientific revolutions (2nd ed.). Chicago, IL: University of
Chicago Press.
Lakatos, I. (1978). The methodology of scientific research programmes. In J. Worrall & G. Currie
(Eds.), Imre Lakatos: Philosophical papers (Vol. 1, pp. 8–101). Cambridge, UK: Cambridge
University Press.
Mayo, D. G. (1996). Error and the growth of experimental knowledge. Chicago, IL: University of
Chicago Press.
Meehl, P. E. (1967). Theory testing in psychology and physics: A methodological paradox.
Philosophy of Science, 34, 103–115.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow
progress in psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
Meehl, P. E. (1997). The problem is epistemology, not statistics: Replace significance tests by
confidence intervals and quantify accuracy of risky numerical predictions. In L. L. Harlow,
S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 393–425).
Mahwah, NJ: Erlbaum.
Neyman, J. (1942). Basic ideas and some recent results of the theory of testing statistical hypotheses.
Journal of the Royal Statistical Society, 105, 292–327.
Neyman, J. (1950). First course in probability and statistics. New York, NY: Holt, Rinehart & Winston.
Neyman, J. (1952). Lectures and conferences on mathematical statistics and probability (2nd ed.).
Washington, DC: U.S. Department of Agriculture.
Neyman, J. (1957). “Inductive behavior” as a basic concept of philosophy of science. Revue
d’Institute Internationale de Statistique, 25, 7–22.
Neyman, J. (1967). R. A. Fisher (1890–1962): An appreciation. Science, 156, 1456–1460.
Neyman, J. (1971). Foundations of behavioristic statistics (with comments). In V. P. Godambe &
D. A. Sprott (Eds.), Foundations of statistical inference (pp. 1–19). Toronto, Canada: Holt,
Rinehart & Winston.
Neyman, J. (1976). Tests of statistical hypotheses and their use in studies of natural phenomena.
Communications in Statistics—Theory and Methods, A5, 737–751.
Neyman, J. (1977). Frequentist probability and frequentist statistics. Synthese, 36, 97–131.
Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for
purposes of statistical inference. Biometrika, 20A, Pt. I: 175–240; Pt. II: 263–294.
Neyman, J., & Pearson, E. S. (1933a). On the problem of the most efficient tests of statistical
hypotheses. Philosophical Transactions of the Royal Society, A, 231, 289–337.
Neyman, J., & Pearson, E. S. (1933b). The testing of hypotheses in relation to probabilities a
priori. Proceedings of the Cambridge Philosophical Society, 29, 492–510.
Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing
controversy. Psychological Methods, 5, 241–301.
Oakes, M. (1986). Statistical inference: A comment for the social and behavioural sciences.
Chichester, UK: Wiley.
Paivio, A. (1971). Imagery and verbal processes. New York, NY: Holt, Rinehart & Winston.

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014

270 Theory & Psychology 23(2)

Paivio, A. (1990). Mental representations. A dual coding approach (2nd ed.). New York, NY:
Oxford University Press.
Pearson, E. S. (1955). Statistical concepts in their relation to reality. Journal of the Royal Statistical
Society, B, 17, 204–207.
Pearson, E. S. (1962). Some thoughts of statistical inference. Annals of Mathematical Statistics,
33, 394–403.
Polkinghorne, D. (1983). Methodology for the human sciences: Systems of inquiry. Albany: State
University of New York Press.
Popper, K. R. (1981). Conjectures and refutations: The growth of scientific knowledge (4th ed.).
London, UK: Routledge & Kegan Paul.
Popper, K. R. (1992). The logic of scientific discovery. London, UK: Hutchinson. (Original work
published 1934)
Preece, D. A. (1982). The design and analysis of experiments: What has gone wrong? Utilitas
Mathematica, Series A, 21, 201–244.
Pruzek, R. M. (1997). An introduction to Bayesian inference and its applications. In L. L. Harlow,
S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 287–318).
Mahwah, NJ: Erlbaum.
Reichardt, C. S., & Gollob, H. F. (1999). Justifying the use and increasing the power of a t test for
a randomized experiment with a convenience sample. Psychological Methods, 4, 117–128.
Rindskopf, D. M. (1997). Testing “small,” not null, hypotheses: Classical and Bayesian approaches.
In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests?
(pp. 319–332). Mahwah, NJ: Erlbaum.
Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge
in psychological science. American Psychologist, 44, 1276–1284.
Scheffé, H. (1959). The analysis of variance. New York, NY: Wiley.
Schmidt, F. L., & Hunter, J. E. (1997). Eight common but false objections to the discontinuation
of significance testing in the analysis of research data. In L. L. Harlow, S. A. Mulaik, & J. H.
Steiger (Eds.), What if there were no significance tests? (pp. 37–64). Mahwah, NJ: Erlbaum.
Serlin, R. C., & Lapsley, D. K. (1993). Rational appraisal of psychological research and the
good-enough principle. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the
behavioral sciences: Methodological issues (pp. 199–228). Hillsdale, NJ: Erlbaum.
Spielman, S. (1973). A refutation of the Neyman–Pearson theory of testing. British Journal for the
Philosophy of Science, 24, 201–222.
Still, A. W., & White, A. F. (1981). The approximate randomization test as an alternative to the F test
in analysis of variance. British Journal of Mathematical and Statistical Psychology, 34, 243–252.
Thompson, B. (1993). The use of statistical significance tests in research: Bootstraps and other
alternatives. Journal of Experimental Education, 61, 361–377.
Westermann, R. (2000). Wissenschaftstheorie und Experimentalmethodik [Philosophy of science
and experimental methods]. Göttingen, Germany: Hogrefe.
Wilkinson, L., and the Task Force on Statistical Inference. (1999). Statistical methods in psychology
journals: Guidelines and explanations. American Psychologist, 54, 594–604.
Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design
(3rd ed.). New York, NY: McGraw-Hill.

Author biography
Willi Hager is professor at the University of Göttingen, Department of Psychology. He is particu-
larly interested in univariate methods for the examination of substantive hypotheses with special
reference to various forms of power analyses. In addition, he deals with methodologies and
methods in general. Address: Institute for Psychology, University of Goettingen, Gosslerstr. 14,
D-37073 Göttingen, Germany. Email: whager@uni-goettingen.de

Downloaded from tap.sagepub.com at SUB Goettingen on September 9, 2014