Tests of Significance and

Measures of Association

Definitons

Test of Significance Given a random sample drawn from a

population, a test of significance is a formal test evaluating
the probability that an event or statistical result based on
the sample could have happened by random chance.

With most tests of significance we look for low p-values. The

lower the p-value, the lower the probability that the event
or result could have happened by random chance. What a
p-value tells us is that 1-the p-value percent of the
possible samples we could have drawn from the population
would contain our estimate.

Null and Alternative

Hypotheses
Null hypothesis Often labeled H0. One perspective: an

estimated result is due to random sampling error. A

statement that the estimated parameter or parameter
difference is random in the population.

Null hypothesis Another perspective is that the estimated

result is consistent with some externally imposed value, say
due to theory or prior research.

Alternative hypothesis Often labeled Ha. Your estimated

result. If we are to have confidence in an estimated result,
then we need evidence that the result could not have
resulted from random chance.

In inferring from a sample to a population, we can be wrong

in either of two ways.

False rejection error. Falsely rejecting a true null hypothesis.

This type of error is usually called Type I Error. Here the
researcher concludes there is a relationship in the
population, when there is actually none.

False acceptance error. Falsely accepting a false null

hypothesis. This type of error is usually called Type II Error.
Here the researcher concludes there is no relationship in the
population, when there actually is one.

Which type of error is worse (if you dont want to mislead

science)? Type I error is worse. Why?

Consider the following graph as a way of organizing your

thoughts on Type I and II error.

Comparing the Means of Two

Groups

Consider the variables from the National Election Studies which

give respondents feeling thermometer about the Democratic
party.

Men and women differ in their feelings about the Democratic

party. However, in order to test the significance of these
differences we need the standard error of the mean differences.
If the variances of the two samples are the same, then we can
simply calculate these from the standard errors for men and
women.

Square the two estimates of the standard error for the two
samples for men and women. Sum the squared standard errors.
Take the square root of the sum of the squared standard errors.

That is,

Two Approaches to
Hypothesis Testing

Confidence Interval Approach- Here we use the standard error to

determine the smallest plausible mean difference in the population.
Or

P-value approach- Here we determine the exact probability of

obtaining the observed sample difference, given that the null
hypothesis is true.

Look up the appropriate p-value in the distribution. The tdistribution depends on the degrees of freedom. Last week
we estimated one parameter, so the degrees of freedom
was N-1. Here we are estimating two parameters so the
degrees of freedom is N-2.
Alternatively, let STATA compute the p-value. It returns a pvalue of 0.0005. In other words, given the sample drawn,
there is only a 0.0005 chance that the true mean difference
in the population is zero. Said differently, only 5 in 10,000 of
the samples drawn from the population would report a
mean of zero.

Comparing Sample
Proportions
The standard error of a difference in proportions is easy to

compute.
Let p1 and p2 be the proportions for two groups. Then, let
q1=(1-p1) and q2=(1-p2) be the complements of these two
proportions. Then, the standard error of the difference in
proportions is given

Example

Is this a statistically significant difference? Using the rule of

thumb that if the estimated coefficient is roughly twice the
standard error, the difference is statistically significant.

We observe an estimated coefficient (0.093) about 3 times

as large as its standard error (0.031).

Chi-square Test of
Significance
A Chi-square test evaluates whether the observed

dispersion of cases deviates from what one would expect if

the null hypothesis were correct.

The Chi-square test is commonly used when one is

conducting a cross-tabular analysis.

The test is based on the difference between the observed

frequencies in a table versus the expected frequency if
there is no relationship.

Example:

Summing the numbers in bold the table, we have a Chisquared statistic of 15.2. We compare this statistic to a
table of Chi-squared statistics with degrees of freedom
equal to (r-1)(c-1)

Measures of Association

Statisticians have developed a number of measures of

association for evaluating the strength of relationships.

PRE- Proportional Reduction in Error. PRE measures give a

number that ranges from 0 to 1. 0 represents no association
and 1 represents a perfect association.

Lambda and Somers d are two different measures of

association, commonly used in evaluating relations in a
cross-tabular analysis.

Here there are 226+358=584 errors in prediction when

hypothesizing that gender makes a difference. Calculate

Somers dyx- An alternative measure of association which

counts concordant and discordant pairs.

Clowfrequent=7*5+7*6=77; Clowoccasional=5*6=30
Thus, there are 77+30=107 pairs that are concordant with
there being a positive relationship. Work diagonally and
down.
Dhighfrequent=3*5+3*4=27; Dhighoccasional=5*4=20
Thus, there are 47 pairs that are discordant with there being
a positive relationship. Work diagonally and up.
There are 60 more pairs that fit a positive than negative
relationship.
However, we need the number of ties to convert this
number to a measure of association from 0 to 1.
Tlow=7*5+7*4+5*4=83; Thigh=3*5+3*6+5*6=63; T=146