EFFECT SIZE, HYPOTHESIS TESTING,

AND CONFIDENCE INTERVAL

PRINCIPLE AND APLICATION

By
Ida Bagus Subanada
SPECIFIC LEARNING OUTCOME

1. The student is expected to be able to understand

the meaning of the effect size, the principle of
hypothesis testing, and the principle of
confidence interval.

2. The student also expected to be able to

elucidate the relationship between effect size,
confidence interval, and hypothesis testing.

3. The student to be able to write the main results

in an abstract of a report showing effect size,
confidence interval, and P-value.
EFFECT SIZE (ES)
• Effect size (delta) is estimation of clinical
outcome differentiation (e.g. RR, OR, mean
difference, difference of proportion, etc.).

• Sometimes called point estimate.

• Need hypothesis testing to determine whether

there is a statistically significant of those effect
size.

• Decided by investigator based on clinical

judgment (not based on references).
• On positive hypothesis, ES is the smallest
difference that clinically important.

• On negative hypothesis, ES is the largest

difference that clinically not important.

• Sample size influenced by effect size

(Effect size↓ sample size ↑)
 ………ES
Sample size formula:
1. 2
 ( Z  Z  ) S 
n1  n2  2  
 ( X 1  X 2 ) 
2. (Z 2PQ  Z  P1Q1  P2Q2 ) 2
n1  n2 
( P1  P2 ) 2

Note: 1
P ( P1  P2 )
2

ES on these formula are:

X1 – X2 dan P1 – P2
 ………ES

Example:
Journal A reported that the recovery rate of
drug 1 for a disease was 50% and drug 2
was 80%. If we shall conduct a replicative
study, it is wrong if we use 30% (80%-50%)
as the value of effect size. There are two
reason for these:
1. Conceptual reason
2. Technical reason
Example of no effect

Effect size measured as No effect

Difference in prevalence 0
Risk difference 0
Mean difference 0
Mean change 0
Correlation 0
Prevalence ratio 1
Relative risk 1
Odds ratio 1
Rate ratio 1
HYPOTHESIS TESTING (HT)

• Is a method to answer the question:

Whether results of the investigation in sample
would be similar if the investigation include
entire population?
 ………HT

• Most studies are conducted on sample only

Whether the result of study in sample would be

similar if the study include entire population ?

To generalize or to draw an inference of study

result in sample, can be done by 2 methods:
1. HT (the end result is P-value)
2. Estimation, to get confidence interval (CI)

• HT provides yes/no answer whether the ES occur

in population rather than samples only, that may
due to sample variation.
 ……...HT

• Just answer whether or not there is

statistical significance.

• Give no information about the magnitude

of difference among groups.
 ……...HT
There are 2 types of hypothesis testing:
1. Null Hypothesis (H0):
there is no association between the
predictor and outcome variables in
the population.

• The formal basis for testing statistical

significance (hypothesis testing based
on H0).
• Statistical test help to estimate the
probability that an association observed
in a study is due to chance.
 ………HT

2. Alternative Hypothesis (HA):

there is association between the
predictor and outcome variables in the
population.

* Cannot be tested directly; it is accepted

by default if the test of statistical
significance rejects the null hypothesis.
 ………HT

• The end result of hypothesis testing is

P-value.

• P-value refer to the probability to get the

observed results (or more extreme results)
if null hypothesis is true.
OR
P-value is the probability that observed
data or outcome would have occurred by
chance when null hypothesis is true.
Example: P = 0.085
• If null hypothesis is true, the probability to
get the observed results or more extreme
results) is 0.085 (8.5%).
OR
If null hypothesis is true, the probability
that observed data or outcome would have
occurred by chance is 8.5%.
 ………HT
Recovery Dead Total
n (%) n (%) n

Standard Drug 60 (60%) 40 (40%) 100

“X” Drug 75 (75%) 25 (25%) 100

Total 135 65 200

X2 = 4.467; df = 1; P = 0.035

P-value interpretation:
If the effectiveness of “X” drug = standard drug, there is
0.035 (3.5%) of probability to get 15% or more of recovery
difference.
OR

If the effectiveness of “X” drug = standard

drug, the probability that observed data or
outcome would have occurred by chance
is 3,5%.
 ………HT

Errors in hypothesis testing

• There are 2 types error:

Population
Different No different
Power
Reject H0 (1 – beta) alpha

Sample
No reject H0 beta (1– alpha)
 ………HT

1. Type I error (false positive = alpha) occurs

if an investigator rejects a null hypothesis
that is actually true in the population.

2. Type 2 error (false negative = beta)

occurs if the investigator fails to reject a
null hypothesis that is actually not true in
the population.
Hipothesis testing ………HT

1. Non-parametric:
* Fisher exact test, Chi-square, Mc Nemar
* Mann-Whitney, Kruskal-Wallis, Kendall’s r,
Wilcoxon, Friedman.
2. Parametric @:
* t test, Anova, Manova, Hotelling’s T2,
Correlasi Pearson

Note: @: Distribution of the variable approxi-

mates be normal (bell-shaped) curve.
 ………HT
Table 1. Hypothesis testing based on scale variable
on independent and dependent variables (1
independent variable = univariate analysis)

Independ. Var. Depend. Var. Hypothesis testing

Nominal Nominal Chi-square, Fischer
exact test, Mc Nemar
Nominal Numeric t test
(dichotomous)
Nominal (> 2 Numeric Anova
value)
Numeric Numeric Regression/
Correlation
 ………HT

Table 2. Multivariate analysis (> 1 independent

variable)

Independ. Var. Depend. Var. Hypothesis testing

Numeric Numeric Multiple

regression
Nominal and/or Nominal Logistic
numeric regression
CONFIDENCE INTERVAL (CI)
• Beside P-value, study report usually reported CI
because give more information about the study results
and recommended by many outstanding medical
journal.

• CI shows a range of values (e.g. mean difference,

difference of proportion, etc.) that more suitable with
the situation in the population.

• There is close relationship between CI and two-side

hypothesis testing.
P = 0.05 (2 sided test) 95% CI
P = 0.01 (2 sided test) 99% CI
 ………CI
• CI depend on 2 factors:

1. Standard error (SE), hence depends on

SD (standard deviation) and N (sample size)
SD ↑ CI ↑
Sample size ↑ CI ↓

2. The degree of CI
99% CI >> 95% CI
Ex.: Mean difference of blood pressure was 5 (95% CI 3 to 7) mmHg
Mean difference of blood pressure was 5 (99% CI -0.5 to10.5)
mmHg
 ………CI
CI formula

CI = P ± (Zα x SE)

CI = confidence interval
P = point estimate = effect size
(e.g. mean difference,
difference of proportion, etc.)
Zα = normal standard deviate,
e.g. 95% CI ~ α = 0.05
Zα = 1.96
99% CI ~ α = 0.01
Zα = 2.58
SE = standard error
 ………CI
CALCULATION OF CI

[1]. 95% CI:

•Upper bound: effect size of outcome + 1.96 SE
•Lower bound: effect size of outcome – 1.96 SE

[2]. 99% CI:

•Upper bound: effect size of outcome + 2.58 SE
•Lower bound: effect size of outcome – 2.58 SE
 ……..CI
• CI of mean difference and difference of
proportion are symmetrical to point estimate.
If zero (“0”) is include in the range of CI
there are not statistically significant.
• CI of RR and OR are asymmetrical to point
estimate.
If one (“1”) is include in the range of CI
there are not statistically significant.
 ……..CI

Example:
• Mean difference of birth weight between
rural and urban of newborn babies was 500
gram, 95% CI 300 to 700, P= 0.002
statistically significant.

•Mean difference of birth weight between

rural and urban of newborn babies was 500
gram, 95% CI -50 to 1.050, P= 0.09
not statistically significant.
 ……..CI

Example:
• OR to suffer from bronchiolitis acute for
babies who exposure to tobacco smoke is
1.29, 95% CI 0.82 to 2.01, P=0.302
not statistically significant.

• OR to suffer from bronchiolitis acute for

babies who exposure to tobacco smoke is
1.29, 95% CI 1.21 t0 1.77, P=0.014
statistically significant.
Relationship between effect size, hypothesis
testing, and confidence interval

Effect size Sample

(sample size)
Inferention/
generalization

Hypothesis Estimation
testing (p-value) (confidence
interval)

Population
How to present data with effect size, confidence
interval, and P-value ?

Example:

Mean difference of blood pressure between diabetic and

non-diabetic patients was 6.0 mmHg, 95% CI 1.1 to 10.9
mmHg, P = 0.02.

Relative risk to suffer from lung cancer for smoker

was 3.3, 95% CI 2.9 to 4.2, P=0.001.