KTEE 310 FINANCIAL

ECONOMETRICS
THE SIMPLE REGRESSION MODEL
Chap 4 S & W
1
Dr TU Thuy Anh
Faculty of International Economics
Output and labor use
2
0
50
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Labor Output





1
Output and labor use
The scatter diagram shows output q plotted against labor use l for a sample
of 24 observations
0
50
100
150
200
250
300
80 100 120 140 160 180 200 220
Output vs labor use
4
Output and labor use
Economic theory (theory of firms) predicts
An increase in labor use leads to an increase
output in the SR, as long as MP
L
>0, other things
being equal
From the data:
consistent with common sense
the relationship looks linear
Setting up:
Want to know the impact of labor use on output
=> Y: output, X: labor use

1
Y
SIMPLE LINEAR REGRESSION MODEL
Suppose that a variable Y is a linear function of another variable X, with
unknown parameters |
1
and |
2
that we wish to estimate.
X Y
2 1
| | + =
|
1

X X
1
X
2
X
3
X
4

Suppose that we have a sample of 4 observations with X values as shown.
SIMPLE LINEAR REGRESSION MODEL
2
X Y
2 1
| | + =
|
1

Y
X X
1
X
2
X
3
X
4

If the relationship were an exact one, the observations would lie on a
straight line and we would have no trouble obtaining accurate estimates of |
1

and |
2
.
Q
1

Q
2

Q
3

Q
4

SIMPLE LINEAR REGRESSION MODEL
3
X Y
2 1
| | + =
|
1

Y
X X
1
X
2
X
3
X
4

P
4

In practice, most economic relationships are not exact and the actual values
of Y are different from those corresponding to the straight line.
P
3

P
2

P
1

Q
1

Q
2

Q
3

Q
4

SIMPLE LINEAR REGRESSION MODEL
4
X Y
2 1
| | + =
|
1

Y
X X
1
X
2
X
3
X
4

P
4

To allow for such divergences, we will write the model as Y = |
1
+ |
2
X + u,
where u is a disturbance term.
P
3

P
2

P
1

Q
1

Q
2

Q
3

Q
4

SIMPLE LINEAR REGRESSION MODEL
5
X Y
2 1
| | + =
|
1

Y
X X
1
X
2
X
3
X
4

P
4

Each value of Y thus has a nonrandom component, |
1
+ |
2
X, and a random
component, u. The first observation has been decomposed into these two
components.
P
3

P
2

P
1

Q
1

Q
2

Q
3

Q
4

u
1

SIMPLE LINEAR REGRESSION MODEL
6
X Y
2 1
| | + =
|
1

Y
1 2 1
X | | +
X X
1
X
2
X
3
X
4

P
4

In practice we can see only the P points.
P
3

P
2

P
1

SIMPLE LINEAR REGRESSION MODEL
7
Y
X X
1
X
2
X
3
X
4

P
4

Obviously, we can use the P points to draw a line which is an approximation
to the line
Y = |
1
+ |
2
X. If we write this line Y = b
1
+ b
2
X, b
1
is an estimate of |
1
and b
2

is an estimate of |
2
.
P
3

P
2

P
1

^
SIMPLE LINEAR REGRESSION MODEL
8
X b b Y
2 1

+ =
b
1

Y
X X
1
X
2
X
3
X
4

P
4

The line is called the fitted model and the values of Y predicted by it are
called the fitted values of Y. They are given by the heights of the R points.
P
3

P
2

P
1

R
1

R
2

R
3

R
4

SIMPLE LINEAR REGRESSION MODEL
9
X b b Y
2 1

+ =
b
1

Y

(fitted value)
Y (actual value)
Y
X X
1
X
2
X
3
X
4

P
4

X X
1
X
2
X
3
X
4

The discrepancies between the actual and fitted values of Y are known as the
residuals.
P
3

P
2

P
1

R
1

R
2

R
3

R
4

(residual)
e
1

e
2

e
3

e
4

SIMPLE LINEAR REGRESSION MODEL
10
X b b Y
2 1

+ =
b
1

Y

(fitted value)
Y (actual value)
e Y Y =

Y
P
4

Note that the values of the residuals are not the same as the values of the
disturbance term. The diagram now shows the true unknown relationship as
well as the fitted line.
The disturbance term in each observation is responsible for the divergence
between the nonrandom component of the true relationship and the actual
observation.
P
3

P
2

P
1

SIMPLE LINEAR REGRESSION MODEL
12
Q
2

Q
1

Q
3

Q
4

X b b Y
2 1

+ =
X Y
2 1
| | + =
|
1

b
1

Y

(fitted value)
Y (actual value)
Y
X X
1
X
2
X
3
X
4

unknown PRF
estimated
SRF
P
4

The residuals are the discrepancies between the actual and the fitted values.
If the fit is a good one, the residuals and the values of the disturbance term
will be similar, but they must be kept apart conceptually.

P
3

P
2

P
1

R
1

R
2

R
3

R
4

SIMPLE LINEAR REGRESSION MODEL
13
X b b Y
2 1

+ =
X Y
2 1
| | + =
|
1

b
1

Y

(fitted value)
Y (actual value)
Y
X X
1
X
2
X
3
X
4

unknown PRF
estimated
SRF
17
( ) L
b b
Min
i
X b b Y
b b
Min
i
e
b b
Min RSS
b b
Min
e X b b Y
e X b b Y
2
,
1
2
2 1
2
,
1
2
2
,
1 2
,
1
2 1
2 1
=

=
=
+ + =
You must prevent negative residuals from cancelling positive ones, and one
way to do this is to use the squares of the residuals.

We will determine the values of b
1
and b
2
that minimize RSS, the sum of the squares of the
residuals.
Least squares criterion:
0
1
2
3
4
5
6
0 1 2 3
1
Y
2
Y
3
Y
DERIVING LINEAR REGRESSION COEFFICIENTS
Y
X b b Y
u X Y
2 1
2 1

: line Fitted
: model True
+ =
+ + = | |
X
This sequence shows how the regression coefficients for a simple regression model are
derived, using the least squares criterion (OLS, for ordinary least squares)
We will start with a numerical example with just three observations: (1,3), (2,5), and (3,6)
1
0
1
2
3
4
5
6
0 1 2 3
1
Y
2
Y
3
Y
2 1 1

b b Y + =
2 1 2
2

b b Y + =
2 1 3
3

b b Y + =
DERIVING LINEAR REGRESSION COEFFICIENTS
Y
b
2

b
1

X
Writing the fitted regression as Y = b
1
+ b
2
X, we will determine the values of b
1
and b
2
that
minimize RSS, the sum of the squares of the residuals.
3
^
X b b Y
u X Y
2 1
2 1

: line Fitted
: model True
+ =
+ + = | |
0
1
2
3
4
5
6
0 1 2 3
1
Y
2
Y
3
Y
2 1 1

b b Y + =
2 1 2
2

b b Y + =
2 1 3
3

b b Y + =
Given our choice of b
1
and b
2
, the residuals are as shown.
DERIVING LINEAR REGRESSION COEFFICIENTS
Y
b
2

b
1

2 1 3 3 3
2 1 2 2 2
2 1 1 1 1
3 6

2 5

b b Y Y e
b b Y Y e
b b Y Y e
= =
= =
= =
4
X b b Y
u X Y
2 1
2 1

: line Fitted
: model True
+ =
+ + = | |
X
2
2 1
2
2 1
2
2 1
2
3
2
2
2
1
) 3 6 ( ) 2 5 ( ) 3 ( b b b b b b e e e RSS + + = + + =
SIMPLE REGRESSION ANALYSIS
0 28 12 6 0
2 1
1
= + =
c
c
b b
b
RSS
0 62 28 12 0
2 1
2
= + =
c
c
b b
b
RSS
50 . 1 , 67 . 1
2 1
= = b b
The first-order conditions give us two equations in two unknowns. Solving them, we find
that RSS is minimized when b
1
and b
2
are equal to 1.67 and 1.50, respectively.
10
2
2 1
2
2 1
2
2 1
2
3
2
2
2
1
) 3 6 ( ) 2 5 ( ) 3 ( b b b b b b e e e RSS + + = + + =
0
1
2
3
4
5
6
0 1 2 3
1
Y
2
Y
3
Y
17 . 3

1
= Y
67 . 4

2
= Y
17 . 6

3
= Y
DERIVING LINEAR REGRESSION COEFFICIENTS
Y
X Y
u X Y
50 . 1 67 . 1

: line Fitted
: model True
2 1
+ =
+ + = | |
X
The fitted line and the fitted values of Y are as shown.
12
1.50
1.67
DERIVING LINEAR REGRESSION COEFFICIENTS
X X
n
X
1

Y
X b b Y
u X Y
2 1
2 1

: line Fitted
: model True
+ =
+ + = | |
1
Y
n
Y
Now we will do the same thing for the general case with n observations.
13
DERIVING LINEAR REGRESSION COEFFICIENTS
X X
n
X
1

Y
b
1

X b b Y
u X Y
2 1
2 1

: line Fitted
: model True
+ =
+ + = | |
1 2 1 1

X b b Y + =
1
Y
b
2

n
Y
n n
X b b Y
2 1

+ =
Given our choice of b
1
and b
2
, we will obtain a fitted line as shown.
14
DERIVING LINEAR REGRESSION COEFFICIENTS
The residual for the first observation is defined.
Similarly we define the residuals for the remaining observations. That for the last one is
marked.
X X
n
X
1

Y
b
1

X b b Y
u X Y
2 1
2 1

: line Fitted
: model True
+ =
+ + = | |
n n n n n
X b b Y Y Y e
X b b Y Y Y e
2 1
1 2 1 1 1 1 1

.....

= =
= =
1 2 1 1

X b b Y + =
1
Y
b
2

n
Y
1
e
n
e
n n
X b b Y
2 1

+ =
16
26
( ) L
b b
Min
i
X b b Y
b b
Min
i
e
b b
Min RSS
b b
Min
e X b b Y
e X b b Y
2
,
1
2
2 1
2
,
1
2
2
,
1 2
,
1
2 1
2 1
=

=
=
+ + =
We will determine the values of b
1
and b
2
that minimize RSS, the sum of the squares of the
residuals.
Least squares criterion:
DERIVING LINEAR REGRESSION COEFFICIENTS
X X
n
X
1

Y
b
1

X b b Y
u X Y
2 1
2 1

: line Fitted
: model True
+ =
+ + = | |
1 2 1 1

X b b Y + =
1
Y
b
2

n
Y
n n
X b b Y
2 1

+ =
X b Y b
2 1
=
We chose the parameters of the fitted line so as to minimize the sum of the squares of the
residuals. As a result, we derived the expressions for b
1
and b
2
using the first order
condition
40
( )( )
( )

=
2
2
X X
Y Y X X
b
i
i i
Practice calculate b1 and b2
Year Output - Y Labor - X
1899 100 100
1900 101 105
1901 112 110
1902 122 118
1903 124 123
1904 122 116
1905 143 125
1906 152 133
1907 151 138
1908 126 121
1909 155 140
1910 159 144
1911 153 145
1912 177 152
1913 184 154
1914 169 149
1915 189 154
1916 225 182
1917 227 196
1918 223 200
1919 218 193
1920 231 193
1921 179 147
1922 240 161
28
29
Model 1: OLS, using observations 1899-1922 (T = 24)
Dependent variable: q

coefficient std. error t-ratio p-value
---------------------------------------------------------
const -38,7267 14,5994 -2,653 0,0145 **
l 1,40367 0,0982155 14,29 1,29e-012 ***

Mean dependent var 165,9167 S.D. dependent var 43,75318
Sum squared resid 4281,287 S.E. of regression 13,95005
R-squared 0,902764 Adjusted R-squared 0,898344
F(1, 22) 204,2536 P-value(F) 1,29e-12
Log-likelihood -96,26199 Akaike criterion 196,5240
Schwarz criterion 198,8801 Hannan-Quinn 197,1490
rho 0,836471 Durbin-Watson 0,763565
INTERPRETATION OF A REGRESSION EQUATION
This is the output from a regression of output q, using gretl.

30
80
100
120
140
160
180
200
220
240
260
100 120 140 160 180 200
q
l
q versus l (with least squares fit)
Y = -38,7 + 1,40X
Here is the scatter diagram again, with the regression line
shown

31
THE COEFFICIENT OF
DETERMINATION
Question: how does the sample regression line fit
the data at hand?
How much does the independent var. explain the
variation in the dependent var. (in the sample)?
We have:



2
2
2

( )
( )
i
i
i
i
Y Y
R
Y Y

2 2 2

( ) ( )
i i i
i i i
Y Y Y Y e = +

total variation
of Y
The share
explained
by model
GOODNESS OF FIT
RSS ESS TSS + =

= =
2
2
2
) (
)

(
Y Y
Y Y
TSS
ESS
R
i
i
( ) ( )

+ =
2
2
2

i i i
e Y Y Y Y
The main criterion of goodness of fit, formally described as the coefficient of
determination, but usually referred to as R
2
, is defined to be the ratio of ESS
to TSS, that is, the proportion of the variance of Y explained by the
regression equation.
GOODNESS OF FIT

=
2
2
2
) (
1
Y Y
e
TSS
RSS TSS
R
i
i

= =
2
2
2
) (
)

(
Y Y
Y Y
TSS
ESS
R
i
i
The OLS regression coefficients are chosen in such a way as to minimize the
sum of the squares of the residuals. Thus it automatically follows that they
maximize R
2
.
( ) ( )

+ =
2
2
2

i i i
e Y Y Y Y RSS ESS TSS + =
34
Model 1: OLS, using observations 1899-1922 (T = 24)
Dependent variable: q

coefficient std. error t-ratio p-value
---------------------------------------------------------
const -38,7267 14,5994 -2,653 0,0145 **
l 1,40367 0,0982155 14,29 1,29e-012 ***

Mean dependent var 165,9167 S.D. dependent var 43,75318
Sum squared resid 4281,287 S.E. of regression 13,95005
R-squared 0,902764 Adjusted R-squared 0,898344
F(1, 22) 204,2536 P-value(F) 1,29e-12
Log-likelihood -96,26199 Akaike criterion 196,5240
Schwarz criterion 198,8801 Hannan-Quinn 197,1490
rho 0,836471 Durbin-Watson 0,763565
INTERPRETATION OF A REGRESSION EQUATION
This is the output from a regression of output q, using gretl.

BASIC (Gauss-Makov) ASSUMPTION OF THE
OLS
35
1. zero systematic error: E(u
i
) =0
2. Homoscedasticity: var(u
i
) =
2
for all i
3. No autocorrelation: cov(u
i
; u
j
) = 0 for all i #j
4. X is non-stochastic
5. u~ N(0,
2
)
BASIC ASSUMPTION OF THE OLS
Homoscedasticity: var(u
i
) =
2
for all i
36
f(u)
X
1
X
2
X
i X
Y
i i
X Y PRF
2 1
: | | + =
M

c

s
u

t

c

a

u
i
BASIC ASSUMPTION OF THE OLS
Heteroscedasticity: var(u
i
) =
i
2

37
f(u)
X
1
X
2
X
i X
Y
i i
X Y PRF
2 1
: | | + =
M

c

s
u

t

c

a

u
i
BASIC ASSUMPTION OF THE OLS
No autocorrelation: cov(u
i
; u
j
) = 0 for all i #j
38
(a) (b)
(c )
i
u +
i
u +
i
u +
i
u +
i
u +
i
u +
i
u
i
u
i
u
i
u
i
u
i
u

Simple regression model: Y = |
1
+ |
2
X + u
We saw in a previous slideshow that the slope coefficient may be decomposed into the
true value and a weighted sum of the values of the disturbance term.
UNBIASEDNESS OF THE REGRESSION COEFFICIENTS
( )( )
( )

+ =

=
i i
i
i i
u a
X X
Y Y X X
b
2
2
2
|
( )

=
2
X X
X X
a
j
i
i

Simple regression model: Y = |
1
+ |
2
X + u
|
2
is fixed so it is unaffected by taking expectations. The first expectation rule states
that the expectation of a sum of several quantities is equal to the sum of their
expectations.
UNBIASEDNESS OF THE REGRESSION COEFFICIENTS
( )( )
( )

+ =

=
i i
i
i i
u a
X X
Y Y X X
b
2
2
2
|
( )

=
2
X X
X X
a
j
i
i
( ) ( ) ( )
( ) ( )
2
2 2
2 2
|
| |
|
=
+ = + =
+ =


i i i i
i i
u E a u a E
u a E E b E
( ) ( ) ( ) ( ) ( )

= + + = + + =
i i n n n n i i
u a E u a E u a E u a u a E u a E ... ...
1 1 1 1

Simple regression model: Y = |
1
+ |
2
X + u
Now for each i, E(a
i
u
i
) = a
i
E(u
i
)

UNBIASEDNESS OF THE REGRESSION COEFFICIENTS
( )( )
( )

+ =

=
i i
i
i i
u a
X X
Y Y X X
b
2
2
2
|
( )

=
2
X X
X X
a
j
i
i
( ) ( ) ( )
( ) ( )
2
2 2
2 2
|
| |
|
=
+ = + =
+ =


i i i i
i i
u E a u a E
u a E E b E

Simple regression model: Y = |
1
+ |
2
X + u

Efficiency

PRECISION OF THE REGRESSION COEFFICIENTS
The GaussMarkov theorem states that, provided that the regression model
assumptions are valid, the OLS estimators are BLUE: Linear, Unbiased, Minimum
variance in the class of all unbiased estimators

probability
density
function of b
2

OLS
other unbiased
estimator
|
2
b
2


Simple regression model: Y = |
1
+ |
2
X + u

PRECISION OF THE REGRESSION COEFFICIENTS
In this sequence we will see that we can also obtain estimates of the
standard deviations of the distributions. These will give some idea of their
likely reliability and will provide a basis for tests of hypotheses.
probability
density
function of b
2

|
2

standard deviation
of density function
of b
2

b
2


Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
Expressions (which will not be derived) for the variances of their
distributions are shown above.
We will focus on the implications of the expression for the variance of b
2
.
Looking at the numerator, we see that the variance of b
2
is proportional to
o
u
2
. This is as we would expect. The more noise there is in the model, the
less precise will be our estimates.

( )

)

+ =

2
2
2 2
1
1
X X
X
n
i
u b
o o
( )
) ( MSD
2
2
2
2
2
X n
X X
u
i
u
b
o o
o =

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
However the size of the sum of the squared deviations depends on two
factors: the number of observations, and the size of the deviations of X
i

around its sample mean. To discriminate between them, it is convenient to
define the mean square deviation of X, MSD(X).
( )

)

+ =

2
2
2 2
1
1
X X
X
n
i
u b
o o
( )
) ( MSD
2
2
2
2
2
X n
X X
u
i
u
b
o o
o =

( )

=
2
1
) ( MSD X X
n
X
i

Simple regression model: Y = |
1
+ |
2
X + u

PRECISION OF THE REGRESSION COEFFICIENTS
This is illustrated by the diagrams above. The nonstochastic component of
the relationship, Y = 3.0 + 0.8X, represented by the dotted line, is the same
in both diagrams.
However, in the right-hand diagram the random numbers have been
multiplied by a factor of 5. As a consequence, the regression line, the solid
line, is a much poorer approximation to the nonstochastic relationship.

-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
Y Y
X X
Y = 3.0 + 0.8X

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
Looking at the denominator, the larger is the sum of the squared deviations
of X, the smaller is the variance of b
2
.
( )

)

+ =

2
2
2 2
1
1
X X
X
n
i
u b
o o
( )
) ( MSD
2
2
2
2
2
X n
X X
u
i
u
b
o o
o =

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
11
( )

)

+ =

2
2
2 2
1
1
X X
X
n
i
u b
o o
( )
) ( MSD
2
2
2
2
2
X n
X X
u
i
u
b
o o
o =

( )

=
2
1
) ( MSD X X
n
X
i
A third implication of the expression is that the variance is inversely
proportional to the mean square deviation of X.

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
In the diagrams above, the nonstochastic component of the relationship is
the same and the same random numbers have been used for the 20 values of
the disturbance term.
-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
Y Y
X X
Y = 3.0 + 0.8X

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
However, MSD(X) is much smaller in the right-hand diagram because the
values of X are much closer together.
-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
Y Y
X X
Y = 3.0 + 0.8X

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
Hence in that diagram the position of the regression line is more sensitive to
the values of the disturbance term, and as a consequence the regression line
is likely to be relatively inaccurate.
-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
-15
-10
-5
0
5
10
15
20
25
30
35
0 5 10 15 20
Y Y
X X
Y = 3.0 + 0.8X

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
( )

)

+ =

2
2
2 2
1
1
X X
X
n
i
u b
o o
( )
) ( MSD
2
2
2
2
2
X n
X X
u
i
u
b
o o
o =

We cannot calculate the variances exactly because we do not know the

variance of the disturbance term. However, we can derive an estimator of o
u
2

from the residuals.

Simple regression model: Y = |
1
+ |
2
X + u
PRECISION OF THE REGRESSION COEFFICIENTS
( )

)

+ =

2
2
2 2
1
1
X X
X
n
i
u b
o o
( )
) ( MSD
2
2
2
2
2
X n
X X
u
i
u
b
o o
o =

( )

= =
2
2
1 1
) ( MSD
i i
e
n
e e
n
e
Clearly the scatter of the residuals around the regression line will reflect the
unseen scatter of u about the line Y
i
= |
1
+ b
2
X
i
, although in general the
residual and the value of the disturbance term in any given observation are
not equal to one another.
One measure of the scatter of the residuals is their mean square error,
MSD(e), defined as shown.
54
Model 1: OLS, using observations 1899-1922 (T = 24)
Dependent variable: q

coefficient std. error t-ratio p-value
---------------------------------------------------------
const -38,7267 14,5994 -2,653 0,0145 **
l 1,40367 0,0982155 14,29 1,29e-012 ***

Mean dependent var 165,9167 S.D. dependent var 43,75318
Sum squared resid 4281,287 S.E. of regression 13,95005
R-squared 0,902764 Adjusted R-squared 0,898344
F(1, 22) 204,2536 P-value(F) 1,29e-12
Log-likelihood -96,26199 Akaike criterion 196,5240
Schwarz criterion 198,8801 Hannan-Quinn 197,1490
rho 0,836471 Durbin-Watson 0,763565
PRECISION OF THE REGRESSION COEFFICIENTS
The standard errors of the coefficients always appear as part of the output of
a regression. The standard errors appear in a column to the right of the
coefficients.

Summing up
55
Simple Linear Regression model:
Verify dependent, independent variables, parameters,
and the error terms
Interpret estimated parameters b
1
& b
2
as they show
the relationship between X and Y.
OLS provides BLUE estimators for the parameters
under 5 Gauss-Makov ass.
What next:
Estimation of multiple regression model