Sunteți pe pagina 1din 29

Econ 414

Lecture 2: Ordinary Least Squares and the Simple Linear Regression Model

Key observation:

If we plot n observations

Y X

,

i

i

,

i

=1,,

n

on a dependent variable Y and an independent variable X in a

scatterplot, then points (

X Y

i

,

i

) not on any straight line

Y = α + β X

Reason: Omitted variables that affect Y (beside X )

Econ 414

Contribution of omitted variables to Y is the residual u

u = Y α β X

See figure for illustration

Econ 414

HOUSEPRICE vs. SQU AREFT

600 500 400 Y u Y X 300 Y =α + β X 200 X
600
500
400
Y
u
Y
X
300
Y =α + β X
200
X
100
1000
1500
2000
2500
3000
3500
HOUSEPRICE

SQUAREFT

Econ 414

How do we find the hidden straight line if there are omitted variables?

Ordinary Least Squares (OLS) method

Compute residuals for i = 1,, n

u

i

= Y α β X

i

i

Compute the sum of squared residuals

S

(α,β)

=

n

1

i =

u

2

i

=

n

1

=

i

(

Y

i

α

β

X

i

)

2

Econ 414

The sum of squared residuals depends on α, β (as is obvious from figure)

the coefficients

Minimize S (α, β ) with respect to α, β

The last step is not obvious and will be justified later.

Econ 414

Math reminder

Summation

n

 

i

=

1

Hence

 

n

(1)

i

=

1

 

n

(2)

i =

1

x

c

i

= x + x

1

=

c

+

c

+

2

+

+

+ x

n

c

=

nc

cx

i

=

cx

1

+

cx

2

+

+

Squares

(3)

( a

+

b c )

+

2

=

a

2

+

b

2

+

cx

c

2

n

=

c

n

x

i

 

i =

1

+

2 ab

+

2bc

+

2 ac

Econ 414

OLS solution

S

(α, β )

=

n

i = 1

(

Y

i

α

β

X

i

)

2

=

by (3)

n

i = 1

by (1) and (2)

(

Y

i

2

α

2

=

+

+

2

β X

2

i

=

n

1

i =

Y

i

2

+

n

α

2

+

β

2

n

=

i

1

X

2

αY

i

2

βX Y

i

i

+

αβX

2

i

2

i

2α

n

=

i

1

Y

i

n

=

i

1

2β

X Y

i

i

)

=

+

n

1

i =

2αβ

X

i

Econ 414

Consider first case α = 0

S

(β)

=

n

1

i =

Y

i

2

+

β

2

n

=

1

i

X

2

i

n

1

i =

2β

X Y

i

i

This is expression in β of form

A + Bβ

2

+ C β

with A, B , C constants.

The graph of such a function (if minimum (see figure)

B > 0

) is a parabola with a

Econ 414

Econ 414 A + B β 2 + C β − 9 9
Econ 414 A + B β 2 + C β − 9 9
Econ 414 A + B β 2 + C β − 9 9

A + B β

2 + C β

Econ 414 A + B β 2 + C β − 9 9
Econ 414 A + B β 2 + C β − 9 9

Econ 414 A + B β 2 + C β − 9 9

Econ 414

How do we find minimum?

At β that minimizes S (β ) the slope of S (β ) is 0.

Slope of S (β ) in β is derivative of S (β )

d S (

β

)

d β

Econ 414

Math reminder

Using this

d S (

β

)

d β

d A

d β

= 0

d B

β

2

d

β

=

B

=

2 B

β +

C

d

β

2

d

β

d C

β

d

β

=

=

2

B

β

C

d

β

d

β

= C

Value of β that minimizes S (β ) solution to

2 Bβ + C = 0

or

β = −

C

2 B

Econ 414

Now

 

B

and

 

ˆ

β

=

n

i = 1

X

2

i

n

i = 1

X Y

i

i

= n

i = 1

X

2

i

C

= − 2

n

i = 1

X Y

i

i

Econ 414

Next, case with α, β

S (α, β )

=

n

1

i =

Y

i

2

+

n

α

2

+

2

β

n

=

1

i

X

2

i

2α

n

1

i =

Y

i

n

1

i =

2β

X Y

i

i

+

n

1

i =

2αβ

X

i

This a quadratic function in α, β . Graph has same shape as parabola, but in 3D.

In minimum slope is again 0.

Econ 414

Now we have slope in α -direction and slope in β -direction. These are the partial derivatives with respect to α and β :

S (

α β

,

)

α

and

S (

α β

,

)

β

Computed like ordinary derivatives, except that other argument is kept constant: for slope/derivative in α -direction we keep β constant and for slope/derivative in β -direction we keep α constant

With

S (α, β )

=

n

1

i =

Y

i

2

we have

+

n

α

2

+

2

β

n

=

1

i

X

2

i

2α

n

1

i =

Y

i

n

i =

1

2β

X Y

i

i

+

n

1

i =

2αβ

X

i

Econ 414

S (

α β

,

)

=

α

2

n

S (

α β

,

)

= −

β

2

α

2

n

1

i =

n

1

i =

X Y

i

i

Y

i

+

n

1

i =

+ 2

β

X

i

+

2

i

n

=

i

1

2

β

X

2

α

n

1

i =

X

i

S (α, β ) is minimal at the value of (α, β ) where the slope in both directions is 0.

Hence

S (

α β

,

)

 

n

=

2

n

α

2

 

i =

1

 

n

2

i =

1

X Y

i

i

α

S (

α β

,

)

= −

β

Y

i

+

+

2

n

i = 1

β

X

i

2

n

=

i

1

β

X

2

i

+

=

0

2

α

n

1

i =

X

i

=

0

Econ 414

These are two (linear) equations in two unknowns α β . These are

called the normal equations.

ˆ

ˆ,

Solution:

1. Solve first equation for α

αˆ

=

1

n

1

n

 

ˆ

 

Y

β

X

=

Y

 

i

 

i

n

i =

1

n

i

=

1

ˆ

β

X

with Y , X the sample average of Y , X

Econ 414

2. Substitute this solution in the second equation

n

1

i =

X Y

i

i

and

+

n

i

= 1

ˆ

β

ˆ

β

n

i = 1

X

2

i

 

1

n

1

 

n

X

2

   

Y

ˆ

X

1

i +

⎛ ⎜ ⎜

n

n

X

i

=

2

1

i

=

β

n

i

=

1

n

X Y

i

1

n

i

=

1

 

i

i

=

1

i

i

 

n

n

i = 1

n

1

i =

X

X

i

i

=

n

1

i =

0

Y

i

Econ 414

Now

n n n 2 ∑ ( X − X ) 2 2 2 2 =
n
n
n
2
(
X
X
)
2
2
2
2
=
(
X
2
X X
X
)
=
X
2
nX
n X
i
i
i
i
i
=
1
i
=
1
i
= 1
2
n
n
n
1 ⎛
2
2
2
=
X
n X
=
X
X
i
i
i
n
i
=
1
i
=
1
i
= 1

and

n n ∑ ( X − X )( Y − Y ) = ∑ (
n
n
(
X
X )( Y
Y
)
=
(
X Y
Y X
XY
+
X Y
)
i
i
i
i
i
i
i =
1
i =
1
n
n
n
1
=
X Y
nY X
n XY
+
n X Y
=
X Y
X
i
i
i
i
n
i =
1
i =
1
i =
1

=

i

n

i = 1

Y

i

2

=

Econ 414

ˆ

Using these results we find for β

n

(

X

i

)(

X Y

i

Y

)

 

n

   
 

(

X

i

X

)

2

ˆ i = 1

β =

i = 1

Econ 414

Hence the values of α, β that minimize the sum of squared residuals are

n ∑ ( X − X Y )( − Y ) i i β ˆ
n
(
X
X Y
)(
Y
)
i
i
β ˆ i = 1
=
n
2
(
X
X
)
i
i = 1
β X
ˆ
α ˆ Y
=

These are the Ordinary Least Squares (OLS) solutions to the problem of fitting a straight line to the points in a scatterplot. The least squares line

Y

=

ˆ

α

+

ˆ

β X

is the straight line that ‘fits’ the scatterplot best (see figure).

Econ 414

Econ 414 21 21

Econ 414

Note

If we divide numerator and denominator by n or n 1, then

ˆ

β =

Sample covariance of

X

and

Y

Sample variance of

X

From the OLS solution for αˆ

ˆ

Y

=

ˆ +

α

β X

In words: The point (Y , X ) is on the least squares line

Econ 414

The residuals with respect to the least squares line are the OLS residuals

e

i

=

Y

i

ˆ

α

ˆ

β X

i

From the normal equations

0

0

=

n n

= −

n

αˆ

+

i =

1

n

1

=

X Y

i

i

=

i

n

i = 1

X e

i

i

ˆ

β

X

Y

i

ˆ

β

i =

2

i

1

n

=

i

1

X

i

ˆ

α

=

n

1

i =

n

=

i

1

X

(

i

Y

i

=

ˆ

ˆ

α

β

X

i

n

=

i

1

(

X Y

i

i

) =

n

1

i =

ˆ

e

i

ˆ

α

β

X )

i

=

Econ 414

Hence

n

1

i

=

n

1

i =

e

i

=

X e

i

i

1

n

=

n

1

i

=

1

n

e

i

n

i =

1

= 0

X e

i

i

= 0

In words: The sample average (and sum) of the OLS residuals is 0 and the sample covariance of these residuals and X is also 0

Econ 414

These are all consequences of the fact that we minimize the sum of squared residuals. How good is the fit of the straight line to the scatterplot?

Define the fitted value

ˆ ˆ

Y

i

=

ˆ

α

+

β X

i

then by the definitions

Y

i

ˆ

= Y + e

i

i

Because the OLS residuals have average 0

ˆ Y = Y
ˆ
Y
= Y

Econ 414

ˆ

In words: Y and Y have the same sample average

Using this we have

(

Y

i

Y have the same sample average Using this we have ( Y i − Y )

Y

)

2

=

(

Y ˆ

i

Y ˆ

)

2

+

e

2

i

+

2

e

i

(

Y ˆ

i

Y ˆ )

If we take the sum over i we first observe

n

1

i

=

=

ˆ

e Y

i

(

i

n

e

i

( ˆ

α

i

=

1

n

Y ˆ

)

=

e Y ˆ

i

i

Y ˆ

i

=

1

n

i

1

=

e

i

=

+

ˆ

β

X

i

)

ˆ

α

=

n

i

1

=

e

i

+

n

i

1

=

ˆ

β

e X

i

i

= 0

Econ 414

Note that this implies that the sample covariance between the OLS

residuals and the OLS fitted values is 0: OLS decomposes

parts (residual and fitted value) that have covariance 0, i.e. are unrelated

Y i into two

Using this we find

n n 1 1 1 2 2 ∑ ( Y − Y ) = ∑
n
n
1
1
1
2
2
(
Y
Y
)
=
( Y ˆ
Y ˆ )
+
i
i
n
n
i
= 1
n i
=
1

n

i = 1

e

2

i

The sample variance of Y is equal to the sum of the sample variance of Y and the sample variance of e or

ˆ

Total Variance = Explained variance + +Unexplained variance

Econ 414

A measure of goodness of fit is

R

2

n

=

1

i

( Y ˆ

i

Y ˆ )

2

Y

= n

(

Y

i

)

2

i

=

1

This is fraction of total variance that is explained by fitted straight line.

Note

are on the straight line.

R

2

= 1

if and only if

Y

i

=

ˆ

Y

i

=

α

ˆ

+

ˆ

β X

i

, i.e. if all observations

Econ 414

Also

R

2

=

0 if and only if

or

ˆ

ˆ

Y

β

(

X

ˆ

(

β X

X

i

)

X

=

i

)

0

=

ˆ

α

+

β

X

i

=

Y ˆ

i

=

Y ˆ

=

Y

for all

case if β =

i

=1,, n

ˆ

. If the

X

i are not all equal, then this can only be the

X does not help in explaining Y .

0 . In that case

These are the extreme values for

R

2 . We have

0

R

2

1