Linear regression with one variable

Model representation
Machine Learning
Andrew Ng

Housing Prices (Portland, OR)

Price (in 1000s of dollars)

500 400 300 200

100
0 0 500 1000 1500 2000 (feet2) 2500 3000

Size Supervised Learning Given the right answer for each example in the data.

Regression Problem Predict real-valued output

Andrew Ng

Training set of housing prices (Portland, OR)

Size in feet2 (x) 2104 1416 1534 852

Price ($) in 1000's (y) 460 232 315 178

Notation: m = Number of training examples xs = input variable / features ys = output variable / target variable

Andrew Ng

Training Set

How do we represent h ?

Learning Algorithm

Size of house

Estimated price
Linear regression with one variable. Univariate linear regression.
Andrew Ng

Andrew Ng

Linear regression with one variable

Cost function
Machine Learning
Andrew Ng

Training Set

Size in feet2 (x) 2104 1416 1534 852

Price ($) in 1000's (y) 460 232 315 178

Hypothesis:

s:

Parameters
s ?
Andrew Ng

How to choose

2
1 0 0 1 2 3

2
1 0 0 1 2 3

2
1 0 0 1 2 3

Andrew Ng

Idea: Choose so that is close to for our training examples

Andrew Ng

Andrew Ng

Linear regression with one variable

Cost function intuition I

Machine Learning
Andrew Ng

Hypothesis:

Simplified

Parameters:

Cost Function:

Goal:

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameter

3 2

3
2 1 0 -0.5 0 0.5 1 1.5 2 2.5

y
1 0 0 1

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameter

3 2

3
2 1 0 -0.5 0 0.5 1 1.5 2 2.5

y
1 0 0 1

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameter

3 2

3
2 1 0 -0.5 0 0.5 1 1.5 2 2.5

y
1 0 0 1

Andrew Ng

Andrew Ng

Linear regression with one variable

Cost function intuition II

Machine Learning
Andrew Ng

Hypothesis: Parameters:
Cost Function:

Goal:
Andrew Ng

(for fixed
500 400

, this is a function of x)

(function of the parameters

Price ($) 300 in 1000s

200 100

0
0 1000

Size in

feet2

2000

3000

(x)

Andrew Ng

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

Andrew Ng

Linear regression with one variable

Machine Learning

Gradient descent
Andrew Ng

Have some function Want Outline:

Start with some

Keep changing to reduce

until we hopefully end up at a minimum

Andrew Ng

J(0,1)

Andrew Ng

J(0,1)

Andrew Ng

Gradient descent algorithm

Correct: Simultaneous update

Incorrect:

Andrew Ng

Andrew Ng

Linear regression with one variable

Gradient descent intuition

Machine Learning
Andrew Ng

Gradient descent algorithm

Andrew Ng

Andrew Ng

If is too small, gradient descent can be slow.

If is too large, gradient descent can overshoot the minimum. It may fail to converge, or even diverge.

Andrew Ng

at local optima Current value of

Andrew Ng

Gradient descent can converge to a local minimum, even with the learning rate fixed.

As we approach a local minimum, gradient descent will automatically take smaller steps. So, no need to decrease over time.
Andrew Ng

Andrew Ng

Linear regression with one variable Gradient descent for linear regression
Machine Learning
Andrew Ng

Gradient descent algorithm

Linear Regression Model

Andrew Ng

Andrew Ng

Gradient descent algorithm

update and simultaneously

Andrew Ng

J(0,1)

Andrew Ng

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

(for fixed

, this is a function of x)

(function of the parameters

Andrew Ng

Batch Gradient Descent

Batch: Each step of gradient descent uses all the training examples.

Andrew Ng

Andrew Ng