CSE 474/574 Introduction to Machine Learning

Homework Set #3
Due: Monday, October 28, 2013, Submit in class
Print Name:_______________________ Student Number:___________________________
Note: For this assignment, please PRINT the hard copy of the problems with paper size of Letter (8.5 11) and
DIRECTLY write down your answers on these sheets. There are three sections, for which please fill in the blank in
required area, as well as briefly validate your answers right below the particular question if needed. After finishing
the assignment, please staple them, print your name and UB number on it, and submit it in class on the due date.
---------------------------------------------------------Reserved for TA Only---------------------------------------------------------
Section 1 Section 2 Section 3 Total


---------------------------------------------------------Reserved for TA Only---------------------------------------------------------
1. Polynomial curve fitting
You are given with a table with data showing below.
k x
k
y
k

1 1 4
2 2 4.5
3 3 6
4 4 8
5 5 8.5

(a) Please fit the points using the function below under least square error. (ie. Minimize

( )

)
(i) If we use form of linear function ( )

, then the fitting function is

() ___________________________;
(ii) If we use form of quadratic function ( )

, the fitting function is

() ___________________________.
(b) If we are trying to fit the points using a polynomial of degree M of the form ( )

For the error function

( )

, derive a closed form solution for

the parameters W defined to be [

. (Hint: represent your answer in matrix form.)

(c) Verify your answer of (a) by using the closed form solution above. (Taking M as 1 and 2)
(d) If we modify the error function by introducing a regularization form, ie.

( )

||

. Please fit the points using the function below under this error function.
(i) If we use form of linear function: ( )

, then the fitting function is

() ___________________________;
(ii) If we use form of quadratic function ( )

, the fitting function is

() ___________________________.


2. Bernoulli distribution maximum likelihood
Assume you have data set of independent and identically distributed binary vectors Y = {Y1, Y2, . . . , YN}
each of which is D-dimensional. A data set consisting of N = 8 data vectors of dimension D = 5 might
look like:

(

To keep things simple we suppose all data are generated from Bernoulli with some unknown (but
identical) parameter q, which is the probability of any dimensions bit flipping on 1 and therefore 1 q
is the probability that it remains off 0.
(a) If for the n-th vector, there appears a number of yn ons, the probability is P(Yn| q)=______________.
(b) Suppose there are yi times on for the i-th vector in data set Y, then the probability of this data set is
P(Y| q)=______________________.
(c) Now if q=0.6, then the probability of the data set given above is P(Y| q)=______________________.
(d) Write a piece of MATLAB code to calculate the probability of Y below:
(1) P(Y| q=0.1)=______________________.
(2) P(Y| q=0.2)=______________________.
(3) P(Y| q=0.3)=______________________.
(4) P(Y| q=0.4)=______________________.
(5) P(Y| q=0.5)=______________________.
(6) P(Y| q=0.6)=______________________.
(7) P(Y| q=0.7)=______________________.
(8) P(Y| q=0.8)=______________________.
(9) P(Y| q=0.9)=______________________.
(e) Derive in full the closed form expression for the maximum likelihood estimate q
ML
in term of the data
points, N and D.



3. Gaussian distribution maximum likelihood
You are given 20 points X={x
k
} (k=1,2,....,20). They are obtained identically and independently
distributed according to X~N(, ).
k x
k
k x
k

1 3 11 3
2 2 12 2
3 3 13 4
4 4 14 2
5 2 15 3
6 3 16 2
7 3 17 3
8 2 18 2
9 4 19 5
10 2 20 1

(a) Express the joint probability of the entire data with and : P(X| ,)=_______________________.
(b) Derive in full the closed form expression for the maximum likelihood estimate
ML
and
ML
, in terms
of xk and N.
(c) Calculate the maximum likelihood estimate x
ML
and x
ML
for the data points given. x
ML=-
_______________,
x
ML =_______________________.













4. Classification (Bayes)
(For this problem, you can use MATLAB)
Besides the data you are given in problem 3, you are also given the additional 20 points Y={y
k
}
(k=1,2,....,20) below:
k y
k
k y
k

1 5 11 5
2 4 12 4
3 5 13 6
4 6 14 4
5 4 15 5
6 5 16 4
7 5 17 5
8 4 18 4
9 6 19 7
10 4 20 3

(a) As 3(c), please calculate the maximum likelihood estimate y
ML
and y
ML
for the data points given.
y
ML=_______________,
y
ML =_______________________.
(b) Use the table of xk and yk as training data to train a Bayes classifier which can classify the two classes
X and Y. Then please classify the testing data points zi (i=1, 2,, 10) below, ie
(|)
(|)

(|)()
(|)()

(Hint: the conditional probability P(datapoint|Class) should be the probability distribution you have
estimated, and the prior probability P(Class) is the probability of taking one datapoint from the whole
training data set.)
zi

1 2 2.5 3 3.5 4 4.5 5 6 7
Classifier
Output


Class
(X/Y)