CSE 474/574 Introduction to Machine Learning

Homework Set #3

Due: Monday, October 28, 2013, Submit in class

Print Name:_______________________ Student Number:___________________________

Note: For this assignment, please PRINT the hard copy of the problems with paper size of Letter (8.5 11) and DIRECTLY write down your answers on these sheets. There are three sections, for which please fill in the blank in

required area, as well as briefly validate your answers right below the particular question if needed. After finishing

the assignment, please staple them, print your name and UB number on it, and submit it in class on the due date.

---------------------------------------------------------Reserved for TA Only---------------------------------------------------------

Section 1 Section 2 Section 3 Total

---------------------------------------------------------Reserved for TA Only---------------------------------------------------------

1. Polynomial curve fitting

You are given with a table with data showing below.

k xk yk

1 1 4

2 2 4.5

3 3 6

4 4 8

5 5 8.5

(a) Please fit the points using the function below under least square error. (ie. Minimize

( ( ) )

)

(i) If we use form of linear function ( ) , then the fitting function is

( ) ___________________________;

(ii) If we use form of quadratic function ( )

, the fitting function is

( ) ___________________________.

(b) If we are trying to fit the points using a polynomial of degree M of the form ( )

For the error function

( ( ) )

, derive a closed form solution for

the parameters W defined to be [ ] . (Hint: represent your answer in matrix form.)

(c) Verify your answer of (a) by using the closed form solution above. (Taking M as 1 and 2)

(d) If we modify the error function by introducing a regularization form, ie.

( ( )

)

| | . Please fit the points using the function below under this error function.

(i) If we use form of linear function: ( ) , then the fitting function is

( ) ___________________________;

(ii) If we use form of quadratic function ( )

, the fitting function is

( ) ___________________________.

2. Bernoulli distribution maximum likelihood

Assume you have data set of independent and identically distributed binary vectors Y = {Y1, Y2, . . . , YN}

each of which is D-dimensional. A data set consisting of N = 8 data vectors of dimension D = 5 might

look like:

(

)

To keep things simple we suppose all data are generated from Bernoulli with some unknown (but

identical) parameter q, which is the probability of any dimensions bit flipping on 1 and therefore 1 q is the probability that it remains off 0.

(a) If for the n-th vector, there appears a number of yn ons, the probability is P(Yn| q)=______________.

(b) Suppose there are yi times on for the i-th vector in data set Y, then the probability of this data set is P(Y| q)=______________________.

(c) Now if q=0.6, then the probability of the data set given above is P(Y| q)=______________________.

(d) Write a piece of MATLAB code to calculate the probability of Y below:

(1) P(Y| q=0.1)=______________________.

(2) P(Y| q=0.2)=______________________.

(3) P(Y| q=0.3)=______________________.

(4) P(Y| q=0.4)=______________________.

(5) P(Y| q=0.5)=______________________.

(6) P(Y| q=0.6)=______________________.

(7) P(Y| q=0.7)=______________________.

(8) P(Y| q=0.8)=______________________.

(9) P(Y| q=0.9)=______________________.

(e) Derive in full the closed form expression for the maximum likelihood estimate qML in term of the data

points, N and D.

3. Gaussian distribution maximum likelihood

You are given 20 points X={xk} (k=1,2,....,20). They are obtained identically and independently

distributed according to X~N(, ).

k xk k xk

1 3 11 3

2 2 12 2

3 3 13 4

4 4 14 2

5 2 15 3

6 3 16 2

7 3 17 3

8 2 18 2

9 4 19 5

10 2 20 1

(a) Express the joint probability of the entire data with and : P(X| ,)=_______________________.

(b) Derive in full the closed form expression for the maximum likelihood estimate ML and ML, in terms of xk and N.

(c) Calculate the maximum likelihood estimate xML and xML for the data points given. xML=-

_______________, xML =_______________________.

4. Classification (Bayes)

(For this problem, you can use MATLAB)

Besides the data you are given in problem 3, you are also given the additional 20 points Y={yk}

(k=1,2,....,20) below:

k yk k yk

1 5 11 5

2 4 12 4

3 5 13 6

4 6 14 4

5 4 15 5

6 5 16 4

7 5 17 5

8 4 18 4

9 6 19 7

10 4 20 3

(a) As 3(c), please calculate the maximum likelihood estimate yML and yML for the data points given.

yML=_______________, yML =_______________________.

(b) Use the table of xk and yk as training data to train a Bayes classifier which can classify the two classes

X and Y. Then please classify the testing data points zi (i=1, 2,, 10) below, ie

( | )

( | ) ( | ) ( )

( | ) ( )

(Hint: the conditional probability P(datapoint|Class) should be the probability distribution you have

estimated, and the prior probability P(Class) is the probability of taking one datapoint from the whole

training data set.)

zi

1 2 2.5 3 3.5 4 4.5 5 6 7

Classifier

Output

Class

(X/Y)