CBMM Summer School, Day 2: Machine Learning

 Lab 2: Regularized Least Squares (RLS)

This lab is about linear Regularized Least Squares. Extract the zip file in a directory and set the MATLAB path to that, follow the instructions below and think/try hard before you call the instructors!

1. Data generation

Start by generating data using the provided function MixGauss:

• 1.A Generate a 2-dimensional, 2-class training set (Xtr, Ytr), with classes centered at (-0.5,-0.5) and (0.5,0.5) and variance 0.5 for both (5 points per class). Adjust the output labels Ytr to be {1,-1}, e.g. using Ytr(Ytr==2)=-1. 

• 1.B Generate a corresponding test set of 200 points per class (Xte, Yte).
  

• 1.C Add noise to the generated data by randomly flipping a percentage of the point labels (e.g. 10%), using the provided function flipLabels. You will obtain a new set of training Ytrn and test Yten output vectors. Plot the various datasets using scatter , e.g.:
  figure; hold on
  scatter(Xtr(Ytr==1,1), Xtr(Ytr==1,2), '.r');
  scatter(Xtr(Ytr==-1,1), Xtr(Ytr==-1,2), '.b');

2. Linear RLS

• 2.A Complete the code in functions regularizedLSTrain and regularizedLSTest for training and testing a regularized Least Squares classifier.

• 2.B Try the functions on the previously generated 2-dimensional, 2-class data from 1.A-1.C. Pick a "reasonable" lambda and check the effect of regularization and the effect of noise. Plot the data in a way that visualizes the obtained results (e.g. a scatter plot with the misclassified points labeled differently,  similar to Lab1, 2.C) and evaluate the classification performance by comparing the estimated to the true outputs. Note: To visualize the separating function (and thus visualize what areas of the 2D plane are associated with each class) you can use the function separatingFRLS.  Superimpose the training and test set data (Xtr, Ytr) and (Xts, Yts), on separate plots, to analyze the generalization properties of the solution.

• 2.C Perform parameter selection using hold-out cross-validation to select lambda in the range {exp(-10), ...exp(0)}, using the function holdoutCVRLS. Plot the training and validation errors for the different values of lambda; apply the best model to the test set (Xte) and check the classification error; show the separating function and generalization of the solution. 

• 2.D Repeat the procedure data generation -> parameter selection -> test multiple times and compare the test error of RLS with that of ordinary least squares (OLS), i.e. with lambda=0. Does regularization improve classification performance?

• 2.E (Optional) Repeat the classification 2.B for the high-dimensional dataset. Generate the same classes to 1 with the Gaussians now residing in a d-dimensional space. How would you choose the class mean vectors? (Note: Obviously, you cannot visualize such a high-dimensional set). Try as indicative values d = 10, p~0.1, N~10. Check what happens with: varying lambda, varying the input space dimension d (i.e., the effect of "distance" between points), varying the effect of noise. Perform parameter selection using hold-out cross-validation for lambda in a reasonable range (using holdoutCVRLS) and find the generalization error of the best model.
  

3. (Optional) - Additional experiments

• 3.A Modify the regularizedLSTrain and regularizedLSTest functions to incorporate an off-set in the linear model (i.e., y = <w,x> + b). Compare the solution with and without offset, in a 2-class data set where classes are centered on (0,0) and (1,1) with variance 0.35 each. 



• 3.B Repeat 2.B and 2.D for different configurations (change training set size, mean vectors position/variance, percentage of noise).

• 3.C Modify the regularizedLSTrain and regularizedLSTest functions to handle multiclass problems.