Regression

Prof. Seungchul LeeIndustrial AI Lab.

Assumption: Linear Model

• In many cases, a linear model is used to predict 𝑦𝑦𝑖𝑖

2

Linear Regression

• �𝑦𝑦𝑖𝑖 = 𝑓𝑓(𝑥𝑥𝑖𝑖 ,𝜃𝜃) in general• In many cases, a linear model is assumed to predict 𝑦𝑦𝑖𝑖

• �𝑦𝑦𝑖𝑖 : predicted output

• 𝜃𝜃 = 𝜃𝜃0𝜃𝜃1

: model parameters

3

Linear Regression as Optimization

• How to find model parameters 𝜃𝜃 = 𝜃𝜃0𝜃𝜃1

• Optimization problem

4

Re-cast Problem as Least Squares

• For convenience, we define a function that maps inputs to feature vectors, 𝜙𝜙

5

Optimization

• Scalar Objective: 𝐽𝐽 = 𝐴𝐴𝑥𝑥 − 𝑦𝑦 2

6

Solve using Linear Algebra

• known as least square

7

Solve using Linear Algebra

8

Scikit-Learn

• Machine Learning in Python• Simple and efficient tools for data mining and data analysis• Accessible to everybody, and reusable in various contexts• Built on NumPy, SciPy, and matplotlib• Open source, commercially usable - BSD license• https://scikit-learn.org/stable/index.html#

9

Scikit-Learn

10

Scikit-Learn: Regression

11

Scikit-Learn: Regression

12

Overfitting

Industrial AI Lab.Prof. Seungchul Lee

Polynomial Regression (d = 1)

14

Polynomial Regression (d = 2)

15

Polynomial Regression (d = 3)

16

Polynomial Regression (d = 4)

17

Polynomial Regression (d = 5)

18

Polynomial Regression (d = 6)

19

Polynomial Regression (d = 7)

20

Polynomial Regression (d = 8)

21

Polynomial Regression (d = 9)

22

Overfitting Problem

• Have you come across a situation where your model performed exceptionally well on train data, but was not able to predict test data ?

• One of the most common problem data science professionals face is to avoid overfitting.

23

Issue with Rich Representation

• Low error on input data points, but high error nearby• Low error on training data, but high error on testing data

24

Regularization (Shrinkage Methods)

• Often, overfitting associated with very large estimated parameters• We want to balance

– how well function fits data– magnitude of coefficients

– multi-objective optimization– 𝜆𝜆 is a tuning parameter

25