machine learning regresion intro

7/24/2019 Machine Learning Regresion Intro

1/30

Regression

A machine learning perspective

Emily Fox & Carlos GuestrinMachine Learning Specialization

University of Washington

1 23456 78%.9 :1; < ="*.1> ?@'>A*%&


2/30

9

Part of a specialization

23456 78%.9 :1; < ="*.1> ?@'>A*%&


3/30

B

This course is a part of theMachine Learning Specialization

23456 78%.9 :1; < ="*.1> ?@'>A*%&

1. Foundations

2. Regression 3. Classification4. Clustering

& Retrieval

5. Recommender

Systems

6. Capstone


4/30

What is the course about?

23456 78%.9 :1; < ="*.1> ?@'>A*%&


5/30

6

What is regression?

From features to predictions

23456 78%.9 :1; < ="*.1> ?@'>A*%&

DataML

MethodIntelligence

Regression

Input x:features derived

from dataPredict y:

continuous output or

response to input

Learn x!yrelationship


6/30

C

Salary after ML specialization

How much will your salary be? (y= $$)

Depends on x=performance in courses, quality ofcapstoneproject, # of forum responses,

23456 78%.9 :1; < ="*.1> ?@'>A*%&

hard work


7/30

D

Stock prediction

23456 78%.9 :1; < ="*.1> ?@'>A*%&

Predict the price of a stock (y)

Depends on x=

-Recent history of stock price

-News events

-Related commodities


8/30

E

Tweet popularity

How many people will retweet your tweet? (y)

Depends on x=# followers,# of followers of followers,

features of text tweeted,popularity of hashtag,

# of past retweets,

23456 78%.9 :1; < ="*.1> ?@'>A*%&


9/30F

Reading your mind

23456 78%.9 :1; < ="*.1> ?@'>A*%&

very sad very happy

Inputs xarebrain regionintensities

Output y


10/3054

Case Study:Predicting house prices

23456 78%.9 :1; < ="*.1> ?@'>A*%&

DataML

MethodIntelligence

$$

$

$ = ??

house size

price

($)

+ houseattributes (x)

Regression

(y)


11/3055

Impact of regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&


12/30

Course outline

23456 78%.9 :1; < ="*.1> ?@'>A*%&


13/305B

Module 1: Simple Regression

What makes it simple?1 inputand just fit a line to data

23456 78%.9 :1; < ="*.1> ?@'>A*%&

house size

price

($)

x

y


14/305G

Module 1: Simple Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

house size

price(

$)

Gradient descent algorithm

Define goodness-of-fitmetric for each possible line

Get estimatedparameters- interpret-use to form

predictionsintercept

slo

pe

betterfit


15/3056

Module 2: Multiple Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

Incorporatemore inputs

- Square feet

- # bathrooms

- # bedrooms

- Lot size

- Year built

-house size

price(

$)

y

x[2]

x[1]

house size

price(

$)

y

x

Fit more complex

relationships thanjust a line


16/305C

Module 3: Assessing Performance

23456 78%.9 :1; < ="*.1> ?@'>A*%&

house size

price(

$)

y

x house size

price(

$)

y

xOVERFIT

Measures of error:- Training

- Test

- True (generalization)


17/305D

Module 3: Assessing Performance

23456 78%.9 :1; < ="*.1> ?@'>A*%&

house size

price(

$)

y

x house size

price(

$)

y

xOVERFIT

Bias-variancetradeoff


18/305E

Module 4: Ridge Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

Ridge total cost =measure of fit+ measure of

model complexity

bias-variance tradeoff

house size

price(

$)

y

x house size

price(

$)

y

xOVERFIT


19/305F

Module 4: Ridge Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

How to choose balance?

(i.e., model complexity)

measure of fit+ measure ofmodel complexity

Validset

!

(2)

error2(!

)

Cross validation


20/3034

Module 5: Feature Selection& Lasso Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

$?

Useful for efficiencyof predictions andinterpretability

)1A >%/'

,%&+.' :"8%.9

H'"* I@%.A

)">A >1.J -*%#'

)">A >".' -*%#'K>LM

:%&%>$'J >LM

N&O&%>$'J >LM

:%&%>$'J I">'8'&A >LM

P Q11*>

:.11*%&+ A9-'>R"*S%&+ A9-'

R"*S%&+ "81@&A

=11.%&+

T'"0&+

7;A'*%1* 8"A'*%".>

U11V A9-'

,A*@#A@*' >A9.'

W%>$X">$'*

?"*I"+' J%>-1>".

!%#*1X"Y'

U"&+' K ZY'&

U'V*%+'*"A1*

[">$'*

W*9'*

)"@&J*9 .1#"01&

T'"0&+ A9-'

\']'J ^@IW'#S

:''J H"*J

)"X&

?"*J'&

,-*%&S.'* ,9>A'8

)1A >%/'

,%&+.' :"8%.9

H'"* I@%.A

)">A >1.J -*%#'

)">A >".' -*%#'K>LM

:%&%>$'J >LM

N&O&%>$'J >LM

:%&%>$'J I">'8'&A >LM

P Q11*>

:.11*%&+ A9-'>R"*S%&+ A9-'

R"*S%&+ "81@&A

=11.%&+

T'"0&+

7;A'*%1* 8"A'*%".>

U11V A9-'

,A*@#A@*' >A9.'

W%>$X">$'*

?"*I"+' J%>-1>".

!%#*1X"Y'

U"&+' K ZY'&

U'V*%+'*"A1*

[">$'*

W*9'*

)"@&J*9 .1#"01&

T'"0&+ A9-'

\']'J ^@IW'#S

:''J H"*J

)"X&

?"*J'&

,-*%&S.'* ,9>A'8


21/3035

Module 5: Feature Selection& Lasso Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

knocks out certain features

sparsity

Lasso total cost =measure of fit+ (different) measure of

model complexity

Coordinate descent algorithm


22/3033

Module 6: Nearest Neighbor& Kernel Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

Here, this is theclosest datapoint

y

house size

price

($)

x$ = ???


23/303B

Module 6: Nearest Neighbor& Kernel Regression

23456 78%.9 :1; < ="*.1> ?@'>A*%&

! !"# !"$ !"% !"& '! !"( !") !"* !"+ #

#

!",

!

!",

#

#",

f(x0)

-./012304567 91:01; ?/ @ !"$A

$ = ???


24/30

3G

Summary of whats covered

Linear regression

Regularization: Ridge (L2), Lasso (L1) Nearest neighbor and kernel regression

Models

Gradient descent Coordinate descentAlgorithms

Loss functions, bias-variance tradeoff,cross-validation, sparsity, overfitting,model selection, feature selection

Concepts

23456 78%.9 :1; < ="*.1> ?@'>A*%&


25/30

Assumed background

25 23456 78%.9 :1; < ="*.1> ?@'>A*%&


26/30

3C

Math background

Basic calculus

-Concept of derivatives

Basic linear algebra

-Vectors

-Matrices

-Matrix multiply

23456 78%.9 :1; < ="*.1> ?@'>A*%&


27/30

3D

Programming experience

Basic Python used

-Can pick up along the way ifknowledge of other language

23456 78%.9 :1; < ="*.1> ?@'>A*%&


28/30

3E

Reliance on GraphLab Create

SFrames will be used, though not required

-open source project of Dato(creators of GraphLab Create)

- can use pandas and numpy instead

Assignments will:1. Use GraphLab Create to

explore high-level concepts2.

Ask you to implementallalgorithms without GraphLab Create

Net result:- learn how to code methods in Python

23456 78%.9 :1; < ="*.1> ?@'>A*%&


29/30

3F

Computing needs

23456 78%.9 :1; < ="*.1> ?@'>A*%&

Basic 64-bit desktop or laptop

Access to internet

Ability to:

-Install and run Python (and GraphLab Create)

-Store a few GB of data


30/30

Lets get started!

machine learning regresion intro

Documents