machine learning in practice lecture 9
DESCRIPTION
Machine Learning in Practice Lecture 9. Carolyn Penstein Ros é Language Technologies Institute/ Human-Computer Interaction Institute. Plan for the Day. Announcements Questions? Assignment 4 Quiz Today’s Data Set: Speaker Identification Weka helpful hints - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/1.jpg)
Machine Learning in PracticeLecture 9
Carolyn Penstein RoséLanguage Technologies Institute/
Human-Computer Interaction Institute
![Page 2: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/2.jpg)
Plan for the Day Announcements
Questions?Assignment 4Quiz
Today’s Data Set: Speaker Identification Weka helpful hints
Visualizing Errors for Regression ProblemsAlternative forms of cross-validationCreating Train/Test Pairs
Intro to Evaluation
![Page 3: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/3.jpg)
Speaker Identification
![Page 4: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/4.jpg)
Today’s Data Set – Speaker Identification
![Page 5: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/5.jpg)
Preprocessing Speech
Record speech to WAV files.Extract a variety of acoustic and prosodic features.
![Page 6: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/6.jpg)
Predictions: which algorithm will perform better?
What previous data set does this remind you of?
J48 .53 KappaSMO .37 KappaNaïve Bayes .16 Kappa
![Page 7: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/7.jpg)
Notice Ranges and Contingencies
![Page 8: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/8.jpg)
Most Predictive Feature
![Page 9: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/9.jpg)
Least Predictive Feature
![Page 10: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/10.jpg)
What would 1R do?
![Page 11: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/11.jpg)
What would 1R do?
.16 Kappa
![Page 12: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/12.jpg)
Weka Helpful Hints
![Page 13: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/13.jpg)
Evaluating Numeric Prediction: CPU data
![Page 14: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/14.jpg)
Visualizing Classifier Errors for Numeric Prediction
![Page 15: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/15.jpg)
Creating Train/Test Pairs
First click here
![Page 16: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/16.jpg)
Creating Train/Test Pairs
If you pick unsupervised,you’ll get non-stratifiedfolds, otherwise you’llget stratified folds.
![Page 17: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/17.jpg)
Stratified versus Non-Stratified Weka’s standard cross-validation is
stratifiedData is randomized before dividing it into foldsPreserves distribution of class values across
foldsReduces variance in performance
Unstratified cross-validation means there is no randomizationOrder is preservedAdvantage for matching predictions with
instances in Weka
![Page 18: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/18.jpg)
Stratified versus Non-Stratified Leave-one-out cross validation
Train on all but one instance Iterate over all instances
Extreme version of unstratified cross-validation If test set only has one instance, the distribution of
class values cannot be preservedMaximizes amount of data used for training on
each fold
![Page 19: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/19.jpg)
Stratified versus Non-Stratified Leave-one-subpopulation out
If you have several data points from the same subpopulation
Speech data from the same speakerMay have data from same subpopulation in
train and testover-estimates overlap between train and test
When is this not a problem?You can manually make sure that won’t
happenYou have to do that by hand
![Page 20: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/20.jpg)
Creating Train/Test Pairs
If you pick unsupervised,you’ll get non-stratifiedfolds, otherwise you’llget stratified folds.
![Page 21: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/21.jpg)
Creating Train/Test Pairs
Now click here
![Page 22: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/22.jpg)
Creating Train/Test Pairs
![Page 23: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/23.jpg)
Creating Train/Test Pairs
You’re going torun this filter 20times altogether.twice for every fold.
![Page 24: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/24.jpg)
Creating Train/Test Pairs
True for Train, falsefor Test
![Page 25: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/25.jpg)
Creating Train/Test Pairs
If you’re doingStratified, make sureyou have to classattribute selectedhere.
![Page 26: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/26.jpg)
Creating Train/Test Pairs
1. Click Apply
![Page 27: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/27.jpg)
Creating Train/Test Pairs
2. Save the file
![Page 28: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/28.jpg)
Creating Train/Test Pairs
3. Undo before youcreate the next file
![Page 29: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/29.jpg)
Doing Manual Train/Test* First load thetraining data on thePreprocess tab
![Page 30: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/30.jpg)
Doing Manual Train/Test
* Now select SuppliedTest Set as the Test Option
![Page 31: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/31.jpg)
Doing Manual Train/Test
Then Click Set
![Page 32: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/32.jpg)
Doing Manual Train/Test* Next Load the Testset
![Page 33: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/33.jpg)
Doing Manual Train/Test
* Then you’re allset, so click on Start
![Page 34: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/34.jpg)
Evaluation Methodology
![Page 35: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/35.jpg)
Intro to Chapter 5 Many techniques illustrated in Chapter 5
(ROC curves, recall-precision curves) don’t show up in applied papersThey are useful for showing trade-offs between
properties of different algorithmsYou see them in theoretical machine learning
papers
![Page 36: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/36.jpg)
Intro to Chapter 5 Still important to understand what they
represent The thinking behind the techniques will show
up in your papersYou need to know what your numbers do and
don’t demonstrateThey give you a unified framework for thinking
about machine learning techniquesThere is no cookie cutter for a good evaluation
![Page 37: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/37.jpg)
Confidence Intervals Mainly important if there is some question about
whether your data set is big enough You average your performance over 10 folds, but
how certain can you be that the number you got is correct?
We saw before that performance varies from fold to fold
0 10 20 30 40( )
![Page 38: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/38.jpg)
Confidence Intervals We know that the distribution of categories found
in the training set and in the testing set affects the performance
Performance on two different sets will not be the same
Confidence intervals allow us to say that the probability of the real performance value being within a certain range from the observed value is 90%
0 10 20 30 40( )
![Page 39: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/39.jpg)
Confidence Intervals Confidence limits come from the normal
distribution Computed in terms of number of standard
deviations from the mean If the data is normally distributed, there is a
15% chance of the real value being more than 1 standard deviation above the mean
![Page 40: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/40.jpg)
What is a significance test? How likely is it that the difference you see
occurred by chance? How could the difference occur by
chance?
0 10 20 30 40( ( ) )
If the mean of one distribution is within theconfidence interval of another, the difference you observe could be by chance.
If you want p<.05, you need the 90% confidence intervals. Find the correspondingZ scores from a standard normal distribution table.
![Page 41: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/41.jpg)
Computing Confidence Intervals 90% confidence interval corresponds to
z=1.655% chance that a data point will occur to the
right of the rightmost edge of the interval f = percentage of successes N = number of trials p = (f + z2/2N +or- z(squrt(f/N – f2/N + z2/4N2)))/(1
+ z2/N) f=75%, N=1000, c=90% -> [0.727,0.773]
![Page 42: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/42.jpg)
Significance Tests If you want to know whether the difference
in performance between Approach A and Approach B is significantGet performance numbers for A and B on each
fold of a 10-fold cross validationYou can use the Experimenter or you can do
the computation in Excel or Minitab If you use exactly the same “folds” across
approaches you can use a paired t-test rather than an unpaired t-test
![Page 43: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/43.jpg)
Significance TestsDon’t forget that you can get a significant result
by chance! The Experimenter corrects for multiple comparisons
Significance tests are less important if you have a large amount of data and the difference in performance between approaches is large
![Page 44: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/44.jpg)
Using the Experimenter* First click New
![Page 45: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/45.jpg)
Using the Experimenter
Make sureSimple is selected
![Page 46: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/46.jpg)
Using the Experimenter
Select .csvas the outputfile formatand click on Browse
Enter file name
Click on Add New
![Page 47: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/47.jpg)
Using the Experimenter
Load data set
![Page 48: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/48.jpg)
Using the Experimenter
10 repetitions isbetter than 1, but1 is faster.
![Page 49: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/49.jpg)
Using the Experimenter
Click on Add Newto add algorithms
![Page 50: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/50.jpg)
Using the Experimenter
Click Choose toselect algorithm
![Page 51: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/51.jpg)
Using the Experimenter
You should addNaïve Bayes, SMO, and J48
![Page 52: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/52.jpg)
Using the Experimenter
Then click onthe Run tab
![Page 53: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/53.jpg)
Using the Experimenter
Click on Start
![Page 54: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/54.jpg)
Using the Experimenter
When it’s done,Click on Analyze
![Page 55: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/55.jpg)
Using the Experimenter
Click File to loadthe results file yousaved
![Page 56: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/56.jpg)
Using the Experimenter
![Page 57: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/57.jpg)
Do Analysis
* Explicitly selectdefault settings here
* Then select Kappa Here
* Then select PerformTest
![Page 58: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/58.jpg)
Do Analysis* Base case is what you are comparing with
![Page 59: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/59.jpg)
![Page 60: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/60.jpg)
CSV Output
![Page 61: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/61.jpg)
Analyze with Minitab
![Page 62: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/62.jpg)
More Complex Statistical Analyses
I put a Minitab manual in the Readings folder on Blackboard.
![Page 63: Machine Learning in Practice Lecture 9](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816829550346895dddbc02/html5/thumbnails/63.jpg)
Take Home Message We focused on practical, methodological
aspects of the topic of Evaluation We talked about the concept of a
confidence interval and significance tests We learned how to create Train/Test pairs
for manual cross-validation, which is useful for preparing for an error analysis
We also learned how to use the Experimenter to do experiments and run significance tests