self-taught learning: transfer learning from … · 2017-04-19 · self-taught learning: transfer...
TRANSCRIPT
![Page 1: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/1.jpg)
SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM
UNLABELED DATA
Richard Wen
April 7, 2017
Article Summary for Knowledge Discovery Course Instructed by Dr. Cherie Ding
1
![Page 8: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/8.jpg)
WHY?
Labeled data is expensive
Abundance of unlabeled data
Less restrictive data requirements
8
![Page 10: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/10.jpg)
TWO STAGES
1. Learn representation with unlabeled data
2. Apply to labeled data for classification
10
![Page 11: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/11.jpg)
LABELED TRAINING DATA
A set of m training examples with:
x as n-dimension feature vectors
y as corresponding labels {1 .. C}
l indicating a labeled example
Raina et al. (2007)
11
![Page 12: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/12.jpg)
UNLABELED TRAINING DATA
A set of k training examples with:
x as n-dimension feature vectors
u indicating an unlabeled example
Raina et al. (2007)
12
![Page 13: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/13.jpg)
LEARNING BASES (BASIC ELEMENTS)
Optimize aj weights and bj bases to:
1. Reconstruct xu as weighted linear combo of bases
2. Encourage aj to be sparse (mostly zero)
Raina et al. (2007)
13
![Page 14: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/14.jpg)
LEARNING FEATURES
Using bj, compute sparse features from labeled data
as input to supervised algorithms
Raina et al. (2007)
14
![Page 17: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/17.jpg)
CLASSIFICATION EXPERIMENTS
Principal Component Analysis (PCA) vs Raw vs
Sparse Coding (SC) Features
Support Vector Machine (SVM)
Gaussian Discriminant Analysis (GDA)
17
![Page 18: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/18.jpg)
HANDWRITTEN DIGITS AND ENGLISH CHARACTERS
Improvements when SC used with Raw for
characters
SC did not perform as well alone for characters
SC generally performed better for digits
Improvements ranged from ~1-7%
18
![Page 19: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/19.jpg)
REUTERS WEBPAGES AND ARTICLES
SC generally performed well for webpages and
articles
Improvements ranged from ~8-21%
19
![Page 20: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/20.jpg)
KERNELS (SIMILARITY FUNCTIONS)
Compared against linear, polynomials, Radial Basis
Function (RBF)
Outperforms standard kernel choices above
Improvements ranged from ~6-13%
20
![Page 22: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/22.jpg)
DISCUSSION
Can be applied to different domains
Can have more basis vectors bj than n-dimensions
Unlabeled data must still have some structure
Other algorithms can be modified for self-taught
learning
22
![Page 23: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/23.jpg)
CONCLUSION
Find higher level representations of patterns
Use of inexpensive unlabeled data
Self-taught learning as a machine learning
framework
23
![Page 24: SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM … · 2017-04-19 · SELF-TAUGHT LEARNING: TRANSFER LEARNING FROM UNLABELED DATA Richard Wen rwen@ryerson.ca April 7, 2017 Article Summary](https://reader030.vdocuments.us/reader030/viewer/2022041017/5ecb1be28f6d1e13595c17e8/html5/thumbnails/24.jpg)
REFERENCES
Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007). Self-taughtlearning. Proceedings of the 24th international conference onMachine learning - ICML '07. doi:10.1145/1273496.1273592
24