automatedmachine learning (automl) · auto feature generation(thislecture) neural architecture...
TRANSCRIPT
![Page 1: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/1.jpg)
Automated Machine Learning (AutoML)SLIDES BY:LYDIA ZHENG and JIANNAN WANG
1
https://www.cs.sfu.ca/~jnwang/
![Page 2: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/2.jpg)
Motivation
1. Machine learning is very successful2. To build a traditional ML pipeline:
➢ Domain experts with longstanding experience➢ Specialized data preprocessing➢ Domain-driven meaningful feature engineering➢ Picking right models➢ Hyper-parameter tuning➢ … …
2
![Page 3: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/3.jpg)
AutoML Vision
3
For Non-ExpertsAutoML allows non-experts to make use of machine learning models and techniques without requiring to become an expert in this field first
For Data ScientistsAutoML aims to augment, rather than automate, the work and work practices of heterogeneous teams that work in data science.Wang, Dakuo, et al. "Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI." Proceedings of the ACM on Human-Computer Interaction 3.CSCW (2019): 1-24.
https://en.wikipedia.org/wiki/Automated_machine_learning
![Page 4: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/4.jpg)
What is AutoML?❖ Automate the process of applying machine learning to real-
world problems
4
![Page 5: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/5.jpg)
H20 Driverless AI Demohttps://www.youtube.com/watch?v=ZqCoFp3-rGc
5
![Page 6: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/6.jpg)
OutlineAuto Feature Selection (Lecture 6)Auto Hyperparameter Tuning (Lecture 6)Auto Feature Generation (This Lecture)Neural Architecture Search (This Lecture)
6
![Page 7: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/7.jpg)
Auto Feature Generation
7
![Page 8: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/8.jpg)
Motivation
❖ The model performance is heavily dependent on quality of features in dataset
❖ It’s time-consuming for domain experts to generate enough useful features
8
![Page 9: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/9.jpg)
Feature Generation❖ Unary operators (applied on a single feature)
◦ Discretize or normalize numerical features◦ Apply rule-based expansions of dates ◦ Mathematical operators (e.g., Log Function)
❖ Higher-order operators (applied on 2+ features)◦ Basic arithmetic operations (e.g., +, -, ×, ÷)◦ Group-by Aggregation (e.g., GroupByThenMax, GroupByThenMin)
9Katz, Gilad, Eui Chul Richard Shin, and Dawn Song. "Explorekit: Automatic feature generation and selection." 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016.
![Page 10: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/10.jpg)
Featuretools
❖ An open source library for performing automated feature engineering
❖ Design to fast-forward feature generation across multi-relational tables
10
![Page 11: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/11.jpg)
Concepts
❖ Entity is the relational tables ❖ An EntitySet is a collection of entities and the
relationships between them❖ Feature Primitives
❖ Unary Operator: transformation (e.g., MONTH)❖ High-order Operator: Group-by Aggregation (e.g., GroupByThenSUM)
11
![Page 12: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/12.jpg)
12
SUM(Product.Price)
$500
...
...
...
Product_id Customer_id Name Price
1 1 Banana $100
2 1 Banana $100
3 1 Orange $300
4 2 Apple $50
... ... ... ...
Customer Product
GroupByThenSUM:
Unary Operator:MONTH
Customer_id Birthdate
1 1995-09-28
2 1980-01-01
3 1999-02-02
... ...
MONTH(Birthdate)
9
1
2
...
Entity sets
FeaturePrimitives
![Page 13: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/13.jpg)
OutlineAuto Feature Selection (Lecture 5)Auto Hyperparameter Tuning (Lecture 5)Auto Feature Generation (This Lecture)Neural Architecture Search (This Lecture)
13
![Page 14: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/14.jpg)
Neural Architecture Search (NAS)
14
![Page 15: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/15.jpg)
Motivation
15
How can someone come out with such an architecture?
He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." CVPR. 2016
![Page 16: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/16.jpg)
Neural Architecture Search:Big Picture
16
![Page 17: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/17.jpg)
Search Space❖ Define which neural
architectures a NAS approach might discover in principle
❖ May have human bias →prevent finding novel architectural building blocks
17
Chain-structured Multi-branch
![Page 18: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/18.jpg)
Search Strategy
❖ Basic Idea➢ Explore search space (often exponentially large or even
unbounded)
❖ Methods➢ Random Search➢ Evolutionary Methods [Angeline et al., 1994]➢ Bayesian Optimization [Bergstra et al., 2013]➢ Reinforcement Learning [Baker et al., 2017]➢ ……
18
![Page 19: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/19.jpg)
Performance Estimation Strategy❖ Basic Idea
➢ The process of estimating predictive performance
❖ Methods➢ Simplest option: perform a training and validation of the
architecture on data➢ Initialize weights of novel architecture based on weights
of other architectures have been trained before➢ Using learning curve extrapolation [Swersky et al., 2014] ➢ …...
19
![Page 20: AutomatedMachine Learning (AutoML) · Auto Feature Generation(ThisLecture) Neural Architecture Search(ThisLecture) 6. AutoFeatureGeneration 7. Motivation](https://reader034.vdocuments.us/reader034/viewer/2022042807/5f7d898e795a727a275019cb/html5/thumbnails/20.jpg)
Summary
What is AutoML and why we need it?How AutoML works?
◦ Auto Feature Selection (Lecture 5)◦ Auto Hyperparameter Tuning (Lecture 5)◦ Auto Feature Generation (This Lecture) ◦ Neural Architecture Search (This Lecture)
20