Download - VSSML16 LR2. Summary Day 2
Class summary
BigML, Inc 2
Day 2 – Morning sessions
BigML, Inc 3
Basic transformations
ExpectationsPoul Petersen
Reality
$
MLready data needs work!!!Any data is always MLready
What does MLready mean?● Machine Learning algorithms consume instances of the question that you want to
model. Each row must describe one of the instances and each column a property of the instance
● Fields can be:– already present in your data– derived from your data– generated using other fields
BigML, Inc 4
Basic transformations
● Select the right model for the problem you want to solve: Classification, regression, cluster analysis, anomaly detection, association discovery
● Perform cleansing, denormalizing, aggregating, pivoting, and other data wrangling tasks to generate a collection of instances relevant to the problem at hand. Finally use a very common format as output format: CSV
● Choose the right format to store each type of feature into a field● Feature engineering: Using domain knowledge and Machine
Learning expertise, generate explicit features that help to better represent the instances (Flatline)
ML-ready steps
BigML, Inc 5
Basic transformations
Cleansing: Homogenize missing values and different types in the same feature, fix input errors, correct semantic issues, etc.
Denormalizing: Data is usually normalized in relational databases, MLReady datasets need the information denormalized in a single file/dataset.
Aggregation: When data is stored as individual transactions, as in log files, an aggregation to get the entity might be needed
Pivoting: Different values of a feature are pivoted to new columns in the result dataset
Regular time windows: Create new features using values over different periods of time
Preprocessing data
BigML, Inc 6
Basic transformations
For numeric features: – Discretization: percentiles, within percentiles, groups– Replacement– Normalization– Exponentiation– Shocks (speed of change compared to stdev)
For text features:– Mispellings– Length– Number of subordinate sentences– Language– Levenshtein distance
Stacking
Compute a field using nonlinear combinations of other fields
Feature engineering
BigML, Inc 7
Basic transformations
● Define a clear idea of the goal.● Understand what ML tasks will achieve the goal.● Understand the data structure to perform those ML tasks.● Find out what kind of data you have and make it MLReady
– where is it, how is it stored?– what are the features?– can you access it programmatically?
● Feature Engineering: transform the data you have into the
data you actually need.● Evaluate: Try it on a small scale● Accept that you might have to start over….● But when it works, automate it!!!
Holistic approach
BigML, Inc 8
Basic transformations
Command line tools:
join, jq, awk, sed, sort, uniq
Automation:
Shell, Python, etc.
Talend
BigML: flatline, bindings, bigmler, API, whizzml
Relational Db:
MySQL
NonRelational Db:
MongoDB
Tools that help
BigML, Inc 9
Feature Engineering
Data + ML Algorithm, is that enough?
The ML Algorithm only knows about the features in the dataset. Features can be useless to the algorithm if:
● They are not correlated to the objective to be predicted● Their values change their meaning when combined with other
features
For ML Algorithms to work there must be some kind of statistical relation between some of the features and the objective. Sometimes, you must transform the available features to find such relations
Feature engineering: the process of transforming raw data into machine learning readydata
Charles Parker
BigML, Inc 10
Feature Engineering
When do you need Feature Engineering?● When the relationship between the feature and the
objective is mathematically unsatisfying● When the relationship of a function of two or more
features with the objective is far more relevant than the one of the original features
● When there is missing data● When the data is timeseries, especially when the
previous time period’s objective is known● When the data can’t be used for machine learning in the
obvious way (e.g., timestamps, text data)
BigML, Inc 11
Feature Engineering
Mathematical transformations● Statistical aggregations (group by, all and allbut)● Better categories– too many detailed categories should be avoided– ordered categories can be translated to numeric values. The model will be able to
extract more information by partinioning the ordered number range● Binning or discretization: consider whether your number is more informative in
ranges (quartiles, deciles, percentiles) even for the objective field● Linearization: nonimportant for decision trees but can be for logistic regression
(watch out for exponential distributions)
Missing data● Missing value induction (replace missings with common values: mean, median,
mode, even with a Machine Learning model)● Missing values presence can be informative, so this can be added as a new feature
BigML, Inc 12
Feature Engineering
Timeseries transformations● Better objective (percent change instead of absolute
values)● Deltas from previous reference time points● Deltas from moving average (time windows)● Recent Volatility...
Problem: Exponential explosion of possible transformations
Caveats:● The regularity in time of the points has to match your training data● You have to keep track of past points to compute your windows● Really easy to get information leakage by including your objective in a
window computation (and can be very hard to detect)!
BigML, Inc 13
Feature Engineering
Datetime features● Cannot be used “as is” in a model. It's a collection of features. BigML is able to
decompose them automatically when they are provided in the most usual formats. With Flatline, you can decompose them all.
● Datetime predicates that the computer does not know (some of them, domain dependent): Working hours? Daylight? Is rush hour?...
Text features● Bag of words: a new feature is associated to each word in the document● Tokenization: how do we select tokens? Do we want ngrams? What about
numbers?● Stemming: grouping forms of the same word in a unique term● Length● Text predicates: Dollar amounts? Dates? Salutations? Please and Thank you?
BigML, Inc 14
Feature Engineering
Machine Learning for Feature engineeringLatent Dirichlet Allocation
• Learn word distributions for topics
• Infer topic scores for each document
• Use the topic scores as features to a model (dimensional reduction)
Distance to cluster Centroids
Stacked Generalization: Classifiers provide new features
BigML, Inc 15
Day 2 – Evening sessions
BigML, Inc 16
REST API, bindings and basic workflows
jao (José Antonio Ortega)
Academics Real world
How do Machine Learning Workflows look like?
We need highlevel tools to face the real world workflows by growing in:
● Automation● Abstraction
BigML, Inc 17
REST API, bindings and basic workflows
The foundations● REST API first applications: Standards in software development.
First level of abstraction
Client side tools● Web UI: Sitting on top of the REST API. Humanfriendly access and
visualizations for all the Machine Learning resources. Workflows must be defined and executed step by step. Second level of abstraction.
● Bindings: Sitting on top of the REST API. Finegrained accessors for the REST API calls. Workflows must be defined and executed step by step. Second level of abstraction.
● BigMLer: Relying on the bindings. Highlevel syntax. Entire workflows can be created in only one command line. Third level of abstraction.
BigML, Inc 18
REST API, bindings and basic workflows
.
BigMLer automation● Basic 1click workflows in one command line● Rich parameterized workflows: feature selection, crossvalidation, etc.
● Models are downloaded to your laptop, tablet, cell phone, etc. once and can be used offline to create predictions
Still..
Great for local predictions
BigML, Inc 19
REST API, bindings and basic workflows
.
Problems of clientside solutions● Complexity Lots of details outside the problem
domain● Reuse No interlanguage compatibility● Scalability Clientside workflows hard to optimize● Extensibility BigMLer hides complexity at the cost of
flexibility● Not enough abstraction
BigML, Inc 20
REST API, bindings and basic workflows
.Solution: bringing automation and abstraction to the serverside
● DSL for ML workflow automation● Framework for scalable, remote execution of ML workflows
Sophisticated serverside optimizationOutofthebox scalability
Clientserver brittleness removedInfrastructure for creating and sharing ML scripts and libraries
WhizzML
BigML, Inc 21
REST API, bindings and basic workflows
.
WhizzML's new REST API resources:
Scripts: Executable code that describes an actual workflow, taking a list of typed inputs and producing a list of outputs.
Executions: Given a script and a complete set of inputs, the workflow can be executed and its outputs generated.
Libraries: A collection of WhizzML definitions that can be imported by other libraries or scripts.
BigML, Inc 22
REST API, bindings and basic workflows
ScriptsCreating scripts
● Usable by any binding (from any language)● Builtin parallelization● BigML resources management as primitives of the language● Complete programming language for workflow definition
Using scripts
Web UI
Bindings
BigMLer
WhizzML
BigML, Inc 23
Advanced WhizzML workflows
Charles Parker
WhizzML offers:● Primitives for all ML resources: (datasets, models, clusters, etc.)
● A complete programming language to compose at will these ML resources.
● Parallelization and Scalability builtin.
This empowers the user to benefit from:● Automated feature engineering: Bestfirst feature selection.
● Automated configuration choice: Randomized parameter optimization, SMACdown.
● Complex algorithms as 1click: Stacked generalization, Boosting.
All of them can be shared, reproduced and reused as one more BigML resource in a languageagnostic way.
BigML, Inc 24
Advanced WhizzML workflows
f5 fn
... ...
......
... ...
f5 f7 f5 fn
... ...
......
... ...
f5 f1
Selectedfields
()
(f5)
The best scoreis obtained forthe model with (f5)
The best scoreis obtained forthe model with (f5 f7)
Following iterations don't improve the score for the modelwith (f5 f7), so the process stops
Step 1
Step 2
f1Bestfirst feature selection
BigML, Inc 25
Advanced WhizzML workflows
A new dataset is generatedwith the predictions for the
hold out data
A new metamodel is createdfrom this dataset
50%
Hold out
Stacked generalization
BigML, Inc 26
Advanced WhizzML workflows
Configurationrandom
generator
... ...
Bestscore
Process stops when you reach the expected performanceor the usergiven iterations limit
+
Randomized parameter optimization
BigML, Inc 27
Advanced WhizzML workflows
Configurationrandom
generator
... ...
+ New configurations are filteredaccording to the predictionsof the model of performances
Only promisingconfigurations are analyzed
SMACdown
BigML, Inc 28
Advanced WhizzML workflows
… …
The final model is an ensemble of models
T0
F0
T1
F1
T2
F2
F8
T8
Boosting
BigML, Inc 29
Advanced WhizzML workflowsScript it once, for everybody anywhere
Publish scripts in the gallery
Add scripts toyour menus