![Page 1: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/1.jpg)
1
Machine Learning Overview
Sargur N. Srihari University at Buffalo, State University of New York
USA
![Page 2: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/2.jpg)
2
Outline 1. What is Machine Learning (ML)? 2. Types of Information Processing
Problems Solved 1. Regression 2. Classification 3. Clustering 4. Modeling Uncertainty/Inference
3. New Developments 1. Fully Bayesian Approach
4. Summary
![Page 3: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/3.jpg)
3
What is Machine Learning? • Programming computers to:
– Perform tasks that humans perform well but difficult to specify algorithmically
• Principled way of building high performance information processing systems – search engines, information retrieval – adaptive user interfaces, personalized
assistants (information systems) – scientific application (computational science) – engineering
![Page 4: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/4.jpg)
4
Example Problem: Handwritten Digit Recognition
• Handcrafted rules will result in large no of rules and exceptions
• Better to have a machine that learns from a large training set
Wide variability of same numeral
![Page 5: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/5.jpg)
5
ML History
• ML has origins in Computer Science • PR has origins in Engineering • They are different facets of the same field • Methods around for over 50 years • Revival of Bayesian methods
– due to availability of computational methods
![Page 6: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/6.jpg)
6
Some Successful Applications of Machine Learning
• Learning to recognize spoken words – Speaker-specific strategies for recognizing
primitive sounds (phonemes) and words from speech signal
– Neural networks and methods for learning HMMs for customizing to individual speakers, vocabularies and microphone characteristics
Table 1.1
![Page 7: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/7.jpg)
7
Some Successful Applications of Machine Learning
• Learning to drive an autonomous vehicle – Train computer-controlled
vehicles to steer correctly – Drive at 70 mph for 90
miles on public highways – Associate steering
commands with image sequences
![Page 8: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/8.jpg)
8
Handwriting Recognition • Task T
– recognizing and classifying handwritten words within images
• Performance measure P – percent of words correctly
classified • Training experience E
– a database of handwritten words with given classifications
![Page 9: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/9.jpg)
9
The ML Approach
Generalization
(Training)
Data Collection Samples
Model Selection Probability distribution to model process
Parameter Estimation Values/distributions
Search Find optimal solution to problem
Decision (Inference
OR Testing)
![Page 10: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/10.jpg)
Types of Problems machine learning is used for
10
1. Classification 2. Regression 3. Clustering (Data Mining) 4. Modeling/Inference
![Page 11: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/11.jpg)
Example Classification Problem • Off-shore oil transfer pipelines • Non-invasive measurement of proportion of
• oil,water and gas • Called Three-phase Oil/Water/Gas Flow
![Page 12: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/12.jpg)
Dual-energy gamma densitometry
• Beam of gamma rays passed through pipe • Attenuation in intensity provides information on
density of material • Single beam insufficient
• Two degrees of freedom: fraction of oil, fraction of water
Detector One beam of Gamma rays of two energies (frequencies or wavelengths)
![Page 13: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/13.jpg)
13
Complication due to Flow Velocity 1. Low Velocity: Stratified configuration
– Oil floats on top of water, gas above oil 2. Medium Velocity: Annular configuration
– Concentric cylinders of Water, oil, gas 3. High-Turbulence: Homogeneous
– Intimately mixed
• Single beam is insufficient – Horizontal beam thru stratified
indicates only oil
![Page 14: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/14.jpg)
Multiple dual energy gamma densitometers
14
• Six Beams • 12 measurements
• attenuation
![Page 15: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/15.jpg)
15
Prediction Problems
1. Predict Volume Fractions of oil/water/gas
2. Predict geometric configuration of three phases
• Twelve Features – Fractions of oil and water along the
paths • Learn to classify from data
![Page 16: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/16.jpg)
16
Feature Space
• Three classes (Stratified,Annular,Homogeneos)
• Two variables shown • 100 points
Which class should x belong to?
![Page 17: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/17.jpg)
17
Cell-based Classification • Naïve approach of cell
based voting will fail – exponential growth of cells
with dimensionality – 12 dimensions discretized
into 6 gives 3 million cells • Hardly any points in each
cell
![Page 18: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/18.jpg)
18
Popular Statistical Models • Generative
– Naïve Bayes – Mixtures of
multinomials – Mixtures of Gaussians – Hidden Markov Models
(HMM) – Bayesian networks – Markov random fields
• Discriminative – Logistic regression – SVMs – Traditional neural
networks – Nearest neighbor – Conditional Random
Fields (CRF)
![Page 19: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/19.jpg)
Regression Problems
19
Forward problem data set
Red curve is result of fitting a two-layer neural network by minimizing sum-of-squared error
Corresponding inverse problem by reversing x and t
Very poor fit to data: GMMs used here
![Page 20: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/20.jpg)
Forward and Inverse Problems • Kinematics of a robot arm
20
Forward problem: Find end effector position given joint angles Has a unique solution
Inverse kinematics: two solutions: Elbow-up and elbow-down
• Forward problems correspond to causality in a physical system have a unique solution e.g., symptoms caused by disease
• If forward problem is a many-to-one mapping, inverse has multiple solutions
![Page 21: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/21.jpg)
21
Clustering • Old Faithful (Hydrothermal Geyser in
Yellowstone) – 272 observations – Duration (mins, horiz axis) vs Time to next
eruption (vertical axis) – Simple Gaussian unable to capture
structure – Linear superposition of two Gaussians is
better • Gaussian has limitations in modeling real
data sets • Gaussian Mixture Models give very
complex densities
πk are mixing coefficients that sum to one
• One –dimension – Three Gaussians in blue – Sum in red
∑=
Σ=K
kkkk xNp
1),|()x( µπ
![Page 22: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/22.jpg)
22
Estimation for Gaussian Mixtures
• Log likelihood function is
• No closed form solution • Use either iterative numerical
optimization techniques or Expectation Maximization
( )∑ ∑= =
Σ=ΣN
n
K
kkknk NXp
1 1,|xln),,|(ln µπµπ
![Page 23: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/23.jpg)
23
Bayesian Representation of Uncertainty • Use of probability
to represent uncertainty is not an ad-hoc choice
• If numerical values represent degrees of belief, – then simple axioms
for manipulating degrees of belief leads to sum and product rules of probability
• Not just frequency of random, repeatable event
• It is a quantification of uncertainty
• Example: Whether Arctic ice cap will disappear by end of century – We have some idea of how
quickly polar ice is melting – Revise it on the basis of fresh
evidence (satellite observations) – Assessment will affect actions
we take (to reduce greenhouse gases)
• Handled by general Bayesian interpretation
![Page 24: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/24.jpg)
Modeling Uncertainty
24
• A Causal Bayesian Network • Example of Inference:
Cancer is independent of Age and Gender given exposure to Toxics and Smoking
![Page 25: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/25.jpg)
25
The Fully Bayesian Approach
• Bayes Rule • Bayesian Probabilities • Concept of Conjugacy • Monte Carlo Sampling
![Page 26: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/26.jpg)
26
Rules of Probability • Given random variables X and Y • Sum Rule gives Marginal Probability
• Product Rule: joint probability in terms of conditional and marginal
• Combining we get Bayes Rule
where
Viewed as Posterior α likelihood x prior
![Page 27: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/27.jpg)
27
Probability Distributions: Relationships Discrete- Binary
Discrete- Multi-valued
Continuous
Bernoulli Single binary variable
Multinomial One of K values = K-dimensional binary vector
Gaussian
Angular Von Mises
Binomial N samples of Bernoulli
Beta Continuous variable between {0,1]
Dirichlet K random variables between [0.1]
Gamma ConjugatePrior of univariate Gaussian precision
Wishart Conjugate Prior of multivariate Gaussian precision matrix
Student’s-t Generalization of Gaussian robust to Outliers Infinite mixture of Gaussians
Exponential Special case of Gamma
Uniform
N=1 Conjugate Prior
Conjugate Prior
Large N
K=2
Gaussian-Gamma Conjugate prior of univariate Gaussian Unknown mean and precision
Gaussian-Wishart Conjugate prior of multi-variate Gaussian Unknown mean and precision matrix
![Page 28: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/28.jpg)
28
Fully Bayesian Approach
• Feasible with increased computational power
• Intractable posterior distribution handled using either – variational Bayes or – stochastic sampling
• e.g., Markov Chain Monte Carlo, Gibbs
![Page 29: Machine Learning Overview - University of Alabamafeihu.eng.ua.edu/NSF_TUES/21_ML_overview.pdfquickly polar ice is melting – Revise it on the basis of fresh evidence (satellite observations)](https://reader033.vdocuments.us/reader033/viewer/2022041923/5e6cba0132c043373d3c2e57/html5/thumbnails/29.jpg)
29
Summary • ML is a systematic approach instead of ad-hockery
in designing information processing systems • Useful for solving problems of classification,
regression, clustering and modeling
• The fully Bayesian approach together with Monte Carlo sampling is leading to high performing systems
• Great practical value in many applications – Computer science
• search engines • text processing • document recognition
– Engineering