01probability and bayesian inference · julian beever . probability & bayesian inference cse...
TRANSCRIPT
![Page 1: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/1.jpg)
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
COURSE INTRODUCTION
![Page 2: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/2.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
2
COMPUTATIONAL MODELING OF VISUAL PERCEPTION
The goal of this course is to provide a framework and computational tools for modeling visual inference, motivated by interesting examples from the recent literature.
Models may be realized as algorithms to solve computer vision problems, or may constitute theories of visual processing in biological systems.
The foundation of the course is a treatment of visual processing as a problem of statistical estimation and inference, grounded in the ecological statistics of the visual world.
![Page 3: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/3.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
3
Topics Bayesian decision theory Principal components and factor analysis Graphical Models
Markov Random Fields Conditional Random Fields Belief Propagation
Clustering Mean Shift Expectation Maximization Spectral Methods (Graph Cuts)
Sampling Gibbs Sampling Markov Chain Monte Carlo
Classifiers Support Vector Machines
Neural Networks
![Page 4: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/4.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
4
Course Format
Each week will consist of two 1.5 hour meetings: Meeting 1. A lecture by the instructor on a specific
computational tool or approach Meeting 2. A discussion, led by a specified student, of
a selected computational vision paper in which this approach is applied to a specific problem.
![Page 5: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/5.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
5
Evaluation
In addition to student presentations of short computational vision papers, two short MATLAB assignments will be collected and graded. The final project will involve application and possibly extension of a technique studied in the class to a problem chosen by the student. Class Participation 10% Paper Presentation 20% Assignment 1 20% Assignment 2 20% Final Project 30%
![Page 6: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/6.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
6
Main Texts
C.M. Bishop Pattern Recognition and Machine Learning. New York: Springer, 2006.
S.J.D. Prince Computer Vision Models. Available in draft form at http://computervisionmodels.blogspot.com/
![Page 7: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/7.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
7
Week Date Topic Required Readings Additional Readings Application Paper
1 M Sept 13 W Sept 15
Probability & Bayesian Inference Probability Distributions & Parametric Modeling
Bishop Ch 1.1-1.2.5 (29 pages) Bishop Ch 2.1-2.3 (skip 2.3.5) (43 pages)
Pearl Ch 1.4-1.6, 2 Howson & Urbach 1991 Prince Ch 1-4 Duda Ch 3.1-3.5
2 M Sept 20 W Sept 22
Probability Distributions & Parametric Modeling (cntd.) Non-Parametric Modeling
Bishop Ch 2.5 (7 pages)
Duda Ch 4.1-4.5
Comaniciu & Meer 2002 (Mean Shift)
3 M Sept 27 W Sept 29
Expectation Maximization Prince Ch 5 (11 pages) Prince Ch 6.1-6.5, 6.8 (24 pages)
Bishop Ch 9
4 M Oct 4 W Oct 6
Linear Subspace Models Prince Ch 6.6-6.7, 6.9 (12 pages) Bishop Ch 12 (40 pages)
Duda Ch 10.13-10.14
M Oct 11 W Oct 13
Reading Week
5 M Oct 18 W Oct 20
Linear Regression Bishop Ch 3 (36 pages) Prince Ch 7.1-7.2
6 M Oct 25 W Oct 27
Linear Classifiers Bishop Ch 4.1-4.3 (34 pages) Duda 5.1-5.8
7 M Nov 1 W Nov 3
Non-Linear Regression & Classification
Bishop Ch 6 (29 pages) Prince Ch 7.3-7.4
8 M Nov 8 W Nov 10
Sparse Kernel Machines Bishop 7.1 (20 pages)
9 M Nov 15 W Nov 17
Graphical Models: Introduction
Bishop Ch 8.1-8.3 (34 pages)
10 M Nov 22 W Nov 24
Graphical Models: Inference
Bishop Ch 8.4 (25 pages)
11 M Nov 29 W Dec 1
Graphical Models: Applications
Prince Ch 10-11 (56 pages)
12 M Dec 6 W Dec 8
Sampling Methods Bishop Ch 11 (32 pages)
![Page 8: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/8.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
8
Approximate Schedule Week Date Topic Required Readings Additional Readings Application Paper
1 M Sept 13 W Sept 15
Probability & Bayesian Inference Probability Distributions & Parametric Modeling
Bishop Ch 1.1-1.2.5 (29 pages) Bishop Ch 2.1-2.3 (skip 2.3.5) (43 pages)
Pearl Ch 1.4-1.6, 2 Howson & Urbach 1991 Prince Ch 1-4 Duda Ch 3.1-3.5
2 M Sept 20 W Sept 22
Probability Distributions & Parametric Modeling (cntd.) Non-Parametric Modeling
Bishop Ch 2.5 (7 pages)
Duda Ch 4.1-4.5
Comaniciu & Meer 2002 (Mean Shift)
3 M Sept 27 W Sept 29
Expectation Maximization Prince Ch 5 (11 pages) Prince Ch 6.1-6.5, 6.8 (24 pages)
Bishop Ch 9
4 M Oct 4 W Oct 6
Linear Subspace Models Prince Ch 6.6-6.7, 6.9 (12 pages) Bishop Ch 12 (40 pages)
Duda Ch 10.13-10.14
M Oct 11 W Oct 13
Reading Week
5 M Oct 18 W Oct 20
Linear Regression Bishop Ch 3 (36 pages) Prince Ch 7.1-7.2
6 M Oct 25 W Oct 27
Linear Classifiers Bishop Ch 4.1-4.3 (34 pages) Duda 5.1-5.8
7 M Nov 1 W Nov 3
Non-Linear Regression & Classification
Bishop Ch 6 (29 pages) Prince Ch 7.3-7.4
8 M Nov 8 W Nov 10
Sparse Kernel Machines Bishop 7.1 (20 pages)
9 M Nov 15 W Nov 17
Graphical Models: Introduction
Bishop Ch 8.1-8.3 (34 pages)
10 M Nov 22 W Nov 24
Graphical Models: Inference
Bishop Ch 8.4 (25 pages)
11 M Nov 29 W Dec 1
Graphical Models: Applications
Prince Ch 10-11 (56 pages)
12 M Dec 6 W Dec 8
Sampling Methods Bishop Ch 11 (32 pages)
![Page 9: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/9.jpg)
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
PROBABILITY AND BAYESIAN INFERENCE
![Page 10: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/10.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
10
Credits
Some of these slides were sourced and/or modified from: Christopher Bishop, Microsoft UK Simon Prince, UCL
![Page 11: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/11.jpg)
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
INTRODUCTION: VISION AS BAYESIAN INFERENCE
![Page 12: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/12.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
12
Helmholtz
Recognized ambiguity of images.
Knowledge of scene properties and image formation used to resolve ambiguity and infer object properties.
“Vision as Unconscious Inference”
Can be formalized as Bayesian Decision Theory.
Hermann von Helmholtz
![Page 13: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/13.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
13
Helmholtz’ Likelihood Principle
Claim 1: The world is uncertain (to the observer) Claim 2: Vision is ill-posed Claim 3: Observers have evolved (are built) to
perform valuable tasks well Conclusion: Vision is probabilistic inference
![Page 14: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/14.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
14
Vision is Ill-Posed
Noise “surface noise”
atmospheric effects
photon noise
neural noise
Dimensionality 1D 2D
2D 3D
Composition e.g. Bilinear problem of colour (lightness) constancy:
![Page 15: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/15.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
15
Vision is Ill-Posed 2D 3D (N:1 Mapping)
Different Objects
Similar Images
From Kersten et al., 2004
![Page 16: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/16.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
16
Vision is ill-posed (bilinearity of image)
1:N Mapping
N:1 Mapping
From Kersten et al., 2004
![Page 17: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/17.jpg)
![Page 18: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/18.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
18
Julian Beever
![Page 19: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/19.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
19
Julian Beever
![Page 20: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/20.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
20
Julian Beever
![Page 21: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/21.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
21
Liu Bolin
![Page 22: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/22.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
22
Liu Bolin
![Page 23: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/23.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
23
Liu Bolin
![Page 24: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/24.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
24
Bayes’ Rule
p(S |I) ∝ p(I |S)p(S)Posterior Likelihood Prior ∝ ×
Scene Property
To Be Inferred
Image Observation
![Page 25: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/25.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
25
Generative Model:
From Kersten et al., 2004
Generative Models and Ideal Observers
![Page 26: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/26.jpg)
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
TOPIC 1. PROBABILITY & BAYESIAN INFERENCE
![Page 27: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/27.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
27
Random Variables
A random variable is a variable whose value is uncertain.
For example, the height of a randomly selected person in this class is a random variable – I won’t know its value until the person is selected.
Note that we are not completely uncertain about most random variables.
For example, we know that height will probably be in the 5’-6’ range.
In addition, 5’6” is more likely than 5’0” or 6’0”.
The function that describes the probability of each possible value of the random variable is called a probability distribution.
![Page 28: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/28.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
28
Probability Distributions
For a discrete distribution, the probabilities over all possible values of the random variable must sum to 1.
![Page 29: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/29.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
29
Probability Distributions For a discrete distribution, we can talk about the probability of a particular score
occurring, e.g., p(Province = Ontario) = 0.36.
We can also talk about the probability of any one of a subset of scores occurring, e.g., p(Province = Ontario or Quebec) = 0.50.
In general, we refer to these occurrences as events.
![Page 30: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/30.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
30
Probability Distributions
For a continuous distribution, the probabilities over all possible values of the random variable must integrate to 1 (i.e., the area under the curve must be 1).
Note that the height of a continuous distribution can exceed 1!
S h a d e d a r e a = 0 . 6 8 3 S h a d e d a r e a = 0 . 9 5 4 S h a d e d a r e a = 0 . 9 9 7
![Page 31: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/31.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
31
Continuous Distributions
For continuous distributions, it does not make sense to talk about the probability of an exact score. e.g., what is the probability that your height is exactly 65.485948467… inches?
55 60 65 70 75 0 0.02 0.04 0.06 0.08
0.1 0.12 0.14 0.16
Height (in)
Prob
abilit
y p
Normal Approximation to probability distribution for height of Canadian females (parameters from General Social Survey, 1991)
?
![Page 32: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/32.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
32
Continuous Distributions
It does make sense to talk about the probability of observing a score that falls within a certain range e.g., what is the probability that you are between 5’3” and 5’7”?
e.g., what is the probability that you are less than 5’10”?
55 60 65 70 75 0 0.02 0.04 0.06 0.08
0.1 0.12 0.14 0.16
Height (in)
Prob
abilit
y p
Normal Approximation to probability distribution for height of Canadian females (parameters from General Social Survey, 1991)
Valid events
![Page 33: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/33.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
33
Probability Densities
Probability density (PDF)
Cumulative distribution (CDF)
![Page 34: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/34.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
34
Transformed Densities
![Page 35: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/35.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
35
Joint Distributions
Marginal Probability
Conditional Probability
Joint Probability
![Page 36: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/36.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
36
Joint Distributions
Sum Rule
Product Rule
![Page 37: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/37.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
37
Joint Distributions: The Rules of Probability
Sum Rule
Product Rule
![Page 38: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/38.jpg)
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
END OF LECTURE 1 SEPT 13, 2010
![Page 39: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/39.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
39
Application Papers Week Date Topic Required Readings Additional Readings Application Papers
1 M Sept 13 W Sept 15
Probability & Bayesian Inference Probability Distributions & Parametric Modeling
Bishop Ch 1.1-1.2.5 (29 pages) Bishop Ch 2.1-2.3 (skip 2.3.5) (43 pages)
Pearl Ch 1.4-1.6, 2 Howson & Urbach 1991 Prince Ch 1-4 Duda Ch 3.1-3.5
2 M Sept 20 W Sept 22
Probability Distributions & Parametric Modeling (cntd.) Non-Parametric Modeling
Bishop Ch 2.5 (7 pages)
Duda Ch 4.1-4.5
Comaniciu & Meer 2002 (Mean Shift)
3 M Sept 27 W Sept 29
Expectation Maximization Prince Ch 5 (11 pages) Prince Ch 6.1-6.5, 6.8 (24 pages)
Bishop Ch 9 Stauffer & Grimson 1998 Weber & Perona 2000
4 M Oct 4 W Oct 6
Subspace Models Prince Ch 6.6-6.7, 6.9 (12 pages) Bishop Ch 12 (40 pages)
Duda Ch 10.13-10.14 Tenenbaum et al 2000 Roweis & Saul 2000
M Oct 11 W Oct 13
Reading Week
5 M Oct 18 W Oct 20
Linear Regression Bishop Ch 3 (36 pages) Prince Ch 7.1-7.2 Moghaddam 2002 Cremers 2003
6 M Oct 25 W Oct 27
Linear Classifiers Bishop Ch 4.1-4.3 (34 pages) Duda 5.1-5.8 Belhumeur et al 1997 Martin et al 2004
7 M Nov 1 W Nov 3
Kernel Methods Bishop Ch 6 (29 pages)
Prince Ch 7.3-7.4 Toyama & Blake 2001 Grochow et al 2004
8 M Nov 8 W Nov 10
Sparse Kernel Machines Combining Models
Bishop 7.1 (20 pages) Bishop Ch 14 (20 pages)
Agarwal & Triggs 2006 Zhang et al 2007
9 M Nov 15 W Nov 17
Graphical Models: Introduction
Bishop Ch 8.1-8.3 (34 pages)
Freeman et al 2000 Shi & Malik 2000
10 M Nov 22 W Nov 24
Graphical Models: Inference
Bishop Ch 8.4 (25 pages)
Boykov & Funka-Lea 2006 He et al 2004
11 M Nov 29 W Dec 1
Graphical Models: Applications
Prince Ch 10-11 (56 pages)
Frey & Jojic 2005 Szeliski et al 2008
12 M Dec 6 W Dec 8
Sampling Methods Bishop Ch 11 (32 pages) Zhu 1999 Yuille & Kersten 2006
![Page 40: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/40.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
40
Marginalization
We can recover probability distribution of any variable in a joint distribution by integrating (or summing) over the other variables
![Page 41: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/41.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
41
Conditional Probability
Conditional probability of X given that Y=y* is relative propensity of variable X to take different outcomes given that Y is fixed to be equal to y*
Written as Pr(X|Y=y*)
![Page 42: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/42.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
42
Conditional Probability
Conditional probability can be extracted from joint probability
Extract appropriate slice and normalize
![Page 43: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/43.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
43
Conditional Probability
More usually written in compact form
• Can be re-arranged to give
![Page 44: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/44.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
44
Independence
If two variables X and Y are independent then variable X tells us nothing about variable Y (and vice-versa)
![Page 45: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/45.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
45
Independence
When variables are independent, the joint factorizes into a product of the marginals:
![Page 46: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/46.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
46
Bayes’ Rule
From before:
Combining:
Re-arranging:
![Page 47: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/47.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
47
Bayes’ Rule Terminology
Posterior – what we know about y after seeing x
Prior – what we know about y before seeing x
Likelihood – propensity for observing a certain value of X given a certain value of Y
Evidence –a constant to ensure that the left hand side is a valid distribution
![Page 48: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/48.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
48
Expectations
Condi3onal Expecta3on (discrete)
Approximate Expecta3on (discrete and con3nuous)
![Page 49: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/49.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
49
Variances and Covariances
![Page 50: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/50.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
50
The Gaussian Distribution
![Page 51: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/51.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
51
Gaussian Mean and Variance
![Page 52: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/52.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
52
The Multivariate Gaussian
![Page 53: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/53.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
53
Gaussian Parameter Estimation
Likelihood func3on
![Page 54: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/54.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
54
Maximum (Log) Likelihood
![Page 55: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/55.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
55
Maximum likelihood estimates of normal parameters
![Page 56: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/56.jpg)
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
APPLYING PROBABILITY THEORY TO INFERENCE
![Page 57: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/57.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
57
Polynomial Curve Fitting
![Page 58: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/58.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
58
Sum-of-Squares Error Function
![Page 59: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/59.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
59
1st Order Polynomial
![Page 60: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/60.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
60
3rd Order Polynomial
![Page 61: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/61.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
61
9th Order Polynomial
![Page 62: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/62.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
62
Over-fitting
Root-‐Mean-‐Square (RMS) Error:
![Page 63: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/63.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
63
Overfitting and Sample Size
9th Order Polynomial
![Page 64: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/64.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
64
Overfitting and Sample Size
9th Order Polynomial
![Page 65: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/65.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
65
Regularization
Penalize large coefficient values
![Page 66: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/66.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
66
Regularization
9th Order Polynomial
![Page 67: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/67.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
67
Regularization
9th Order Polynomial
![Page 68: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/68.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
68
Regularization
9th Order Polynomial
![Page 69: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/69.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
69
Probabilistic View of Curve Fitting
![Page 70: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/70.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
70
Maximum Likelihood
Determine by minimizing sum-‐of-‐squares error, .
![Page 71: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/71.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
71
MAP: A Step towards Bayes
Determine by minimizing regularized sum-‐of-‐squares error, .
![Page 72: 01Probability and Bayesian Inference · Julian Beever . Probability & Bayesian Inference CSE 6390/PSYC 6225 Computational Modeling of Visual Perception J. Elder 19 Julian Beever](https://reader034.vdocuments.us/reader034/viewer/2022052014/602b871cd3302357417dc1fd/html5/thumbnails/72.jpg)
Probability & Bayesian Inference
J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
72
Some Key Ideas
Change of variables and transformed densities Derivation of sum and product rules of probability Maximum likelihood and bias Least-squares as optimal probabilistic modeling