ibs-09-sl rm 501 – ranjit goswami 1 basic probability
TRANSCRIPT
![Page 1: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/1.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
1
Basic Probability
![Page 2: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/2.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
2
Introduction
• Probability is the study of randomness and uncertainty.
• In the early days, probability was associated with games of chance (gambling).
![Page 3: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/3.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
3
Simple Games Involving Probability
Game: A fair die is rolled. If the result is 2, 3, or 4, you win $1; if it is 5, you win $2; but if it is 1 or 6, you lose $3.
Should you play this game?
![Page 4: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/4.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
4
Random Experiment
• a random experiment is a process whose outcome is uncertain.
Examples:
• Tossing a coin once or several times
• Picking a card or cards from a deck
• Measuring temperature of patients
• ...
![Page 5: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/5.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
5
Sample SpaceThe sample space is the set of all possible outcomes.
Simple EventsThe individual outcomes are called simple events.
EventAn event is any collectionof one or more simple events
Events & Sample Spaces
![Page 6: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/6.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
6
Example
Experiment: Toss a coin 3 times.
• Sample space
= {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}.
• Examples of events include
• A = {HHH, HHT,HTH, THH}
= {at least two heads}
• B = {HTT, THT,TTH}
= {exactly two tails.}
![Page 7: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/7.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
7
Basic Concepts (from Set Theory)
• The union of two events A and B, A B, is the event consisting of all outcomes that are either in A or in B or in both events.
• The complement of an event A, Ac, is the set of all outcomes in that are not in A.
• The intersection of two events A and B, A B, is the event consisting of all outcomes that are in both events.
• When two events A and B have no outcomes in common, they are said to be mutually exclusive, or disjoint, events.
![Page 8: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/8.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
8
Example
Experiment: toss a coin 10 times and the number of heads is observed.
• Let A = { 0, 2, 4, 6, 8, 10}.
• B = { 1, 3, 5, 7, 9}, C = {0, 1, 2, 3, 4, 5}.
• A B= {0, 1, …, 10} = .
• A B contains no outcomes. So A and B are mutually exclusive.
• Cc = {6, 7, 8, 9, 10}, A C = {0, 2, 4}.
![Page 9: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/9.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
9
Rules
• Commutative Laws: • A B = B A, A B = B A
• Associative Laws: • (A B) C = A (B C )
• (A B) C = A (B C) .
• Distributive Laws:• (A B) C = (A C) (B C)
• (A B) C = (A C) (B C)
![Page 10: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/10.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
10
Venn Diagram
A BA∩B
![Page 11: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/11.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
11
Probability
• A Probability is a number assigned to each subset (events) of a sample space .
• Probability distributions satisfy the following rules:
![Page 12: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/12.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
12
Axioms of Probability
• For any event A, 0 P(A) 1.
• P() =1.
• If A1, A2, … An is a partition of A, then
P(A) = P(A1) + P(A2) + ...+ P(An)
(A1, A2, … An is called a partition of A if A1 A2 … An = A and A1, A2, … An are mutually exclusive.)
![Page 13: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/13.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
13
Properties of Probability
• For any event A, P(Ac) = 1 - P(A).
• If A B, then P(A) P(B).
• For any two events A and B,
P(A B) = P(A) + P(B) - P(A B).
For three events, A, B, and C,
P(ABC) = P(A) + P(B) + P(C) -
P(AB) - P(AC) - P(BC) + P(AB C).
![Page 14: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/14.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
14
Example
• In a certain population, 10% of the people are rich, 5% are famous, and 3% are both rich and famous. A person is randomly selected from this population. What is the chance that the person is
• not rich?
• rich but not famous?
• either rich or famous?
![Page 15: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/15.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
15
Intuitive Development (agrees with axioms)
• Intuitively, the probability of an event a could be defined as:
Where N(a) is the number that event a happens in n trialsWhere N(a) is the number that event a happens in n trials
![Page 16: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/16.jpg)
Here We Go Again: Not So Basic Probability
![Page 17: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/17.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
17
More Formal:
• is the Sample Space:• Contains all possible outcomes of an experiment
• in is a single outcome
• A in is a set of outcomes of interest
![Page 18: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/18.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
18
Independence
• The probability of independent events A, B and C is given by:
P(A,B,C) = P(A)P(B)P(C)
A and B are independent, if knowing that A has happened A and B are independent, if knowing that A has happened does not say anything about B happeningdoes not say anything about B happening
![Page 19: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/19.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
19
Bayes Theorem
• Provides a way to convert a-priori probabilities to a-posteriori probabilities:
![Page 20: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/20.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
20
Conditional Probability
• One of the most useful concepts!
AABB
![Page 21: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/21.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
21
Bayes Theorem
• Provides a way to convert a-priori probabilities to a-posteriori probabilities:
![Page 22: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/22.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
22
Using Partitions:
• If events Ai are mutually exclusive and partition
![Page 23: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/23.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
23
Random Variables
• A (scalar) random variable X is a function that maps the outcome of a random event into real scalar values
X(X())
![Page 24: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/24.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
24
Random Variables Distributions
• Cumulative Probability Distribution (CDF):
• Probability Density Function (PDF):Probability Density Function (PDF):
![Page 25: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/25.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
25
Random Distributions:
• From the two previous equations:
![Page 26: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/26.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
26
Uniform Distribution
• A R.V. X that is uniformly distributed between x1 and x2 has density function:
XX11 XX22
![Page 27: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/27.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
27
Gaussian (Normal) Distribution
• A R.V. X that is normally distributed has density function:
![Page 28: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/28.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
28
Statistical Characterizations
• Expectation (Mean Value, First Moment):
•Second Moment:Second Moment:
![Page 29: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/29.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
29
Statistical Characterizations
• Variance of X:
• Standard Deviation of X:
![Page 30: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/30.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
30
Mean Estimation from Samples
• Given a set of N samples from a distribution, we can estimate the mean of the distribution by:
![Page 31: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/31.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
31
Variance Estimation from Samples
• Given a set of N samples from a distribution, we can estimate the variance of the distribution by:
![Page 32: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/32.jpg)
Pattern Classification
![Page 33: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/33.jpg)
Chapter 1: Introduction to Pattern Recognition (Sections 1.1-1.6)
• Machine Perception
• An Example
• Pattern Recognition Systems
• The Design Cycle
• Learning and Adaptation
• Conclusion
![Page 34: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/34.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
34
Machine Perception
• Build a machine that can recognize patterns:
• Speech recognition
• Fingerprint identification
• OCR (Optical Character Recognition)
• DNA sequence identification
![Page 35: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/35.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
35
An Example
• “Sorting incoming Fish on a conveyor according to species using optical sensing”
Sea bass
Species
Salmon
![Page 36: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/36.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
36
• Problem Analysis
• Set up a camera and take some sample images to extract features
• Length
• Lightness
• Width
• Number and shape of fins
• Position of the mouth, etc…
• This is the set of all suggested features to explore for use in our classifier!
![Page 37: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/37.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
37
• Preprocessing
• Use a segmentation operation to isolate fishes from one another and from the background
• Information from a single fish is sent to a feature extractor whose purpose is to reduce the data by measuring certain features
• The features are passed to a classifier
![Page 38: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/38.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
38
![Page 39: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/39.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
39
• Classification
• Select the length of the fish as a possible feature for discrimination
![Page 40: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/40.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
40
![Page 41: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/41.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
41
The length is a poor feature alone!
Select the lightness as a possible feature.
![Page 42: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/42.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
42
![Page 43: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/43.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
43
• Threshold decision boundary and cost relationship
• Move our decision boundary toward smaller values of lightness in order to minimize the cost (reduce the number of sea bass that are classified salmon!)
Task of decision theory
![Page 44: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/44.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
44
• Adopt the lightness and add the width of the fish
Fish xT = [x1, x2]
Lightness Width
![Page 45: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/45.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
45
![Page 46: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/46.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
46
• We might add other features that are not correlated with the ones we already have. A precaution should be taken not to reduce the performance by adding such “noisy features”
• Ideally, the best decision boundary should be the one which provides an optimal performance such as in the following figure:
![Page 47: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/47.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
47
![Page 48: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/48.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
48
• However, our satisfaction is premature because the central aim of designing a classifier is to correctly classify novel input
Issue of generalization!
![Page 49: IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability](https://reader035.vdocuments.us/reader035/viewer/2022062304/56649e7b5503460f94b7bc61/html5/thumbnails/49.jpg)
IBS-09-SL RM 501 – Ranjit Goswami
49