lecture 1 - chapter 2 of stats textbook

7/28/2019 Lecture 1 - Chapter 2 of Stats Textbook

http://slidepdf.com/reader/full/lecture-1-chapter-2-of-stats-textbook 1/66

Statistics Probability

Introduction to probability

STA 113 Chapter 2 of Devore

Aug 31, Sep 2, 2010





Table of contents

1 Statistics

2 ProbabilityGeneral discussions

Basic conceptsSet theoryAxiomsPropertiesCombinatorics – countingConditional probabilityIndependence





Statistics

Definition from Wikipedia:Statistics is a mathematical science pertaining to the collection, analysis, interpretation or explanation, and presentation of data.

Statistics arose between 17th to 18th century, from the needof states to collect data on their people and economies, inorder to administer them.

Mathematical methods of statistics emerged from probabilitytheory.

Key element: account for (model) randomness anduncertainty in the observations.

Two current main schools: frequenists and Bayesian.





Examples

Famous story - does smoking cause lung cancer?

How to model and predict freeway traffic flows?

How to detect cancerous genes from microarray data?

How to estimate the effect of air pollution on public health

from observational environmental data?

How to determine if new drugs and medical devices help fromclinical trials?

Does lowering the drink age increase car accident rate (DUI

among youngsters)?U.S. presidential election: do incumbents have advantage?

...





Example: Needle Exchange Program

Needle exchange programs (NEP): give free clean needles inexchange of used ones to injection drug users. Hope to reduceHIV incidence.

Scientific question: Will NEP help in the long run?Q1: What data should we collect? - Design.

Q2: What information statements can we make when we havedata? - Inference.

Q3: What action to make? - Decision.In this course, we will focus on “inference”.


S i i P b bili G l di i B i S h A i P



Statistics Probability General discussions Basic concepts Set theory Axioms Pro

Probability: some discussion

From Encyclopaedia Britannica: Probability theory is the

branch of mathematics concerned with analysis of randomphenomena.

Probability theory is a mathematical foundation for statistics.


St ti ti P b bilit G l di i B i t S t th A i P




Probability: some discussion

Useful strategy used in much of science:For a given experiment

attribute all that is known or theorized to a mechanistic model(mathematical function)

attribute everything else to randomness, even if the process under study is known not the be “random” in any sense of the word Use probability to quantify the uncertainty in your conclusionsEvaluate the sensitivity of your conclusions to the assumptionsof your model

Probability has been found extraordinarily useful, even if truerandomness is an elusive, undefined, quantity


Statistics Probabilit General disc ssions Basic concepts Set theor Axioms Pro




Two types of Probability

Two types

1 Discrete – male or female, heads or tails, number of radioactive particles, how many buses go by, counts of nucleotides

2 Continuous – height of people, size of the head of a crab,length of time between volcanic eruptions, images (?), speechsignals






Experiment

DefinitionBroadly, an experiment is any action or process whose outcome is subject to uncertainty.

Examples of the outcome of an experiment:a collection of measurement from a sampled population.

measurement from a laboratory experiment

the result of a clinical trial

survival time of a cancer patientthe result of a simulated computer experiment

...






Sample space

Definition

A sample space S is the set of all possible outcomes of anexperiment.

Examples:

1 Experiment is flipping one coin – Sample space is {H , T }.

2 Experiment is flipping two coins – Sample space is{HH , TT , HT , TH }.

3 Experiment is flipping n coins – How many elements in thesample space ?






Sample space

Definition

A sample space S is the set of all possible outcomes of anexperiment.

Examples:5 Experiment is playing five rounds of Russian roulette – Sample

space is {D , LD , LLD , LLLD , LLLD }. Why is this differentthan coin flipping.

6 Experiment is sequencing three nucleotides – Sample space is{AAA, CCC , GGG , TTT , AAC , AAT , AAG , ..., }. How big isthis sample space ? (Hint: There are four nucleotides.)






An event

Definition

A event is any collection of outcomes contained in the sample space, S .

Examples:

1 Flipping one coin – Getting heads.

2 Flipping two coins – Getting two heads.

3 Playing Russian roulette – Getting shot on the fourth spin.

4 Sequencing nucleotides – Your genome.





y p y

Set theory

An event is a subset of the sample space S . Normal set operationshave particular interpretations in this setting

Definition

1 A ⊂ B implies that the occurrence of A implies the occurrence of B

2 Intersection A ∩ B: implies the event that both A and B occur

3 Union A ∪ B: implies the event that at least one of A or B occur

4 The complement of an event A denoted A (also notated Ac

or A): the set of all outcomes in S not contained in A,A = S\A - the event that A does not occur





More set theory

Definition

Two events A and B are mutually exclusive or disjoint if they

have no outcomes in common, i.e. A ∩ B = Ø.

Being a Duke fan and a Carolina fan is mutually exclusive.What is the event and what is the experiment ?





Fun aside: Barber paradox

A laymen version of the famous Russell’s paradox, discovered byBertrand Russell in 1901.

Consider a barber who shaves men if and only if they do notshave themselves.

Should the barber shave himself or not?If yes, then the barber should not shave himself, by his rule.If no, then the barber should shave himself, by his rule.

Generally, consider R , the set containing all sets that do not

contain themselves as members.Does R contains itself? Contradiction





What is probability?

Given an experiment and a sample space S , the objective of probability is to assign to each event A a number Pr(A), called theprobability of the event A, which will give a precise measure of thechance that A will occur.

1 Event: A = Duke wins next year’s NCAA basketball title?.What is Pr(A) ?

2 Event: A = someone here wins the lottery. What is Pr(A) ?

3 Event: A = I spin a quarter and it comes up heads. What is

Pr(A) ?





Probability: Frequentist interpretation

There are several possible interpretation of probability.

There is not agreement, at all, in how probabilities should beinterpreted

There is (nearly) complete agreement on the mathematical

rules probability must follow (axioms)Frequentist interpretation: A probability is the frequency of anevent will occur in repeated identical repetitions of anexperiment.

Often associated with Jerzy Neyman and Egon Pearson whodescribed the logic of statistical hypothesis testing.

The single main stream school until recently.





Probability: Bayesian interpretation

An alternative interpretation of probability is so-called“Bayesian”,

Named after the 18th century Presbyterian minister andmathematician Thomas Bayes.

A Bayesian interprets probability as a subjective degree of belief: For the same event, two separate people could have differing probabilities.

Bayesian interpretations of probabilities avoid some of the

philosophical difficulties of frequency interpretations.Largely popularized by revolutionary advance in computationaltechnology and methods during the last twenty years.





Axiomatic Foundations of Probability

The foundations of modern probability theory, the axiomaticbasis, are laid by Andrey Kolmogorov in 1933.

The axiomatic approach is not concerned with theinterpretation of probabilities.

Concerned only that probabilities are defined by a functionsatisfying the axioms.





Axioms of probability

DefinitionAxiom 1: For any event A, Pr(A) ≥ 0 .Axiom 2: Pr(S ) = 1.Axiom 3: If A1, A2, ..., Ak are mutually exclusive events, then

Pr

A1

A2

· · ·

Ak

=

k i =1

Pr(Ai ).

Let k → ∞, A1, A2, A3, ... are mutually exclusive events

Pr

A1

A2

· · ·

=∞i =1

Pr(Ai ).





Axioms of probability

Example: coin flipS = {H , T } and H

T = S

1 = Pr(S )

1 = Pr(H ) + Pr(T ) = Pr(S )

Pr(T ) = 1 − Pr(H ) = p





Properties of probability – Venn diagrams

Following properties can all be derived from the axioms, andunderstood from Venn diagrams.

Proposition

For any event A, Pr(A) = 1 − Pr(A)

Sometimes A is easier to compute than A.Example: Flip n coins with Pr(H ) = p . What is the probability of the event one or more heads ?

Pr(A) = Pr(TTTTT · · · ) = (1 − p )n

Pr(A) = 1 − (1 − p )n.






Proposition

If A and B are mutually exclusive, then Pr (A

B ) = 0 and A

B = ∅.



P i f b bili V di




Proposition

For any two events A and B,

Pr

A

B

= Pr(A) + Pr(B ) − Pr

A

B

.



P i f b bili V di




For any three events A, B , C ,

Pr AB C = Pr(A) + Pr(B ) + Pr(C )

− Pr

A

B

− Pr

A

C

− Pr

B

C

+ Pr

A

B

C

.



P i f b bili V di




For any three events A, B , C ,

Pr AB C = Pr(A) + Pr(B ) + Pr(C )

− Pr

A

B

− Pr

A

C

− Pr

B

C

+ Pr

A

B

C

.



P ti f b bilit V di




In an experiment with N outcomes that are equally likely theprobability of any outcome A is

Pr(A) = p =1

N .

Roll a six sided dice. What is p ?



C ti 101



Counting 101

When you have to count, go through the following checklist

1 Does order matter ?

2

Can objects be chosen only once ?3 Can I break the problem into simpler computations ?

4 Is the complement simpler to work with ?

5 Can I simplify computations by conditioning ?



Order matters with repetition




Given a set of objects for example A, B , C , D .Assume

1 Order matters: ABC is different from CBA.

2 Objects can be selected more than once, i.e. AAA is possible.







An ordered collection of k objects is a k -tuple.

Your genome is a k -tuple, {AAACTGATTTCCC · · · }, with k ≈ 3billion.

A fixed price meal is a k -tuple, {app, main, dessert}, with k = 3.



Order matters product rule



Order matters - product rule

Proposition

A set consists of ordered collections of k-tuples with n1 choices for element 1, n2 choices for element 2 ,..., and nk choices for element k. The number of possible k-tuples are n1n2 · · · nk .



Fixed price menu



Fixed price menu



Fixed price menu



Fixed price menu

Appetizers: n1 = 6Mains: n2 = 6Desserts n3 = 6Number of meals: n1 × n2 × n3 = 216.



An argument for intelligent design



An argument for intelligent design

Your genome is a k -tuple, with k ≈ 3 billion.William A. Dembski makes the following argument.

1 Assume that A, C , T , G are equally likely.

2 Assume that order matters and an amino acid in one positiondoes not effect others.

3 The probability of your genome is 1/430,000,000,000.

4 Astronomically small chance so cannot be random, or evolve

from random mutations.



Order matters without repetition - permutation



Order matters without repetition permutation


1 Order matters: ABC is different from CBA.2 Objects cannot be selected more than once: AAA is not

possible, neither is AAB . ACB is possible.



Order matters without repetition - permutation



Order matters without repetition permutation

Example: Pick a starting lineup for Duke given the 12 people onthe roster.

P 5,12 = 12 × 11 × 10 × 9 × 8.

More generally

P k ,n =n!

(n − k )!,

where n! = n(n − 1)(n − 2)(n − 3) · · · 1 and 0! = 1.



Order doesn’t matter without repetition - combinations



Order doesn t matter without repetition combinations


1 Order does not matter: ABC is the same as CBA.2 Objects cannot be selected more than once: AAA is not

possible, neither is AAB . ACB is possible.






Order doesn t matter without repetition combinations

Definition

Given n distinct objects, any unordered subset of size k objects is a

combination. The number of combinations is nk

or n choose k.

A bridge hand is a combination.

So is a poker hand.






p

An example: Given the set {A, B , C , D , E } how manycombinations of size three ?

1 First observation: P 3,5 > 53, why ?

Permutations{ABC }, {ACB }, {BCA}, {BAC }, {CBA}, {CAB }Combination {ABC }.

2 Second observation: for any combination of size 3 there are

3! = 6 permutations.






p

An example: Given the set {A, B , C , D , E } how manycombinations of size three ?

3 Third observation:

P 3,5 =5

3

× 3!5

3

=

P 3,5

3!=

5!

3!(5 − 3)!.

4

Induction:

n

k

=

n!

k !(n − k )!.



Poker hands



52 playing cards and you are dealt 5 cards (a hand).4 suits : ♣, ♥, ♠, ♦.Each suit has 13 cards 2 − 9, J , Q , K , A.You want the best hand.

Some poker handsNone the samePair

Two pairsThree of a kind

Full houseFour of a kind

Royal flush



Poker hands



Number of total poker hands:

525

= 2, 598, 960.



None the same



Number of none the same:Counting strategy 1: break into cases and then multiplyCases: face cards and suits.Choices of faces

135

: 5 different faces out of 13 for example

{3 8 5 2 9}.Choices of suits 4 × 4 × 4 × 4 × 4: each card can be one of foursuits for example {♣♥♠♦♥}Total:

135

× 45 = 1, 317, 888

Pr(none the same) =1, 317, 888

2, 598, 960 = 50.7%.



Flush



Choices of faces

135

: 5 different faces out of 13.

Choices of suits 4.Total: 13

5× 4 = 5148

Pr(flush) =5148

2, 598, 960= 0.2%.



None the same no flush



This example uses the idea that sometimes the complement iseasier to count.The set {None the same and no flush} can be written as

{None the same and no flush} = {None the same}−{None the same and

Observe: {None the same and flush} = {flush}.Total:

135

× 45 −

135

× 4 = 1, 312, 740

Pr(none the same no flush) =

5148

2, 598, 960 = 50.5%.



Conditional probability, motivation



The probability of getting a one when rolling a (standard) dieis usually assumed to be one sixth

Suppose you were given the extra information that the die rollwas an odd number (hence 1, 3 or 5)

Conditional on this new information, the probability of a oneis now one third



Conditional probability



Definition

Given two events A and B with Pr(B ) > 0, the conditional

probability of A given B has occurred is defined as

Pr(A|B ) =Pr(A

B )

Pr(B ).

When B is the sample space: P (B |B ) = 1



Example



Consider our die roll example

B = {1, 3, 5}

A = {1}

P (one given that roll is odd) = P (A | B )

=P (A ∩ B )

P (B )

=P (A)

P (B )

=1/6

3/6=

1

3






Proposition (Multiplication rule)

Pr(B

A) = Pr(B |A) × Pr(A).

Proposition (Law of total probability)

Let A1, ..., Ak be mutually exclusive and exhaustive events (i.e.,k i =1 Ai = S ). Then for any other event B

Pr(B ) =k

i =1

Pr(B |Ai ) × Pr(Ai ).

Applies when k go to ∞.Such collection of sets A1, ..., Ak is also called a partition of samplespace.






Proof of Law of total probability

By logic

B = (A1 and B ) or (A2 and B ) or ... or (Ak and B ).

In equation form

B = (A1

B )

(A2

B )

...

(AK

B )).

This implies

Pr(B ) = Pr

(A1

B )

(A2

B )

...

(AK

B )

.






Proof of Law of total probability

Pr(B ) = Pr

(A1

B )

(A2

B )

...

(AK

B )

.

Since Ai are exclusive and exhaustive

Pr(B ) =k

i =1

Pr(Ai

B ) =

k i =1

Pr(B |Ai ) × Pr(Ai ).

The last step is by the multiplication rule.



Bayes’ theorem



Theorem

Let A1, ..., Ak be mutually exclusive and exhaustive events withPr(Ai ) > 0 for all i = 1, ..., k. Then for any other event B withPr(B ) > 0

Pr(A j |B ) =Pr(A j

B )

Pr(B )=

Pr(B |A j )Pr(A j )k i =1 Pr(B |Ai )Pr(Ai )

, j = 1, ..., k .

P (Ai ) is often called prior probability; P (Ai |B ) is called posterior

probability.



Bayes’ theorem



A common application of Bayes’ theorem is the following:

Pr(model|data) =Pr(data|model) Pr(model)

Pr(data).

The important quantities are:

1 Likelihood: Pr(data|model)

2 Prior: Pr(model)

3 Posterior: Pr(model|data)



The reverend Thomas Bayes



Stigler’s Law of Eponymy: no scientificdiscovery is named after its original discovers



Monty Hall



http://en.wikipedia.org/wiki/Monty_Hall_problem

Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats.

You pick a door, say No. 1, and the host, who knows what’sbehind the doors, opens another door, say No. 3, which he knowshas a goat. He then says to you, “Do you want to pick door No.2?” Is it to your advantage to switch your choice?



Monty Hall





A shortcut answer:

Event A: You first pick the right doorEvent B : You switch to the correct door given the host open agoat doorBy the law of total probability

Pr(B ) = Pr(B |A)Pr(A) + Pr(B |A

)Pr(A

)= 0 × 1/3 + 1 × 2/3

= 2/3

Good to switch. What notational shortcut did I take here?In fact, event B is a combination to two events: (1) you switch(B 1), (2) the host open a goat door (B 2).Write out the above equation in terms of A, B 1, B 2 yourself.



Example: diagnostic tests



Let + and − be the events that the result of a diagnostic testis positive or negative respectively

Let D and D c be the event that the subject of the test has or

does not have the disease respectivelyThe sensitivity is the probability that the test is positivegiven that the subject actually has the disease, P (+ | D )

The specificity is the probability that the test is negativegiven that the subject does not have the disease, P (− | D c )



More definitions



The positive predictive value is the probability that thesubject has the disease given that the test is positive,P (D | +)

The negative predictive value is the probability that thesubject does not have the disease given that the test isnegative, P (D c | −)

The prevalence of the disease is the marginal probability of disease, P (D )



More definitions



The diagnostic likelihood ratio of a positive test, labeledDLR +, is P (+ | D )/P (+ | D c ), which is the

sensitivity /(1 − specificity )

The diagnostic likelihood ratio of a negative test, labeledDLR −, is P (− | D )/P (− | D c ), which is the

(1 − sensitivity )/specificity



Example



A study comparing the efficacy of HIV tests, reports on anexperiment which concluded that HIV antibody tests have asensitivity of 99.7% and a specificity of 98.5%

Suppose that a subject, from a population with a .1%

prevalence of HIV, receives a positive test result. What is theprobability that this subject has HIV?

Mathematically, we want P (D | +) given the sensitivity,P (+ | D ) = .997, the specificity, P (− | D c ) = .985, and the

prevalence P (D ) = .001



Using Bayes’ formula



P (D | +) =P (+ | D )P (D )

P (+ | D )P (D ) + P (+ | D c )P (D c )

=P (+ | D )P (D )

P (+ | D )P (D ) + {1 − P (− | D c )}{1 − P (D )}

= .997 × .001.997 × .001 + .015 × .999

= .062

In this population a positive test result only suggests a 6%probability that the subject has the disease

(The positive predictive value is 6% for this test)



Statistical independence



Definition

A and B are independent if and only if

Pr(A

B ) = Pr(A) × Pr(B ).

If A and B are independent, then

P (A | B ) =P (A)P (B )

P (B )= P (A).

The above is equivalent to the definition (alternative definition).



Statistical independence



Knowing B does not provide any info about event A.

Independence is not mutually exclusive (P (A ∩ B ) = 0).

Independence is usually on assumption, usually hard to prove.

Many gambling games provide models of independent events.

If A and B are independent events,A and B , A and B , A and B are also independent.



Example



Volume 309 of Science reports on a physician who was ontrial for expert testimony in a criminal

Based on an estimated prevalence of sudden infant deathsyndrome (SIDS) of 1 out of 8, 543, Dr Meadow testified thatthat the probability of a mother having two children with

SIDS was

18,8543

2

The mother on trial was convicted of murder

What was Dr Meadow’s mistake(s)?



Example: continued



For the purposes of this class, the principal mistake was toassume that the probabilities of having (SIDS) within a familyare independent

That is, P (A1 ∩ A2) is not necessarily equal to P (A1)P (A2)

Biological processes that have a believed genetic or familiarenvironmental component, of course, tend to be dependentwithin families

In addition, the estimated prevalence was obtained from anunpublished report on single cases; hence having noinformation about recurrence of SIDs within families



Mutual independence



Extend the concept of independence into more than twoevents.

Is the condition P (A ∩ B ∩ C ) = P (A)P (B )P (C ) enough?

Is the condition “all the pairs are independent” enough?

Definition

A collection of events A1, ..., An are mutually independent if for any subcollection Ai 1 , ..., Ai k (k > 1), we have

Pr(Ai 1

Ai 2

...Ai k ) = Pr(Ai 1 ) × Pr(Ai 2 ) × ... Pr(Ai k ).


lecture 1 - chapter 2 of stats textbook

Documents