daidd gainesville, florida december 2013 jim scott, ph.d, m.a., m.p.h
TRANSCRIPT
DAIDDGainesville, FloridaDecember 2013
Jim Scott, Ph.D, M.A., M.P.H.
By the end of this talk, I hope you’ll:
Have a good sense of what model evaluation is, why it’s important, and how it’s tied to your research question
Know some of the characteristics that are desirable in models
2
Specific question
Identify relevant factors and information
Model formulation
Mathematics
Evaluation
3
“E” topics: Edison, Thomas Epicycle Epidemiology Elasticity Eratosthenes Euler Existence Evolution Extrapolation Eradication
“M” topics: Malthus, Thomas Mars Maternity Maximum likelihood Maxwell, James C. Misery Monte Carlo Moon Mortality Mumps
4
Let’s look at a few different types of models
5
Source: Wikipedia
6Source: Wikipedia
7Source: Wikipedia
8
Source: XKCD
9
Thomas Jefferson
John Adams
John Adams
Thomas Jefferson
Thomas Jefferson
James Madison
James Madison
DeWitt Clinton
James Monroe
John Quincy Adams
Andrew Jackson
Andrew Jackson
John Quincy Adams
Henry Clay
Andrew Jackson
Martin Van Buren
William Henry Harrison
William Henry Harrison
Martin Van Buren
James K. PolkHenry Clay
Lewis Cass
Zachary Taylor
Winfield Scott
Franklin Pierce
John C. Fremont
James Buchanan
Stephen A. Douglas
Abraham Lincoln
Abraham Lincoln
George B. McClellan
Uylsses S. Grant
Ulysses S. Grant
Horace Greeley
Rutherford B. Hayes
Samuel J. Tilden
James A. GarfieldWinfield HancockGrover ClevelandJames G. BlaineGrover ClevelandBenjamin HarrisonGrover Cleveland
Benjamin Harrison
William McKinley
William J. Bryan
William McKinley
William J. Bryan
Theodore Roosevelt
Alton B. Parker
William Howard Taft
William J. BryanWoodrow Wilson
Theodore Roosevelt
Woodrow Wilson
Charles E. Hughes
James Cocks
Warren G. Harding
Calvin Coolidge
John W. Davis
Herbert Hoover
Al Smith
Franklin D. Roosevelt
Herbert Hoover
Alf Landon
Franklin D. Roosevelt
Franklin D. Roosevelt
Wendell Willkie
Franklin D. Roosevelt
Thomas E. Dewey
Harry S. Truman
Thomas E. Dewey
Dwight D. Eisenhower
Adlai Stevenson
Dwight D. Eisenhower
Adlai Stevenson
John F. KennedyRichard Nixon
Lyndon Johnson
Barry Goldwater
Richard NixonHubert Humphrey
Richard Nixon
George McGovern
Gerald FordJimmy Carter
Jimmy Carter
Ronald Reagan
Ronald Reagan
Walter Mondale
George H. W. Bush
Michael Dukakis
Bill Clinton
George H. W. Bush
Bill Clinton
Bob Dole
George W. BushAl GoreGeorge W. Bush
John Kerry
Barack Obama
John McCain
30
40
50
60
70
160 170 180 190 200heightcm
Influence of Candidate Height on Presidential Elections
Estimated Percent Popular Vote = 1.07 + 0.26 * Height
10
Source: Witlock & Schluter, Analysis of Biological Data
11
The Krebs Cycle
The Eye
= birth rate
N = S + I
= infection rate
I = Weibull mortality
S I I
N SI /N IS
0.0
0.2
0.4
0.6
0.8
1.0
0 10 20 30Time (years)
P(s
urv
ivin
g)
Normal (Weibull 2)
Exponential(Weibull 1)
Slide credit: J. Hargrove/B. Williams
Consider the previous examples
Talk with someone next to you
Come up with a list of characteristics that a “good” model should have
For example, you might say “simplicity”
13
Accurate (i.e. low bias)
14
A model is accurate if estimates based on the model match the truth E.g. models that are used to predict the
weather are reasonably accurate when predicting tomorrow’s weather. They are much less accurate at predicting the weather at times further into the future.
Does the model fit the data
15
16Source: Wikipedia
17
Farr initially believed in the miasma theory of disease transmission – disease was propagated by “bad air”
The higher the elevation, the better the air
Mortality at terrace level X = Mortality at terrace 1 / terrace level
X18
19
20r = 0.9972 Est. Temp = 155.3 + 1.90 * Pressure
195
200
205
210
215
Tem
p -
F
20 22 24 26 28 30Pressure - Inches of Hg
Boiling Point of Water vs Pressure
Source: Wikipedia
Accurate (i.e. low bias)
Descriptively realistic
21
A model is descriptively realistic if it’s derived from a correct description of the mechanism involved in whatever is being modeled Corollary: underlying assumptions are correct Statistical models are not descriptively realistic
▪ For example, a linear equation only models a pattern in the data – there’s nothing telling us what’s going on behind the scene
An SIR model is more descriptively realistic▪ A mechanism for transmission is specified
22
23
33
.13
.23
.33
.4
.0047 .0048 .0049 .005 .0051 .0052InverseT
Source: Wikipedia
Ln P
Accurate (i.e. low bias)
Descriptively realistic
Precise (i.e. low variability)
24
A model is precise if the estimates that the model produces have low variability E.g. a model that estimates that it will
start to rain in the next 3 – 6 hours is more precise than a model that estimates it will start to rain in the next 3 – 6 days
25
26Weekly U.S. Influenza Surveillance Report, http://www.cdc.gov/flu/weekly/
27
I(a,t) is the incidence at age a at time t
P(a,t) is the age-specific prevalence at age a at time t
*
* Simplified model
28
Accurate (i.e. low bias)
Descriptively realistic
Precise (i.e. low variability)
General
29
A model is general if it applies to a wide variety of situations e.g. the law of supply and demand
30Source: Wikipedia
31
N
SI
dt
dS I
N
SI
dt
dI I
dt
dR
S I R
33
A
BX
X
ZX
A
Z
Each pie represents a sufficient cause for disease (i.e. disease is inevitable)
Each letter represents a component cause for a disease
The component cause X is a necessary cause (i.e. disease cannot occur without it)
Accurate (i.e. low bias)
Descriptively realistic
Precise (i.e. low variability)
General
Robust
34
A model is robust if it is relatively immune to errors in the data and/or immune to small violations of model assumptions Is the model very sensitive to relatively small
changes in estimated input parameters? ▪ Model is NOT robust
Do model predictions remain accurate even when some key assumptions do not strictly hold?▪ Model IS robust
35
36
Poor Hygiene
Poo
r S
anit
atio
n
% d
isea
se a
ttri
buta
ble
to w
ater
No shedding of pathogens (contamination) into the water ( = 0)
Poor Hygiene
Poo
r S
anit
atio
n
% d
isea
se a
ttri
buta
ble
to w
ater
Moderate contamination ( = 1.0)
Poor Hygiene
Poo
r S
anit
atio
n
% d
isea
se a
ttri
buta
ble
to w
ater
Very high contamination ( = 2.0)
Accurate (i.e. low bias)
Descriptively realistic
Precise (i.e. low variability)
General
Robust
Simple / Parsimonious
40
A model is parsimonious if it can “accomplish a lot without much” E.g. a model that selects a relatively small
number of the most useful parameters
Simple isn’t always better
The research question should drive the complexity of the model
41
42r = 0.9972 Est. Temp = 155.3 + 1.90 * Pressure
195
200
205
210
215
Tem
p -
F
20 22 24 26 28 30Pressure - Inches of Hg
Boiling Point of Water vs Pressure
Source: Wikipedia
43
-1.5
-1-.
50
.5R
esid
ual
s
195 200 205 210 215Fitted values
Hmmm…
What does this mean?
Outlier!
Strong evidenceof non-linearity
44
. regress bpt pressure pressure2
Source | SS df MS Number of obs = 16-------------+------------------------------ F( 2, 13) =18942.88 Model | 527.718935 2 263.859468 Prob > F = 0.0000 Residual | .181079777 13 .013929214 R-squared = 0.9997-------------+------------------------------ Adj R-squared = 0.9996 Total | 527.900015 15 35.1933344 Root MSE = .11802
------------------------------------------------------------------------------ bpt | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- pressure | 3.754805 .2026619 18.53 0.000 3.31698 4.192629 pressure2 | -.0358576 .0039463 -9.09 0.000 -.044383 -.0273322 _cons | 131.7824 2.570509 51.27 0.000 126.2292 137.3357------------------------------------------------------------------------------
195
200
205
210
215
20 22 24 26 28 30Pressure
Est. Temp = 131.8 + 3.75 Pressure – 0.036 Pressure2
45
IN
SI
dt
dI vs.
46
Used as a model selection tool Penalizes models with excessive parameter
spaces AIC = 2k – 2ln(L) AICc = AIC + 2k(k+1) / (n – k – 1)
AICc is often used to avoid over-fitting when the sample size is small or the parameter space is large
Lower AICc more parsimonious model
47
“Best” model (#1)
“Mass action”Model (#5)
IN
SI
dt
dI
Accurate (i.e. low bias)
Descriptively realistic
Precise (i.e. low variability)
General
Robust
Simple / Parsimonious
Useful
48
A model is useful if: its conclusions are useful it points the way to other good models
E.g. Modeling HIV exercise▪ The early models weren’t necessarily
accurate but they were useful
49
= birth rate
N = S + I
= infection rate
I = Weibull mortality
S I I
N SI /N IS
0.0
0.2
0.4
0.6
0.8
1.0
0 10 20 30Time (years)
P(s
urv
ivin
g)
Normal (Weibull 2)
Exponential(Weibull 1)
Slide credit: J. Hargrove/B. Williams
0.0
0.2
0.4
0.6
0.8
1.0
1980 1990 2000 2010 2020Year
0.00
0.05
0.10
0.15
0.20
Pre
vale
nce
Inci
denc
e/m
orta
lity
Slide credit: J. Hargrove/B. Williams
0.0
0.2
0.4
0.6
0.8
1.0
1985 1990 1995 2000Year
Re
lativ
e t
ran
smis
sio
n
.
~
S I
I N SI /N I
S ~
= birth rate
N = population = C(t)
I = mortality
~
~
C(t)
Including controlSlide credit: J. Hargrove/B. Williams
0.0
0.1
0.2
0.3
0.4
1980 1990 2000 2010 2020Year
0.00
0.02
0.04
0.06
Pre
vale
nce
Inci
denc
e/m
orta
lity
Slide credit: J. Hargrove/B. Williams
Accurate (i.e. low bias)
Descriptively realistic
Precise (i.e. low variability)
General
Robust
Simple / Parsimonious
Useful
Inexpensive
Others???54
Accurate (i.e. low bias)
Descriptively realistic
Precise (i.e. low variability)
General
Robust
Simple / Parsimonious
Useful
Inexpensive
Others???55
It really depends on what your original research question was Was the goal to accurately predict
something? Was the goal to determine a relationship
between two or more parameters? Was the goal to understand a system in
general terms? Was the goal to test a hypothesis? Or to
generate one?56
Consider each of the models presented today What are the good
things about each model?
What are the shortcomings of each model?
Final word:
57Source: XKCD
Concepts of Mathematical Modeling, Walter Meyer, McGraw-Hill, 1984
Probability and Statistics, Charles Stone, Duxbury, 1996 Modeling Infectious Diseases in Humans and Animals, Keeling and
Rohoni, Princeton, 2008 Mathematical Models for Communicable Diseases, Brauer and
Castillo-Chavez, SIAM, 2013 The Analysis of Biological Data, Whitlock and Schluter, Roberts
and Company, 2008 Wikipedia XKCD
58