modeling the expert - ocw.mit.edu · 1 ask the experts! • critical decisions are often made by...
TRANSCRIPT
![Page 1: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/1.jpg)
MODELING THE EXPERT Image removed due to copyright restrictions
An Introduction to Logistic Regression
15.071 – The Analytics Edge
![Page 2: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/2.jpg)
Ask the Experts!
• Critical decisions are often made by people with expert knowledge
• Healthcare Quality Assessment • Good quality care educates patients and controls costs • Need to assess quality for proper medical interventions • No single set of guidelines for defining quality of
healthcare • Health professionals are experts in quality of care
assessment
15.071x –Modeling the Expert: An Introduction to Logistic Regression 1
![Page 3: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/3.jpg)
Experts are Human
• Experts are limited by memory and time
• Healthcare Quality Assessment • Expert physicians can evaluate quality by examining a
patient’s records • This process is time consuming and inefficient • Physicians cannot assess quality for millions of patients
15.071x –Modeling the Expert: An Introduction to Logistic Regression 2
![Page 4: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/4.jpg)
Replicating Expert Assessment
• Can we develop analytical tools that replicate expert assessment on a large scale?
• Learn from expert human judgment • Develop a model, interpret results, and adjust the model
• Make predictions/evaluations on a large scale
• Healthcare Quality Assessment • Let’s identify poor healthcare quality using analytics
15.071x –Modeling the Expert: An Introduction to Logistic Regression 3
![Page 5: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/5.jpg)
Claims Data
Medical Claims Diagnosis, Procedures,
Doctor/Hospital, Cost
Pharmacy Claims Drug, Quantity, Doctor,
Medication Cost
Pharmacy Claims
• Electronically available
• Standardized
• Not 100% accurate
• Under-reporting is common
• Claims for hospital visits can be vague
15.071x –Modeling the Expert: An Introduction to Logistic Regression
![Page 6: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/6.jpg)
Creating the Dataset – Claims Samples
Claims Sample • Large health insurance claims database
• Randomly selected 131 diabetes patients
• Ages range from 35 to 55 • Costs $10,000 – $20,000 • September 1, 2003 – August
31, 2005
15.071x –Modeling the Expert: An Introduction to Logistic Regression
![Page 7: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/7.jpg)
Creating the Dataset – Expert Review
Claims Sample
Expert Review
• Expert physician reviewed claims and wrote descriptive notes:
“Ongoing use of narcotics”
“Only on Avandia, not a good first choice drug”
“Had regular visits, mammogram, and immunizations”
“Was given home testing supplies”
15.071x –Modeling the Expert: An Introduction to Logistic Regression
![Page 8: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/8.jpg)
Creating the Dataset – Expert Assessment
Claims Sample
Expert Review
Expert Assessment
• Rated quality on a two-point scale (poor/good)
“I’d say care was poor – poorly treated diabetes”
“No eye care, but overall I’d say high quality”
15.071x –Modeling the Expert: An Introduction to Logistic Regression
![Page 9: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/9.jpg)
Creating the Dataset – Variable Extraction
Claims Sample
Expert Review
Expert Assessment
Variable Extraction
• Dependent Variable • Quality of care
• Independent Variables • ongoing use of narcotics • only on Avandia, not a good first
choice drug
• Had regular visits, mammogram, and immunizations
• Was given home testing supplies
15.071x –Modeling the Expert: An Introduction to Logistic Regression
![Page 10: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/10.jpg)
Creating the Dataset – Variable Extraction
Claims Sample
Expert Review
Expert Assessment
Variable Extraction
• Dependent Variable • Quality of care
• Independent Variables • Diabetes treatment • Patient demographics
• Healthcare utilization
• Providers • Claims
• Prescriptions
15.071x –Modeling the Expert: An Introduction to Logistic Regression
![Page 11: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/11.jpg)
Predicting Quality of Care
• The dependent variable is modeled as a binary variable • 1 if low-quality care, 0 if high-quality care
• This is a categorical variable • A small number of possible outcomes
• Linear regression would predict a continuous outcome
• How can we extend the idea of linear regression to situations where the outcome variable is categorical? • Only want to predict 1 or 0 • Could round outcome to 0 or 1 • But we can do better with logistic regression
15.071x –Modeling the Expert: An Introduction to Logistic Regression 10
![Page 12: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/12.jpg)
Formulas - Logistic Regression
p1, p2, . . . , pn
g1, g2, . . . , gm
⇧
⇧ (pi)
pi
⇧ (gj)
gj
⇧ (p1) , . . . ,⇧ (pn)
⇧ (g1) , . . . ,⇧ (gm)
L (⇧) = ⇧ (p1) . . .⇧ (pn) [1 ⇧ (g1)] . . . [1 ⇧ (gm)]
⇧ (p1) = 0.9,⇧ (p2) = 0.8,⇧ (q1) = 0.5
⇧ (p1) = ⇧ (p2) = ⇧ (q1) = 0.5
⇧ (x1, x2, . . . , xk) = logistic ( 0 + 1x1 + . . .+ kxk)
logistic (r) =
1
1 + exp ( r)
0, 1, . . . , k
L (⇧)
⇧ (Visits,Narcotics) = logistic ( 2.4 + 0.06⇥ Visits + 0.08⇥ Narcotics)
1
h
)
� �
�
�
Logistic Regression
• Predicts the probability of poor care • Denote dependent variable “PoorCare” by y • P (y = 1)
• Then P (y = 0) = 1 - P (y = 1)
• Independent variables x1, x2, . . . , xk
• Uses the Logistic Response Function 1
P (y = 1) = 1 + e -(/0+/1x1+/2x2+...+/k xk )
• Nonlinear transformation of linear regression equation to produce number between 0 and 1
15.071x –Modeling the Expert: An Introduction to Logistic Regression 11
![Page 13: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/13.jpg)
Understanding the Logistic Function
1 P (y = 1) =
1 + e -(/0+/1x1+/2x2+...+/k xk )
• Positive values are predictive of class 1
• Negative values are predictive of class 0 0.
0 0.
2 0.
4 0.
6 0.
8 1.
0 P
(y =
1)
-4 -2 0 2 4
15.071x –Modeling the Expert: An Introduction to Logistic Regression 12
![Page 14: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/14.jpg)
Understanding the Logistic Function
1 P (y = 1) =
1 + e -(/0+/1x1+/2x2+...+/k xk )
• The coefficients are selected to • Predict a high probability for the poor care cases • Predict a low probability for the good care cases
15.071x –Modeling the Expert: An Introduction to Logistic Regression 13
![Page 15: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/15.jpg)
Understanding the Logistic Function
1 P (y = 1) =
1 + e -(/0+/1x1+/2x2+...+/k xk )
• We can instead talk about Odds (like in gambling) P (y = 1)
Odds = P (y = 0)
• Odds > 1 if y = 1 is more likely • Odds < 1 if y = 0 is more likely
15.071x –Modeling the Expert: An Introduction to Logistic Regression 14
![Page 16: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/16.jpg)
The Logit
• It turns out that
log(Odds) = �0 + �1x1 + �2x2 + . . . + �kxk
• This is called the “Logit” and looks like linear regression
• The bigger the Logit is, the bigger P (y = 1)
15.071x –Modeling the Expert: An Introduction to Logistic Regression 15
![Page 17: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/17.jpg)
Model for Healthcare Quality
• Plot of the independent variables • Number of Office
Visits • Number of Narcotics
Prescribed
• Red are poor care • Green are good care
15.071x –Modeling the Expert: An Introduction to Logistic Regression 16
![Page 18: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/18.jpg)
Threshold Value
• The outcome of a logistic regression model is aprobability
• Often, we want to make a binary prediction • Did this patient receive poor care or good care?
• We can do this using a threshold value t
• If P(PoorCare = 1) ≥ t, predict poor quality • If P(PoorCare = 1) < t, predict good quality
• What value should we pick for t?
15.071x –Modeling the Expert: An Introduction to Logistic Regression 17
![Page 19: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/19.jpg)
Threshold Value
• Often selected based on which errors are “better”
• If t is large, predict poor care rarely (when P(y=1) is large) • More errors where we say good care, but it is actually poor care • Detects patients who are receiving the worst care
• If t is small, predict good care rarely (when P(y=1) is small) • More errors where we say poor care, but it is actually good care • Detects all patients who might be receiving poor care
• With no preference between the errors, select t = 0.5 • Predicts the more likely outcome
15.071x –Modeling the Expert: An Introduction to Logistic Regression 18
![Page 20: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/20.jpg)
Selecting a Threshold Value
Compare actual outcomes to predicted outcomes using a confusion matrix (classification matrix)
Predicted = 0 Predicted = 1
Actual = 0
Actual = 1
True Negatives (TN) False Positives (FP)
False Negatives (FN) True Positives (TP)
15.071x –Modeling the Expert: An Introduction to Logistic Regression 19
![Page 21: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/21.jpg)
tion of poor
ive rateity) on x-axis
tion of good
True
pos
itive
rate
0.2
0.4
0.6
0.8
Receiver Operator Characteristic (ROC) Curve
Receiver Operator Characteristic Curve• True positive rate
• Propor
•
(1-specific • Propor
(sensitivity) on y-axis
care caught
False posit
care labeled as poor 0.0
1.0
0.0 0.2 0.4 0.6 0.8 1.0
care False positive rate
15.071x –Modeling the Expert: An Introduction to Logistic Regression 20
![Page 22: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/22.jpg)
Selecting a Threshold using ROC
• Captures all thresholds simultaneously
• High threshold • High specificity • Low sensitivity
• Low Threshold • Low specificity • High sensitivity
True
pos
itive
rate
0.0
0.2
0.4
0.6
0.8
1.0
Receiver Operator Characteristic Curve
0.0 0.2 0.4 0.6 0.8 1.0
False positive rate
15.071x –Modeling the Expert: An Introduction to Logistic Regression 21
![Page 23: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/23.jpg)
Selecting a Threshold using ROC
Receiver Operator Characteristic Curve
• Choose best threshold for best trade off • cost of failing to
detect positives • costs of raising false
alarms Tr
ue p
ositi
ve ra
te0.
0 0.
2 0.
4 0.
6 0.
8 1.
0
0.0 0.2 0.4 0.6 0.8 1.0
False positive rate
15.071x –Modeling the Expert: An Introduction to Logistic Regression 22
![Page 24: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/24.jpg)
Selecting a Threshold using ROC
• Choose best threshold for best trade off • cost of failing to
detect positives • costs of raising false
alarms
Receiver Operator Characteristic Curve
True
pos
itive
rate
0.0
0.2
0.4
0.6
0.8
1.0
0.07
0.
25
0.44
0.
63
0.81
1
0.0 0.2 0.4 0.6 0.8 1.0
False positive rate
15.071x –Modeling the Expert: An Introduction to Logistic Regression 23
![Page 25: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/25.jpg)
Selecting a Threshold using ROC
• Choose best threshold for best trade off • cost of failing to
detect positives • costs of raising false
alarms
Receiver Operator Characteristic Curve
True
pos
itive
rate
0.0
0.2
0.4
0.6
0.8
1.0
0.07
0.25
0.44
0.63
0.81
1
00.1
0.2
0.30.40.50.6
0.7
0.80.9
1
0.0 0.2 0.4 0.6 0.8 1.0
False positive rate
15.071x –Modeling the Expert: An Introduction to Logistic Regression
![Page 26: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/26.jpg)
Interpreting the Model
• Multicollinearity could be a problem • Do the coefficients make sense? • Check correlations
• Measures of accuracy
15.071x –Modeling the Expert: An Introduction to Logistic Regression 25
![Page 27: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/27.jpg)
Compute Outcome Measures
Confusion Matrix:
Predicted Class = 0 Predicted Class = 1
Actual Class = 0
Actual Class = 1
True Negatives (TN) False Positives (FP)
False Negatives (FN) True Positives (TP)
N = number of observations
Overall accuracy = ( TN + TP )/N Overall error rate = ( FP + FN)/N
Sensitivity = TP/( TP + FN) False Negative Error Rate = FN/( TP + FN)
Specificity = TN/( TN + FP) False Positive Error Rate = FP/( TN + FP)
15.071x –Modeling the Expert: An Introduction to Logistic Regression 26
![Page 28: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/28.jpg)
Making Predictions
• Just like in linear regression, we want to make predictions on a test set to compute out-of-sample metrics
> predictTest = predict(QualityLog, type=“response”, newdata=qualityTest)
• This makes predictions for probabilities
• If we use a threshold value of 0.3, we get the following confusion matrix
Predicted Good Care Predicted Poor Care
Actually Good Care
Actually Poor Care
19 5
2 6
15.071x –Modeling the Expert: An Introduction to Logistic Regression 27
![Page 29: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/29.jpg)
AUC = 0.775
Area Under the ROC Curve (AUC)
• Just take the area under Receiver Operator Characteristic Curve
the curve • Interpretation
• Given a random positive and negative, proportion of the time you guess which is which correctly
• Less affected by sample balance than
0.0 0.2 0.4 0.6 0.8 1.0accuracy False positive rate
True
pos
itive
rate
0.0
0.2
0.4
0.6
0.8
1.0
15.071x –Modeling the Expert: An Introduction to Logistic Regression 28
![Page 30: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/30.jpg)
Area Under the ROC Curve (AUC)
• What is a good AUC? Receiver Operator Characteristic Curve
• Maximum of 1 (perfect prediction)
True
pos
itive
rate
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0
False positive rate
15.071x –Modeling the Expert: An Introduction to Logistic Regression 29
![Page 31: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/31.jpg)
Area Under the ROC Curve (AUC)
• What is a good AUC? Receiver Operator Characteristic Curve
• Maximum of 1 (perfect prediction)
• Minimum of 0.5 (just guessing)
True
pos
itive
rate
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0
False positive rate
15.071x –Modeling the Expert: An Introduction to Logistic Regression 30
![Page 32: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/32.jpg)
Conclusions
• An expert-trained model can accurately identify diabetics receiving low-quality care • Out-of-sample accuracy of 78% • Identifies most patients receiving poor care
• In practice, the probabilities returned by the logistic regression model can be used to prioritize patients for intervention
• Electronic medical records could be used in the future
15.071x –Modeling the Expert: An Introduction to Logistic Regression 31
![Page 33: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/33.jpg)
The Competitive Edge of Models
• While humans can accurately analyze small amounts of information, models allow larger scalability
• Models do not replace expert judgment • Experts can improve and refine the model
• Models can integrate assessments of many experts into one final unbiased and unemotional prediction
15.071x –Modeling the Expert: An Introduction to Logistic Regression 32
![Page 34: MODELING THE EXPERT - ocw.mit.edu · 1 Ask the Experts! • Critical decisions are often made by people with expert knowledge • Healthcare Quality Assessment • Good quality care](https://reader030.vdocuments.us/reader030/viewer/2022040704/5dd145e6d6be591ccb650cfe/html5/thumbnails/34.jpg)
MIT OpenCourseWare https://ocw.mit.edu/
15.071 Analytics Edge Spring 2017
For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.