important concepts and considerations in predictive modeling€¦ · important concepts and...
TRANSCRIPT
![Page 1: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/1.jpg)
Important concepts and considerations in predictive
modeling
Oscar Miranda-Domínguez, PhD, MSc.
Research Assistant Professor
Developmental Cognition and Neuroimaging Lab, OHSU
![Page 2: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/2.jpg)
Models try to identify associations between variables:
𝑋, predictor variables 𝑦, outcome variables
2
![Page 3: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/3.jpg)
Models in clinical research have specific problems:
Models in clinical research have specific problems:
- Limited samples- Multiple variables
- Thousands!- Unknown model structure
Entire population
3
![Page 4: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/4.jpg)
While it is easy to obtain models that can describe within-sample data…
Models in clinical research have specific problems:
- Limited samples- Multiple variables
- Thousands!- Unknown model structure
Entire population
4
![Page 5: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/5.jpg)
it is hard to obtain models that can predict outcome in out-of-sample data
Models in clinical research have specific problems:
- Limited samples- Multiple variables
- Thousands!- Unknown model structure
Entire population
5
![Page 6: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/6.jpg)
The question is why?
More importantly, what can be done to improve predictions across datasets?
6
![Page 7: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/7.jpg)
Topics
• Partial-least squares Regression• Feature Selection
• Cross-Validation
• Null Distribution/Permutations
• An Example
• Regularization• Truncated singular value decomposition
• Connectotyping: model based functional connectivity
• Example: models that generalize across datasets!
7
![Page 8: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/8.jpg)
Feature SelectionHow relevant is the balance between the number of variables and observations?
8
![Page 9: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/9.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
![Page 10: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/10.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴3.9 = 2.1𝐴
![Page 11: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/11.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴 → 𝐴 = 2.003.9 = 2.1𝐴 → 𝐴 ≈ 1.86
![Page 12: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/12.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴 → 𝐴 = 2.003.9 = 2.1𝐴 → 𝐴 ≈ 1.86
Select the solution with the lowest mean square error!
![Page 13: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/13.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴 → 𝐴 = 2.003.9 = 2.1𝐴 → 𝐴 ≈ 1.86
Select the solution with the lowest mean square error!
4.03.9
=2.02.1
𝐴
𝑦 = 𝑥𝐴
![Page 14: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/14.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴 → 𝐴 = 2.003.9 = 2.1𝐴 → 𝐴 ≈ 1.86
Select the solution with the lowest mean square error!
4.03.9
=2.02.1
𝐴
𝑦 = 𝑥𝐴
Using linear algebra (𝒙 pseudo-inverse)
𝐴 = 𝑥′𝑥 −1𝑥′𝑦
𝐴 ≈ 1.9286
This 𝑨 minimizes σ𝐫𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬𝟐
![Page 15: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/15.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴 → 𝐴 = 2.003.9 = 2.1𝐴 → 𝐴 ≈ 1.86
Select the solution with the lowest mean square error!
4.03.9
=2.02.1
𝐴
𝑦 = 𝑥𝐴
Using linear algebra (𝒙 pseudo-inverse)
𝐴 = 𝑥′𝑥 −1𝑥′𝑦
𝐴 ≈ 1.9286
This 𝑨 minimizes σ𝐫𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬𝟐
# Measurements < # Variables
What about (real) limited data:
8 = 4𝛼 + 𝛽
There are 2 variables (𝛼 and 𝛽) and 1 measurements.
![Page 16: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/16.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴 → 𝐴 = 2.003.9 = 2.1𝐴 → 𝐴 ≈ 1.86
Select the solution with the lowest mean square error!
4.03.9
=2.02.1
𝐴
𝑦 = 𝑥𝐴
Using linear algebra (𝒙 pseudo-inverse)
𝐴 = 𝑥′𝑥 −1𝑥′𝑦
𝐴 ≈ 1.9286
This 𝑨 minimizes σ𝐫𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬𝟐
# Measurements < # Variables
What about (real) limited data:
8 = 4𝛼 + 𝛽
There are 2 variables (𝛼 and 𝛽) and 1 measurements.
Solving the system:
8 − 4𝛼 = 𝛽
All the points on 𝛽 = 8 − 4𝛼 solve thesystem.
In other words, there is an infinite number of solutions!
![Page 17: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/17.jpg)
For predictive models it’s important to limit the number of features
relative to your sample size
![Page 18: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/18.jpg)
• This ‘feature reduction’ can be done in a number of ways.
• For partial least squares regression you reduce features based on how well models predict outcome.• What do I mean by that?
18
![Page 19: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/19.jpg)
Let’s revisit Principal Components AnalysisLet say you have a set of predictorvariables with some correlation
19
![Page 20: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/20.jpg)
If you define a new set of axis, you might have a better description of the system
20
![Page 21: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/21.jpg)
As most of the variance is observed across the black line, we can use it as a new base or axis
21
![Page 22: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/22.jpg)
You can add more axis to explain more varianceAdditional axis are selected to be perpendicular to each other (orthogonal)
22
![Page 23: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/23.jpg)
While useful, PCA does not take into account the outcome variable
23
![Page 24: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/24.jpg)
In partial least squares regression (PLSR) you add an extra constrain selecting a rotation that maximizes outcome prediction
24
![Page 25: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/25.jpg)
You can reduce the number of features by selecting different number of components (axis) and make predictions with those components
25
![Page 26: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/26.jpg)
Example
Let’s suppose we like to predict an outcome given 401 variables and 60 observations
26
![Page 27: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/27.jpg)
Observations
27
![Page 28: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/28.jpg)
Predictions using only one component
28
![Page 29: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/29.jpg)
Two components
29
![Page 30: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/30.jpg)
More components:- Low error- > likelihood of overfitting
30
![Page 31: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/31.jpg)
For partial least squares regression, within sample tests
can lead to over fitting
![Page 32: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/32.jpg)
The question is, how many components do we need for a generalizable model?
32
![Page 33: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/33.jpg)
How do we avoid over fitting with cross validation?
33
![Page 34: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/34.jpg)
Cross-Validation
Definition: Using different samples to model and predict
- hold-out: you use the unique dataset you have to make random partitions, one to model and the other to predict
Other forms of out of sample sampling
- Bootstrapping : random sampling with replacement
34
![Page 35: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/35.jpg)
Let’s use an example to illustrate the problem of overfitting and how hold-out cross validation can
minimize it
35
![Page 36: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/36.jpg)
Imagine an “executive functioning” score is related to mean functional connectivity
The modeler does not know the model structure but it is given by a third order polynomial:
𝑥 = mean fconn between the Fronto-parietal and default networksscore= 𝑝0 + 𝑝1𝑥 + 𝑝2𝑥
2 + 𝑝3𝑥3
36
![Page 37: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/37.jpg)
Data was measured on multiple participants
· Unique
participant
Noiseless data
37
![Page 38: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/38.jpg)
However, data was collected on two sites
Noiseless data
38
![Page 39: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/39.jpg)
and each site has a different scanner’s noise profile,
Noiseless data
fconn’s noise
39
![Page 40: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/40.jpg)
which leads to significant batch effects.
fconn’s noise
=+
Measured dataNoiseless data
40
![Page 41: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/41.jpg)
We, however, only have access to OHSU data.
Measured data
41
![Page 42: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/42.jpg)
Modeling approach
• Predict executive functioning score based on mean fconn using polynomials of different order• Starting from simplest to more
complex models
• Estimate “goodness of the fit” (mean square errors in predictions)
• Select the model with the “best fit” i.e., lowest error
42
![Page 43: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/43.jpg)
Mean Square Error
OHSU
1 22.35
2 21.22
3 16.21
4 15.61
5 14.14
6 14.13
Polynomial
order
First order
43
![Page 44: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/44.jpg)
Second order
Mean Square Error
OHSU
1 22.35
2 21.22
3 16.21
4 15.61
5 14.14
6 14.13
Polynomial
order
44
![Page 45: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/45.jpg)
Third order
Mean Square Error
OHSU
1 22.35
2 21.22
3 16.21
4 15.61
5 14.14
6 14.13
Polynomial
order
45
![Page 46: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/46.jpg)
Fourth order
Mean Square Error
OHSU
1 22.35
2 21.22
3 16.21
4 15.61
5 14.14
6 14.13
Polynomial
order
46
![Page 47: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/47.jpg)
Fifth order
Mean Square Error
OHSU
1 22.35
2 21.22
3 16.21
4 15.61
5 14.14
6 14.13
Polynomial
order
47
![Page 48: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/48.jpg)
Sixth order
Mean Square Error
OHSU
1 22.35
2 21.22
3 16.21
4 15.61
5 14.14
6 14.13
Polynomial
order
48
![Page 49: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/49.jpg)
Fifth order seems to be the best fit
Mean Square Error
OHSU
1 22.35
2 21.22
3 16.21
4 15.61
5 14.14
6 14.13
Polynomial
order
49
![Page 50: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/50.jpg)
Let’s use OHSU’s models on Minn’s data
50
![Page 51: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/51.jpg)
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
First order
51
![Page 52: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/52.jpg)
Second order
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
52
![Page 53: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/53.jpg)
Third order
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
53
![Page 54: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/54.jpg)
Third order
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
54
![Page 55: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/55.jpg)
Fourth order
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
55
![Page 56: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/56.jpg)
Fifth order
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
56
![Page 57: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/57.jpg)
Sixth order
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
57
![Page 58: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/58.jpg)
Take-home message
Testing performance on the same data used to obtain a model leads to overfitting. Do not do it.
58
![Page 59: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/59.jpg)
How to know that the best model is a third order polynomial?
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
59
![Page 60: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/60.jpg)
How to know that the best model is a third order polynomial?
Use hold-out cross-validation!
OHSU Minn
1 22.35 23.16
2 21.22 23.27
3 16.21 39.03
4 15.61 36.77
5 14.14 44.55
6 14.13 49.96
Polynomial
order
Mean Square Error
60
![Page 61: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/61.jpg)
Let’s use hold-out cross-validation to fit the most generalizable model for this data set
61
![Page 62: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/62.jpg)
Make two partitions: Let’s use 90% of the sample for modeling and hold 10% out for testing
62
![Page 63: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/63.jpg)
Use the partition modeling to fit the simplest model. Then predict in-sample and out-sample data
A reasonable cost function is the mean of the sum of squares’s residuals
63
![Page 64: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/64.jpg)
Resample and repeat
Keep track of the errors.
64
![Page 65: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/65.jpg)
Repeat N times
65
![Page 66: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/66.jpg)
Increase model complexity,
Increase order complexity
Keep track of the errors.
66
![Page 67: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/67.jpg)
Third order
67
![Page 68: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/68.jpg)
Fourth order
68
![Page 69: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/69.jpg)
Visualize results
Pick the best (lowest out-of-sample prediction)
Notice how the in-sample (modeling) error decreases as order increases: OVERFITTING
69
![Page 70: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/70.jpg)
Take-home message
Cross-validation is a useful tool towards predictive modeling.
Partial-least squares regression requires cross-validation for predictive modeling to avoid overfitting
70
![Page 71: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/71.jpg)
Generating Null hypothesis dataWhy is it important to generate a null distribution?
71
![Page 72: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/72.jpg)
How do you know that your model behaves better than chance?
• What is chance in the context of modeling and hold-out cross-validation?
72
![Page 73: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/73.jpg)
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
Let’s suppose this is your data
Original data
73
![Page 74: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/74.jpg)
Make two random partitions: modeling and validation
Original data Modeling Validation
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
74
![Page 75: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/75.jpg)
Randomize predictor and outcomes in the partition used for modeling
Original data Modeling Validation
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 77−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 201𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 217𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
75
![Page 76: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/76.jpg)
Estimate out-of-sample performance:
Original data Modeling Validation
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 77−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 201𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 217𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
- Calculate the model in the partition “Modeling”
- Predict outcome on the partition “Validation”
- Estimate “goodness of the fit”: mean square error
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
76
![Page 77: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/77.jpg)
Repeat and keep track of the errors
Original data Modeling Validation
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 622𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 197𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 20
9𝑥1 − 7𝑥2 +⋯− 4𝑥𝑛 = 21−𝑥1 + 9𝑥2 +⋯+ 2𝑥𝑛 = 192𝑥1 + 7𝑥2 +⋯+ 2𝑥𝑛 = 771𝑥1 − 6𝑥2 +⋯+ 1𝑥𝑛 = 207𝑥1 − 2𝑥2 +⋯− 9𝑥𝑛 = 62
- Calculate the model in the partition “Modeling”
- Predict outcome on the partition “Validation”
- Estimate “goodness of the fit”: mean square error
77
![Page 78: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/78.jpg)
Compare performance (mean squares error in out-of-sample data) to determine if your model predicts better than chance!
Mean Square Errors
78
![Page 79: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/79.jpg)
Example using Neuroimaging datacross-validation, regularization and PLSRfconn_regression tool
79
![Page 80: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/80.jpg)
I’ll use as a case the study of cueing in freezing of gait in Parkinson’s disease
http://parkinsonteam.blogspot.com/2011/10/prevencion-de-caidas-en-personas-con.html
https://en.wikipedia.org/wiki/Parkinson's_disease
Freezing of gait, a pretty descriptive name, is an additional symptom present on some patients
Freezing can lead to falls, which adds an extra burden in Parkinson’s disease
80
![Page 81: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/81.jpg)
Auditory cues, like beats at a constant rate, are an effective intervention to reduce freezing episodes in some patients
Open loop
Ashoori A, Eagleman DM, Jankovic J. Effects of Auditory Rhythm and Music on Gait Disturbances in Parkinson’s Disease [Internet]. Front Neurol 2015;
81
![Page 82: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/82.jpg)
The goal of the study is to determine whether improvement after cueing can be predicted by resting state functional connectivity
82
![Page 83: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/83.jpg)
Available data
Resting state functional MRI
83
![Page 84: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/84.jpg)
Approach
1. Calculate rs-fconn• Group data per functional network pairs: Default-Default, Default-Visual, …
2. Use PLSR and cross-validation to determine whether improvement can be predicted using connectivity from specific brain networks
3. Explore outputs
4. Report findings
84
![Page 85: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/85.jpg)
First step is to calculate resting state functional connectivity and group data per functional system pairs
85
![Page 86: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/86.jpg)
PLSR and cross-validation
Parameters
• Partition size• Hold-one out• Hold-three out
• How many components:• 2, 3, 4,…
• Number of repetitions• 100?, 500?,…
• Calculate null-hypothesis data • Number of repetitions: 10,000?
This can be done using the tool fconn_regression
86
![Page 87: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/87.jpg)
Comparing distribution of prediction errors for real versus null-hypotheses dataSorted by Cohen effect size
Visual and subcortical
Effect size = 0.87
Auditory and default
Effect size = 0.81
Somatosensory lateral and Ventral attention
Effect size = 0.78
Visual Auditory
DefaultSubcortical
Ventral Attn
Somatosensory lateral
Mean square error Mean square error Mean square error 87
![Page 88: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/88.jpg)
We have a virtual machine and a working example
Let us know if you are interested in a break-out session
88
![Page 89: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/89.jpg)
Topics
• Partial-least squares Regression• Feature Selection
• Cross-Validation
• Null Distribution/Permutations
• An Example
• Regularization• Truncated singular value decomposition
• Connectotyping: model based functional connectivity
• Example: models that generalize across datasets!
89
![Page 90: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/90.jpg)
RegularizationTruncated singular value decomposition
90
![Page 91: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/91.jpg)
# Measurements = # Variables
The system
4 = 2𝐴
has a unique solution
𝐴 = 2
# Measurements > # Variables
What about repeated measurements (real data with noise)
4.0 = 2.0𝐴 → 𝐴 = 2.003.9 = 2.1𝐴 → 𝐴 ≈ 1.86
Select the solution with the lowest mean square error!
4.03.9
=2.02.1
𝐴
𝑦 = 𝑥𝐴
Using linear algebra (𝒙 pseudo-inverse)
𝐴 = 𝑥′𝑥 −1𝑥′𝑦
𝐴 ≈ 1.9286
This 𝑨 minimizes σ𝐫𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬𝟐
# Measurements < # Variables
What about (real) limited data:
8 = 4𝛼 + 𝛽
There are 2 variables (𝛼 and 𝛽) and 1 measurements.
Solving the system:
8 − 4𝛼 = 𝛽
All the points on 𝛽 = 8 − 4𝛼 solve thesystem.
In other words, there is an infinite number of solutions!
![Page 92: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/92.jpg)
What if you can’t reduce the number of features?
Regularization is a powerful approach to handle this kind of problems (ill-posed systems)
92
![Page 93: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/93.jpg)
We know that the pseudo-inverse offers the optimal solution (lowest least squares) for systems with more measurements than observations
93
![Page 94: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/94.jpg)
We can use the pseudo-inverse to calculate a solution in systems with more measurements than observations
94
![Page 95: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/95.jpg)
Example: Imagine a given outcome can be predicted by 379 variables,…
𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥3791)
95
![Page 96: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/96.jpg)
And that you have 163 observations:
𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
…𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
1)
2)
3)
163)
96
![Page 97: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/97.jpg)
Using the pseudo-inverse you can obtain a solution with high predictability
97
![Page 98: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/98.jpg)
Using the pseudo-inverse you can obtain a solution with high predictability
This solution, however, is problematic:
*unstable beta weights*over fitting*not applicable to outside dataset
98
![Page 99: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/99.jpg)
What does “unstable beta weights” mean?
Let’s suppose age and weight are two variables used in your model
For one participant you used
• Age: 10.0 years
• Weight: 70 pounds
• Corresponding outcome: “score” of 3.7
There was, however, an error in data collection and the real values are:
• Age: 10.5 years
• Weight: 71 pounds
99
![Page 100: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/100.jpg)
Updating predictions in the same model
Let’s suppose age and weight are two variables used in your model
For one participant you used
• Age: 10.0 years
• Weight: 70 pounds
• Corresponding outcome: “score” of 3.7
There was, however, an error in data collection and the real values are:
• Age: 10.5 years
• Weight: 71 pounds
Stable beta-weights:
score ~ 3.9
Unstable beta weights:
score ~ -344,587.42
100
![Page 101: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/101.jpg)
What is the best solutions for the system?
𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
…𝑦 = 𝛽1𝑥1 + 𝛽2𝑥2 +⋯𝛽379𝑥379
1)
2)
3)
163)
𝑦 = 𝑋𝛽
101
![Page 102: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/102.jpg)
Remember the PCA section?
We said that we can rotate X (the data) to find optimal projections
We can use different number of axis
Adding more axis leads to:• More explained variance• More over-fitting
102
![Page 103: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/103.jpg)
In truncated singular value decomposition, we follow a similar approach
• Decompose X in such a way that we can explore effect of inclusion/exclusion of components (singular value decomposition)
• Make a new X truncating some components
• Solve the system plugging 𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑into the pseudo-inverse
• Select the optimal number of components
𝑋 = 𝑈Σ𝑉𝑇
𝛴 =𝜎1 ⋯ 0⋮ ⋱ 00 ⋯ 𝜎𝑀
,
𝜎1 ≥ 𝜎2 ≥ ⋯ ≥ 𝜎𝑀 ≥ 0.
The smaller singular values of 𝑋 are more unstable (susceptible to
noise)
103
![Page 104: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/104.jpg)
In truncated singular value decomposition, we follow a similar approach
• Decompose X in such a way that we can explore effect of inclusion/exclusion of components (singular value decomposition)
• Make a new X truncating some components
• Solve the system plugging 𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑into the pseudo-inverse
• Select the optimal number of components
𝑋 = 𝑈Σ𝑉𝑇
𝛴𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑 =𝜎1 ⋯ 0⋮ ⋱ 00 ⋯ 0
,
𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑 = 𝑈Σ𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑𝑉𝑇
104
![Page 105: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/105.jpg)
In truncated singular value decomposition, we follow a similar approach
• Decompose X in such a way that we can explore effect of inclusion/exclusion of components (singular value decomposition)
• Make a new X truncating some components
• Solve the system plugging 𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑into the pseudo-inverse
• Select the optimal number of components
𝛽 = 𝑋′𝑋 −1𝑋′𝑦
Pseudo-inverse
𝛽𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑 = 𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑′𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑
−1𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑′𝑦
105
![Page 106: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/106.jpg)
In truncated singular value decomposition, we follow a similar approach
• Decompose X in such a way that we can explore effect of inclusion/exclusion of components (singular value decomposition)
• Make a new X truncating some components
• Solve the system plugging 𝑋𝑡𝑟𝑢𝑛𝑐𝑎𝑡𝑒𝑑into the pseudo-inverse
• Select the optimal number of components
AccuracyNorm of the residuals
?
106
![Page 107: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/107.jpg)
Unstable Pseudo-inverse solution
Let’s get back to our example:379 variables and 163 observations
107
![Page 108: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/108.jpg)
Solving the system preserving only the largest singular value
AccuracyNorm of the residuals
108
![Page 109: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/109.jpg)
Preserving two singular values
AccuracyNorm of the residuals
109
![Page 110: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/110.jpg)
Keeping 3
AccuracyNorm of the residuals
110
![Page 111: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/111.jpg)
All minus one
AccuracyNorm of the residuals
111
![Page 112: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/112.jpg)
Keeping all
AccuracyNorm of the residuals
112
![Page 113: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/113.jpg)
You can select the “optimal” number of components using cross-validation and maximizing predictions of out-of-sample data
AccuracyNorm of the residuals
Use tsvd and cross-validation
*more stable beta weights*less over fitting*applicable to outside dataset
?
113
![Page 114: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/114.jpg)
Section’s summary
• Testing performance on the same data used to obtain a model leads to overfitting. Do not do it. Use cross-validation instead.
• Modeling is hard, especially when the number of “unknowns” exceeds the number of measurements: “ill-posed” systems
• These types of problems are common on neuroimaging projects
• Regularization and cross-validation can minimize the risk of overfitting and lead to better out-of-sample performance
114
![Page 115: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/115.jpg)
Towards estimates of functional connectivity that generalize across datasetsCorrelations might not be enough with limited data (~5 mins)
115
![Page 116: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/116.jpg)
Connectotyping
The activity of each brain region can be predicted by the weighted contribution of all the other brain regions
Ƹ𝑟1
Ƹ𝑟2Ƹ𝑟3
116
![Page 117: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/117.jpg)
How can we make an educated guess of “blue” given “red” and “green”
Ƹ𝑟1
Ƹ𝑟2Ƹ𝑟3
117
![Page 118: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/118.jpg)
We can combine them linearly and estimate the beta weights
β1,2
β1,3
Ƹ𝑟1
Ƹ𝑟2Ƹ𝑟3
118
![Page 119: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/119.jpg)
And formulate this mathematically
β1,2
β1,3
Ƹ𝑟1 = 𝟎 𝑟1 + β1,2 𝑟2 + β1,3 𝑟3
119
![Page 120: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/120.jpg)
Notice that blue does not depend on blue
β1,2
β1,3
Ƹ𝑟1 = 𝟎 𝑟1 + β1,2 𝑟2 + β1,3 𝑟3
120
![Page 121: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/121.jpg)
Repeat approach for red
β2,1
β2,3
Ƹ𝑟2 = β2,1 𝑟1 + 0 𝑟2+ β2,3 𝑟3
Red does not depend on red
121
![Page 122: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/122.jpg)
And green
β3,1
β3,2
Ƹ𝑟3 = β3,1 𝑟1 + β3,2 𝑟2+ 𝟎 𝑟3
Green does not depend on green
122
![Page 123: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/123.jpg)
Which can be represented as a 3x3 matrix
Ƹ𝑟1Ƹ𝑟2Ƹ𝑟3
=
𝟎 β1,2 β1,3β2,1 𝟎 β2,3β3,1 β3,2 𝟎
𝑟1𝑟2𝑟3
Matricial form
Ƹ𝑟1 = 0 𝑟1 + β1,2 𝑟2 + β1,3 𝑟3
Ƹ𝑟2 = β2,1 𝑟1 + 0 𝑟2+ β2,3 𝑟3
Ƹ𝑟3 = β3,1 𝑟1 + β3,2 𝑟2+ 0 𝑟3
123
![Page 124: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/124.jpg)
General case (“M” instead of 3 ROIs):A bigger matrix
General case
Ƹ𝑟1Ƹ𝑟2⋮Ƹ𝑟𝑀
=
0 β1,2β2,1 0
… β1,𝑀… β2,𝑀
⋮ ⋮β𝑀,1 β𝑀,2
⋱ ⋮… 0
𝑟1𝑟2⋮𝑟𝑀
Ill-posed system (more unknowns that data)
Solved by regularization and cross validation
124
![Page 125: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/125.jpg)
And the solution is an individualizedconnectivity matrix
Solution!
General case
Ƹ𝑟1Ƹ𝑟2⋮Ƹ𝑟𝑀
=
0 β1,2β2,1 0
… β1,𝑀… β2,𝑀
⋮ ⋮β𝑀,1 β𝑀,2
⋱ ⋮… 0
𝑟1𝑟2⋮𝑟𝑀
125
![Page 126: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/126.jpg)
Connectivity matrices (models)can be compared
Ƹ𝑟1Ƹ𝑟2⋮Ƹ𝑟𝑀
=
0 β1,2β2,1 0
… β1,𝑀… β2,𝑀
⋮ ⋮β𝑀,1 β𝑀,2
⋱ ⋮… 0
𝑟1𝑟2⋮𝑟𝑀
Ƹ𝑟1Ƹ𝑟2⋮Ƹ𝑟𝑀
=
0 β1,2β2,1 0
… β1,𝑀… β2,𝑀
⋮ ⋮β𝑀,1 β𝑀,2
⋱ ⋮… 0
𝑟1𝑟2⋮𝑟𝑀
Ƹ𝑟1Ƹ𝑟2⋮Ƹ𝑟𝑀
=
0 β1,2β2,1 0
… β1,𝑀… β2,𝑀
⋮ ⋮β𝑀,1 β𝑀,2
⋱ ⋮… 0
𝑟1𝑟2⋮𝑟𝑀
Subject 1
Subject 2
Subject 3
126
![Page 127: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/127.jpg)
- models can also predict brain activity
127
![Page 128: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/128.jpg)
To predict brain activity- Start with the original fMRI data (after cleaning)
128
![Page 129: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/129.jpg)
Fresh data
Modeling
Modeling
Fresh data
Next, split the data randomly in 2 sections:One for modeling, the other for prediction
129
![Page 130: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/130.jpg)
Use the section modeling for connectotypingCalculate the beta weights (connectivity matrix)!
Fresh data
Modeling
ConnectotypeƸ𝑟1Ƹ𝑟2Ƹ𝑟3
=
𝟎 β1,2 β1,3β2,1 𝟎 β2,3β3,1 β3,2 𝟎
𝑟1𝑟2𝑟3
130
![Page 131: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/131.jpg)
Use the matrix to predictbrain activity in fresh data
Modeling
Connectotype
Predicted dataFresh data
131
![Page 132: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/132.jpg)
Compare fresh data with predicted dataYou may use correlation coefficients!
Modeling
Connectotype
Predicted dataFresh data
R1
R2
R3
ഥ𝑹
Compare fresh vs predicted data
132
![Page 133: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/133.jpg)
ValidationData sets
HUMANS:
• 27 healthy adult humans (16 females) age 19 to 35 years
• Subset scanned a second timetwo weeks later
(Validated in data from 11 macaques too)
133
![Page 134: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/134.jpg)
ValidationStep 1
Approach:
1. A model was calculated for each participant using partial data
2. Each model was used to predict fresh data for each scan
3. Correlation between predicted and observed timecourses were calculated
134
![Page 135: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/135.jpg)
ValidationStep 2
Approach:
1. A model was calculated for each participant using partial data
2. Each model was used to predict fresh data for each scan
3. Correlation between predicted and observed timecourses were calculated
135
![Page 136: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/136.jpg)
ValidationStep 3
Approach:
1. A model was calculated for each participant using partial data
2. Each model was used to predict fresh data for each scan
3. Average correlation between predicted and observed timecourses was calculated
136
![Page 137: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/137.jpg)
When the model and fresh data came from the same participants, ഥ𝑹 ≈ 𝟎. 𝟖𝟕
Fresh dataBaseline
Sub
ject
137Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 138: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/138.jpg)
When the model and fresh data came from different participants, ഥ𝑹 ≈ 𝟎. 𝟔𝟒
Fresh dataBaseline
Sub
ject
138Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 139: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/139.jpg)
Notice that by looking at a single number (ഥ𝑹)we can characterize individuals, since there was no overlap in predicting self versus others
Fresh dataBaseline
Sub
ject
0.6 0.7 0.8 0.9Correlations
139Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 140: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/140.jpg)
As further validation, we predicted fresh data acquired 2 weeks later, finding the same trend:
Fresh dataBaseline
Sub
ject
0.6 0.7 0.8
Correlations
Accurate characterization
of individuals
shared variance
140Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 141: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/141.jpg)
Same trend is also observed in macaquesഥ𝑹 are reduced
Fresh dataBaseline
Sub
ject
0.2 0.4 0.6Correlations
Accurate characterization
of individuals
shared variance
141Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 142: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/142.jpg)
These findings suggest that
We are all equipped with functional networks that process certain stimuli in the same way
… on top of this…
we all each have unique salient functional networks that make us unique
0.6 0.7 0.8 0.9Correlations
shared variance
142Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 143: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/143.jpg)
These findings suggest that
We are all equipped with functional networks that process certain stimuli in the same way
… on top of this…
we all each have unique salient functional networks that make us unique
0.6 0.7 0.8 0.9Correlations
Accurate characterization
of individuals
143Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 144: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/144.jpg)
So, the next question is“What brain systems make a connectome unique”
144
![Page 145: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/145.jpg)
To do this, we look at how similar or different the models were across participants
Variance Across Subjects
Subjects
145Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 146: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/146.jpg)
Fronto-parietal cortexmakes a connectome unique
More individual
More conserved
146Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 147: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/147.jpg)
In contrast, notice how similar motor systems are across individuals
More individual
More conserved
147Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 148: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/148.jpg)
How much data is needed to connectotype?
148
![Page 149: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/149.jpg)
2.5 minutes of data is enough to connectotype!
• Self vs others experiment was repeated using different amounts of data
• 2.5 minutes of data is enough to connectotype!
Time2.5 minutes
149Miranda-Dominguez O, et al.. PLoS One. 2014
![Page 150: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/150.jpg)
In summary, connectotyping
• Identifies connectivity patterns unique to individuals
• The connectotype is robust in adults and can be obtained with limited amounts of data
• fronto-parietal systems are highly variable amongst individuals.
150
![Page 151: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/151.jpg)
Can we use connectotyping in youth?
151
![Page 152: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/152.jpg)
Participants
Controls passing QC:• N=188 scans (159 subjects)
• 131 subjects with 1 scan• 27 subjects with 2 scans• 1 subjects with 3 scans
• Age: 7-15
• 60% males
• Siblings (16 pairs)• 16 families with 2 siblings each
“Gordon” parcellation schema
152
Gordon et al, Cerebral Cortex, 2014
![Page 153: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/153.jpg)
Connectotyping in youth Step 1
Approach:
1. A model was calculated for each scan (N=188)
2. Each model was used to predict fresh data for each scan (N=188)
3. Average correlation between predicted and observed timecourses were calculated (N = 188 x 188)
4. Average correlations were grouped based on the datasets used for modeling and prediction
153
![Page 154: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/154.jpg)
Connectotyping in youthStep 2
Approach:
1. A model was calculated for each scan (N=188)
2. Each model was used to predict fresh data for each scan (N= 188 x 188 x ROIs)
3. Average correlation between predicted and observed timecourses were calculated (N = 188 x 188)
4. Average correlations were grouped based on the datasets used for modeling and prediction
154
![Page 155: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/155.jpg)
Connectotyping in youthStep 3
Approach:
1. A model was calculated for each scan (N=188)
2. Each model was used to predict fresh data for each scan (N= 188 x 188 x ROIs)
3. Average correlation between predicted and observed timecourses were calculated (N = 188 x 188)
4. Average correlations are grouped based on the datasets used for modeling and prediction
155
![Page 156: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/156.jpg)
Connectotyping in youthStep 4
Approach:
1. A model was calculated for each scan (N=188)
2. Each model was used to predict fresh data for each scan (N=188 x 188 x ROIs)
3. Average correlation between predicted and observed timecourses were calculated (N = 188 x 188)
4. Average correlations were grouped based on the datasets used for modeling and prediction
I. Same scanII. Same participantIII. SiblingIV. Unrelated
156
![Page 157: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/157.jpg)
Connectotyping in youth Predicting time courses
Same scan (N=188)
157
![Page 158: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/158.jpg)
Predicting fresh data from the same scan
Same scan (N=188)
Distributions of correlations (per group)
0.25 1.00
Average correlations
0.50 0.75
158Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 159: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/159.jpg)
Predicting data from the same participant acquired 1 or 2 years later
1 or 2 years later
Same scan (N=188) Same
participant(N=60)
Difference in years when data was acquired
Distributions of correlations (per group)
0.25 1.00
Average correlations
0.50 0.75
159Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 160: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/160.jpg)
Predicting timecourses amongst siblings
Same scan (N=188)
Siblings(N=46)
Same participant(N=60)
1 or 2 years later
Difference in years when data was acquired
Distributions of correlations (per group)
0.25 1.00
Average correlations
0.50 0.75
160Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 161: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/161.jpg)
Predicting timecourses amongst unrelated
Same scan (N=188)
Siblings(N=46)
Same participant(N=60)
Unrelated(N=35,050)
1 or 2 years later
Difference in years when data was acquired
Distributions of correlations (per group)
0.25 1.00
Average correlations
0.50 0.75
161Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 162: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/162.jpg)
Characterization of individuals are stable(at least over a period of 2 years)
Same participant(N=60)
Unrelated(N=35,050)
1 or 2 years later
Difference in years when data was acquired
Distributions of correlations (per group)
0.25 1.00
Average correlations
0.50 0.75
Same scan (N=188)
162Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 163: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/163.jpg)
Siblings cluster together higher than unrelated
Siblings(N=46)
Unrelated(N=35,050)
Difference in years when data was acquired
Distributions of correlations (per group)
0.25 1.00
Average correlations
0.50 0.75
163Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 164: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/164.jpg)
These findings suggest that
The connectotype is similarly predictive in children as shown in adults, across a wider timespan, and some features appear to be familial
164
![Page 165: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/165.jpg)
What if we now use multivariate statistics (instead of using the average correlation) to compare connectomes?
165
![Page 166: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/166.jpg)
Can we identify heritable patterns of functional connectivity?
• Some mental disorders run strongly among families
• It might be useful to identify what is the “baseline” shared connectome across siblings?
166
![Page 167: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/167.jpg)
There is evidence of similarthoughts among siblings
http://edition.cnn.com/2015/09/06/tennis/tennis-venus-serena-bouchard/http://www.tampabay.com/news/politics/national/bush-dynasty-continues-to-impact-republican-politics/1248057
167
![Page 168: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/168.jpg)
Datasets
OHSU Human Connectome Project
Data from 198 unique participants
1 hour of data each
22-36 yo, 45% males
79 pairs of siblings:
• 10 identical twins
• 11 non-identical twins
• 58 sibling non-twins
Data from 32 unique participants
5 mins of low-head movement of RS
7-15 yo, 60% males
Siblings (16 pairs)
16 families with 2 siblings each
168
![Page 169: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/169.jpg)
Approach
Within dataset• Calculate functional connectivity
• Connectotyping• Correlations
• Compared each participant pair• Connectotyping: predicting timecourses• Correlations: spatial correlations
• Train classifiers (SVM) to identify each pair of participants as siblings or unrelated
Between datasets
• Test classifiers’ performance across datasets
169
![Page 170: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/170.jpg)
Within OHSU resultsOut-of-sample performance
170
Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 171: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/171.jpg)
Within HCP resultsOut-of-sample performance
171
Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 172: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/172.jpg)
Within HCP resultsOut-of-sample performance
172
Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 173: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/173.jpg)
Predictions across datasetsOnly connectotyping was able to predict kinship
173Miranda-Domínguez O, et al. Heritability of the human connectome: A connectotyping study. Netw Neurosci 2018.
![Page 174: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/174.jpg)
174
![Page 175: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/175.jpg)
Rules of thumb
• In selecting predictor variables• Make sure predictor variables are related to outcome
• Try to select variables with the lowest redundancy
• It is better to have more observations than variables
• Regardless of modeling framework, you should use• Cross-validation to have an estimate of out-of-sample performance
• Regularization to obtain more stable beta weights
• Test performance on null data, to determine whether your models predict better than chance
175
![Page 176: Important concepts and considerations in predictive modeling€¦ · Important concepts and considerations in predictive modeling Oscar Miranda-Domínguez, PhD, MSc. Research Assistant](https://reader033.vdocuments.us/reader033/viewer/2022060509/5f2560b8597c1d0c2d355273/html5/thumbnails/176.jpg)
Acknowledgements
176
Members of the DCAN Lab
Funding: Parkinson’s Center of Oregon Pilot Grant, OHSU Fellowship for Diversity, Tartar Family grant, NIMH
AJ Mitchell Alice Graham Alina Goncharova Anders Perrone Anita Randolph Anjanibhargavi Ragothaman Anthony Galassi Bene Ramirez Binyam Nardos Damien Fair Elina Thomas Eric Earl
Eric Feczko Greg Conan Johnny Uriarte-Lopez Kathy Snider Lisa Karstens Lucille Moore Michaela Cordova Mollie Marr Olivia Doyle Robert Hermosillo Samantha Papadakis Thomas Madison
DCAN Lab