intensive longitudinal data analysis and visual data ... · intensive longitudinal data analysis...
TRANSCRIPT
Intensive Longitudinal Data Analysis and Visual
Data Analytics of Student, Teacher & Leader
Data for Decision Making in Schools
Alex J. Bowers, Ph.D.
Associate Professor of Education Leadership
Teachers College, Columbia University
February 12, 2014
Overview of the Talk
Patterning Student Data to Predict Outcomes & Inform Interventions:
• What are the most accurate “dropout flags”? • Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out? A Review of
the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity. The
High School Journal.96(2), 77-100.
• Are there significantly different trajectories of student grade patterns
that predict high school drop out? • Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories Associated with
Dropping Out of High School: A Growth Mixture Model Analysis. The Journal of Educational
Research, 105(3), 176-195.
• How early can we detect these patterns? • Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of
Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster
Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18.
Understanding Leadership Effects through Subgroup Trajectory Analysis:
• Does principal training influence school achievement trajectories? • Bowers, A.J., White, B.R. (in press) Do Principal Preparation and Teacher Qualifications
Influence Different Types of School Growth Trajectories in Illinois? A Growth Mixture Model
Analysis.Journal of Educational Administration.
• Are there different types of school leadership? • Bowers, A.J., Blitz, M., Halverson H. (in prep) Is There a Typology of Teacher Responders
to the Comprehensive Assessment of Leadership for Learning (CALL) and Do They
Cluster in Different Types of Schools? A Two-Level Latent Class Analysis of CALL Survey
Data
Bowers 2014
ELDARG:
Education Leadership Data Analytics Research Group
• Intensive longitudinal data analysis
• Visual data analytics
• School data driven decision making & resource
allocation
• Grades as useful data for decision making
• Early dropout prediction and intervention
• STEM high school pipeline transitions to college
• Mapping effects of high school curricular choices
• Effective school and district leadership practices
• Instructional leadership “best practices”
• Leadership training for school improvement
• School district systems and leadership
• School technology, facilities & finance
• Effects of gaming and computer use on
achievement
• School facility maintenance achievement effects
• School capital facility financing
If you’re interested in any of these research topics, please let me
know. We’re currently recruiting students to join the group!
http://www.tc.columbia.edu/academics/?facid=ab3764
Multi-Level
Subgroup
Identification Longitudinal
(Organization/Classroom Effects)
(Change over time)
(Typology Detection)
Intensive Longitudinal Data Analysis & Visual Data Analytics Bowers 2014
Hierarchical Linear Models
(HLM)
Multi-Level
Subgroup
Identification Longitudinal
(Organization/Classroom Effects)
(Change over time)
(Typology Detection)
Intensive Longitudinal Data Analysis & Visual Data Analytics Bowers 2014
Hierarchical Linear Models
(HLM)
Multi-Level
Subgroup
Identification Longitudinal
(Organization/Classroom Effects)
(Change over time)
(Typology Detection)
Intensive Longitudinal Data Analysis & Visual Data Analytics
Survival Modeling
Hazard Modeling
Longitudinal Risk Models
Bowers 2014
Hierarchical Linear Models
(HLM)
Hierarchical
Growth
Models
Multi-Level
Subgroup
Identification Longitudinal
(Organization/Classroom Effects)
(Change over time)
(Typology Detection)
Intensive Longitudinal Data Analysis & Visual Data Analytics
Survival Modeling
Hazard Modeling
Longitudinal Risk Models
Bowers 2014
Hierarchical Linear Models
(HLM)
Hierarchical
Growth
Models
Multi-Level
Subgroup
Identification Longitudinal
(Organization/Classroom Effects)
(Change over time)
(Typology Detection)
Intensive Longitudinal Data Analysis & Visual Data Analytics
Latent Class Analysis
(LCA)
Survival Modeling
Hazard Modeling
Longitudinal Risk Models
Bowers 2014
Hierarchical Linear Models
(HLM)
Hierarchical
Growth
Models
Multi-Level
Subgroup
Identification Longitudinal
Multilevel
LCA
(Organization/Classroom Effects)
(Change over time)
(Typology Detection)
Intensive Longitudinal Data Analysis & Visual Data Analytics
Latent Class Analysis
(LCA)
Survival Modeling
Hazard Modeling
Longitudinal Risk Models
Bowers 2014
Hierarchical Linear Models
(HLM)
Hierarchical
Growth
Models
Growth Mixture
Modeling
(GMM)
Cluster Analysis
Heatmaps
Multi-Level
Subgroup
Identification Longitudinal
Multilevel
GMM
Multilevel
LCA
(Organization/Classroom Effects)
(Change over time)
(Typology Detection)
Intensive Longitudinal Data Analysis & Visual Data Analytics
Latent Class Analysis
(LCA)
Survival Modeling
Hazard Modeling
Longitudinal Risk Models
Bowers 2014
K-12 Schooling Outcomes
• Dropout risk
– Accuracy, precision, sensitivity & specificity
• Longitudinal patterns in teacher assigned grades
• Data driven decision making (3DM)
• Targeted and data-informed resource allocation
to improve schooling outcomes
Ryan Sprott
Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out?
A Review of the Predictors of Dropping out of High School: Precision,
Sensitivity and Specificity. The High School Journal, 96(2), 77-100.
doi:10.1353/hsj.2013.0000 ( Preprint Available )
Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories
Associated with Dropping Out of High School: A Growth Mixture Model
Analysis. The Journal of Educational Research, 105(3), 176-195.
doi:10.1080/00220671.2011.552075 ( Preprint Available ) Sherry Taff
Bowers 2014
Issues of Accuracy in Predicting High School Dropout
• Dropping out of high school in the U.S. is associated with a multitude of negative outcomes.
• However, other than a select number of demographic and background variables, we know little about the accuracy of current dropout predictors that are school-related.
• Current predictions accurately predict only about 50-60% of students who actually dropout.
• According to Gleason & Dynarski (2002), accurate prediction of who will dropout is a resource and efficiency issue:
– A large percentage of students are mis-identified as at-risk.
– A large percentage of students who are at-risk are never identified.
• Current reporting of dropout “flags” across the literature is haphazard. – Almost none report accuracy
– Many report specificity or sensitivity, but rarely both
• A dropout flag may be highly precise, in that almost all of the students with the flag dropout, but may not be accurate since the flag may identify only a small proportion of the dropouts.
Balfanz et al. (2007); Bowers (2010); Gleason & Dynarski (2002); Pallas (2003); Rumberger (2004) Bowers 2014
Re-analyzing Past Dropout Flags for Accuracy,
Precision, Sensitivity and Specificity
• Literature search. • Queried multiple databases:
– JSTOR, Google Scholar, EBSCO, Educational Full Text Wilson Web
• Studies were included that: – Were published since 1979 – Examined High School dropout – Examined school-wide characteristics and included all students – A focus on the student level – Reported frequencies for re-analysis
• Initially yield 6,434 overlapping studies – 301 studies were read in full – 140 provided school-wide samples and quantifiable data – 36 articles provided enough data for accuracy re-calculations – Yield 110 separate dropout flags
• Relative Operating Characteristic (ROC) – Hits versus False-Alarms
Bowers 2014
Dropout Graduate
Dropout a
True-positive
(TP)
Correct
b
False-positive
(FP)
Type I Error
a+b
Graduate c
False-negative
(FN)
Type II Error
d
True-negative
(TN)
Correct
c+d
a+c b+d a+b+c+d=N
Event
Pre
dic
tor
Precision = a/(a + b) Positive Predictive Value
True-Positive Proportion = a/(a + c) Sensitivity
True-Negative Proportion = d/(b + d) Specificity
False-Positive Proportion = b/(b + d) 1-Specificity
Event table for calculating dropout contingency proportions
Fawcett (2004); Hanley & McNeil (1982); Swets (1988); Zwieg & Campbell (1993)
“Hits”
“False Alarms”
Bowers 2014
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Perfect prediction
Better prediction
Worse prediction
Low attendance
An example of the true-positive proportion plotted against the false-positive
proportion for Balfanz et al. (2007) comparing the relative operating characteristics
(ROC) of each dropout flag.
Bowers 2014
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Relative operating characteristics (ROC) of all dropout flags reviewed, plotted
as the true-positive proportion against the false-positive proportion. Numbers
refer to dropout indicator IDs.
Bowers 2014
Bowers 2014
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Relative operating characteristics (ROC) of all dropout flags reviewed, plotted
as the true-positive proportion against the false-positive proportion. Numbers
refer to dropout indicator IDs.
Bowers 2014
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Relative operating characteristics (ROC) of all dropout flags reviewed, plotted
as the true-positive proportion against the false-positive proportion. Numbers
refer to dropout indicator IDs.
Bowers 2014
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Relative operating characteristics (ROC) of all dropout flags reviewed, plotted
as the true-positive proportion against the false-positive proportion. Numbers
refer to dropout indicator IDs.
Bowers 2014
Demographics:
Student: Female
African American Asian
Hispanic Non-Traditional Family
SES
School: Urban Rural
% Students Free Lunch
Behavior & Structure:
Student: Extracurricular
Retained Negative Behavior
School:
Student-Teacher Ratio Academic Press
Small School Large School
Extra-Large School
High School Dropout
Longitudinal Grades Time1, Time2, Time3
A Growth Mixture Model to Identify Grade Trajectories Associated with Dropout
Bowers 2014
Demographics:
Student: Female
African American Asian
Hispanic Non-Traditional Family
SES
School: Urban Rural
% Students Free Lunch
Behavior & Structure:
Student: Extracurricular
Retained Negative Behavior
School:
Student-Teacher Ratio Academic Press
Small School Large School
Extra-Large School
High School Dropout
A Growth Mixture Model to Identify Grade Trajectories Associated with Dropout
Longitudinal Grades Time1, Time2, Time3
Bowers 2014
Demographics:
Student: Female
African American Asian
Hispanic Non-Traditional Family
SES
School: Urban Rural
% Students Free Lunch
Behavior & Structure:
Student: Extracurricular
Retained Negative Behavior
School:
Student-Teacher Ratio Academic Press
Small School Large School
Extra-Large School
High School Dropout
Latent Trajectory
Classes C
A Growth Mixture Model to Identify Grade Trajectories Associated with Dropout
Longitudinal Grades Time1, Time2, Time3
Bowers 2014
Demographics:
Student: Female
African American Asian
Hispanic Non-Traditional Family
SES
School: Urban Rural
% Students Free Lunch
Behavior & Structure:
Student: Extracurricular
Retained Negative Behavior
School:
Student-Teacher Ratio Academic Press
Small School Large School
Extra-Large School
High School Dropout
Latent Trajectory
Classes C
Intercepts Slopes
GPA 9S1 GPA 9S2 GPA 10S1
A Growth Mixture Model to Identify Grade Trajectories Associated with Dropout
Bowers 2014
Growth mixture model for the simultaneous estimation of latent trajectory classes
using non-cumulative GPA from the first three semesters of high school.
GPA 9S1 GPA 9S2 GPA 10S1
Intercepts Slopes
High School
Dropout
Behavior & Structure:
Student:
Extracurricular
Retained
Negative Behavior
School:
Student-Teacher Ratio
Academic Press
Small School
Large School
Extra-Large School
Demographics:
Student:
Female
African American
Asian
Hispanic
Non-Traditional Family
SES
School:
Urban
Rural
% Students Free Lunch
Latent
Trajectory
Classes
C
Bowers 2014
A:
B:
0
1
2
3
4
9S1 9S2 10S1Semester
No
n-C
um
ula
tive
GP
A
Mid-DecreasingLow-IncreasingMid-AchievingHigh-Achieving
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
0
1
2
3
4
0
1
2
3
4
9S1 9S2 10S1
Semester
0
Non
-cum
ula
tive
GP
A
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
0
1
2
3
4
9S1 9S2 10S1
Semester
Low-Increasing: 13.8%Mid-Decreasing: 10.8% Mid-Achieving: 56.5% High-Achieving: 18.9%
Dropout: 39.7% 52.1% 7.0% 1.2%
Longitudinal Non-Cumulative GPA Trajectories in the First
Three Semesters of High School
n=5,400
Mid-decreasing & Low-increasing accounted for:
24.6% of the sample
91.% of the dropouts
Bowers 2014
Growth Mixture Modeling & Student Outcome
Prediction
Next Steps
• Why grades are predictive of outcomes.
• Examine other outcomes – College-going
– College major
– College graduation
– Later employment & careers
• Two-level GMM – What are the high school-level effects on trajectory
probabilities? • Are there some schools that have higher proportions of certain
grade trajectory groups. What predicts it? (aka “dropout factories)
– Leadership and teacher effects on grade trajectories
Bowers 2014
Overview of the Talk
Patterning Student Data to Predict Outcomes & Inform Interventions:
• What are the most accurate “dropout flags”? • Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out? A Review of
the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity. The
High School Journal.96(2), 77-100.
• Are there significantly different trajectories of student grade patterns
that predict high school drop out? • Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories Associated with
Dropping Out of High School: A Growth Mixture Model Analysis. The Journal of Educational
Research, 105(3), 176-195.
• How early can we detect these patterns? • Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of
Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster
Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18.
Understanding Leadership Effects through Subgroup Trajectory Analysis:
• Does principal training influence school achievement trajectories? • Bowers, A.J., White, B.R. (in press) Do Principal Preparation and Teacher Qualifications
Influence Different Types of School Growth Trajectories in Illinois? A Growth Mixture Model
Analysis.Journal of Educational Administration.
• Are there different types of school leadership? • Bowers, A.J., Blitz, M., Halverson H. (in prep) Is There a Typology of Teacher Responders
to the Comprehensive Assessment of Leadership for Learning (CALL) and Do They
Cluster in Different Types of Schools? A Two-Level Latent Class Analysis of CALL Survey
Data
Bowers 2014
Cluster Analysis & Heatmap Visualizations • Research Question:
– To what extent are there similar user data patterns? – What do the data patterns look like?
• Method: – Cluster analysis is a descriptive statistic
• No hypothesis test
– Similar to the original Netflix and Amazon.com user preference algorithms
• “Other users who also liked/bought/viewed this item, also liked/bought/viewed these other items that you have not yet considered”
– Fewer assumption violation issues than with regression statistics so it can take a wide range of data types
– Attempts to use all of the available data and is robust to missing data issues
• Heatmaps are adapted from bioinformatics and cancer biology
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students:
Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical
Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf Bowers 2014
A
B
C
D
F
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade Year
Gra
de M
ark
ing
Hypothetical Hierarchical Clustering Data
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
Bowers 2014
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
A
B
C
D
F
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade Year
Gra
de M
ark
ing
Hypothetical Hierarchical Clustering Data
Bowers 2014
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
A
B
C
D
F
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade Year
Gra
de M
ark
ing
Hypothetical Hierarchical Clustering Data
Bowers 2014
Hypothetical Hierarchical Clustering Data: Clustergram
Student 3
Student 7
Student 5
Student 4
Student 8
Student 6
Student 1
Student 2
A
B
C
D
F
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade Year
Bowers 2014
K 1 2 3 4 5 6 7 8 9 10 1112
Grade Year
A B C D F
Hypothetical Hierarchical Clustering Data: Clustergram
Student 3
Student 7
Student 5
Student 4
Student 8
Student 6
Student 1
Student 2
A
B
C
D
F
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade Year
Bowers 2014
K 1 2 3 4 5 6 7 8 9 10 1112
Grade Year
A B C D F
Hypothetical Hierarchical Clustering Data: Clustergram
Student 3
Student 7
Student 5
Student 4
Student 8
Student 6
Student 1
Student 2
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Su
bje
cts
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Su
bje
cts
Students
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Su
bje
cts
Students
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Su
bje
cts
Students
Bowers 2014
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Su
bje
cts
Students
+3 -3 0 No Data Bowers 2014
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th
Bowers 2014
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th
Bowers 2014
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th
Bowers 2014
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th High-High
Bowers 2014
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th High-High
Low-Low
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th High-High
Low-Low
Low-High
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th High-High
Low-Low
Low-High
High-Low
0
0.5
1
1.5
2
2.5
3
3.5
4
K 1 2 3 4 5 6 7 8 9S1 9S2 10S1 10S2 11S1 11S2 12S1 12S2
Grade-Level
Mean N
on-c
um
ula
tive G
PA
High-High
Mean non-cumulative GPA trends for clusters high-high,
low-low, low-high and high-low, K-12
Bowers 2014
0
0.5
1
1.5
2
2.5
3
3.5
4
K 1 2 3 4 5 6 7 8 9S1 9S2 10S1 10S2 11S1 11S2 12S1 12S2
Grade-Level
Mean N
on-c
um
ula
tive G
PA
High-High Low-Low
Mean non-cumulative GPA trends for clusters high-high,
low-low, low-high and high-low, K-12
Bowers 2014
0
0.5
1
1.5
2
2.5
3
3.5
4
K 1 2 3 4 5 6 7 8 9S1 9S2 10S1 10S2 11S1 11S2 12S1 12S2
Grade-Level
Mean N
on-c
um
ula
tive G
PA
Mean non-cumulative GPA trends for clusters high-high,
low-low, low-high and high-low, K-12
High-High Low-Low High-Low
Bowers 2014
0
0.5
1
1.5
2
2.5
3
3.5
4
K 1 2 3 4 5 6 7 8 9S1 9S2 10S1 10S2 11S1 11S2 12S1 12S2
Grade-Level
Mean N
on-c
um
ula
tive G
PA
Mean non-cumulative GPA trends for clusters high-high,
low-low, low-high and high-low, K-12
High-High Low-Low High-Low Low-High
Bowers 2014
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th
Bowers 2014
Cluster analysis future work:
• Replicate with a national sample
• Partner with school districts to replicate and extend the work
• Cluster in two dimensions: Teacher x Student
• Include other types of student data & outcomes
• Test other types of cluster algorithms
• Integrate into data dashboard systems for principals & admin
Bowers 2014
Overview of the Talk
Patterning Student Data to Predict Outcomes & Inform Interventions:
• What are the most accurate “dropout flags”? • Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out? A Review of
the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity. The
High School Journal.96(2), 77-100.
• Are there significantly different trajectories of student grade patterns
that predict high school drop out? • Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories Associated with
Dropping Out of High School: A Growth Mixture Model Analysis. The Journal of Educational
Research, 105(3), 176-195.
• How early can we detect these patterns? • Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of
Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster
Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18.
Understanding Leadership Effects through Subgroup Trajectory Analysis:
• Does principal training influence school achievement trajectories? • Bowers, A.J., White, B.R. (in press) Do Principal Preparation and Teacher Qualifications
Influence Different Types of School Growth Trajectories in Illinois? A Growth Mixture Model
Analysis.Journal of Educational Administration.
• Are there different types of school leadership? • Bowers, A.J. , Blitz, M., Halverson H. (in prep) Is There a Typology of Teacher
Responders to the Comprehensive Assessment of Leadership for Learning (CALL) and Do
They Cluster in Different Types of Schools? A Two-Level Latent Class Analysis of CALL
Survey Data
Bowers 2014
Do principal training and experience influence
school achievement trajectories?
• Bowers, A.J., White, B.R. (in press) Do Principal Preparation and Teacher
Qualifications Influence Different Types of School Growth Trajectories in
Illinois? A Growth Mixture Model Analysis. Journal of Educational
Administration.
Brad White
Illinois Education Research Council
Bowers 2014
Purpose:
• The purpose of this study is to examine the extent to which principal preparation and professional experience influences different types of school proficiency growth trajectories
Background:
• Principal leadership can have a strong influence on student achievement.
• However, direct effects models are highly problematic (Hallinger & Heck, 1996, 2011).
• Recent research has begun to examine the effects of principal leadership, experience and training on growth in student achievement over time.
– Principal effects appear to be stronger in high poverty schools.
– Principal education appears to be unrelated to student achievement growth in NYC and Illinois.
– Principal on-the-job experiences are related to student achievement growth, such as time as an assistant principal and tenure as principal in the school.
• (Branch, Hanushek & Rivkin, 2009, 2012; Clark, Martorell & Rockoff, 2009; White & Bowers, 2011)
• All of these studies fit the schools to a single proficiency growth trajectory.
• But what if there are multiple statistically significantly different growth trajectories? – Hallinger & Heck (2011) have proposed that leadership by influence different school growth
trajectories in different ways.
Purpose & Background
Bowers 2014
• To what extent are there different trajectories of elementary and middle school test score proficiency across multiple years of data?
• To what extent are principal background and experience variables related to school proficiency trajectory?
Research Questions
Bowers 2014
Sample:
• All Illinois elementary and middle schools – 2000-2001 through 2005-2006
• Illinois Standard Achievement Test (ISAT) – School-level percent met or exceeded standard
• Non-Chicago (n=2,584) and Chicago schools (n=499)
• Teacher Variables – % Inexperienced teachers
– ITAC (Index of Teacher Academic Capital): • Teacher ACT scores
• Teacher undergraduate college competitiveness
• Proportion of teachers emergency or provisional
• Proportion of teachers who failed the Illinois basic skills test on first attempt
• Principal Variables: – Age
– Female
– Minority
– Selectivity of undergraduate institution (Barron’s)
– Selectivity of graduate institution (Barron’s)
– First year principal or Principal 6+ years (vs. 2-5 years)
– Years principal had served as assistant principal in same school
Sample
Bowers 2014
Method
• Intensive Longitudinal Data (ILD) analysis – Collins, 2006; Shiyko, et al., 2012; Walls & Schafer, 2006 – Hierarchical linear growth models
• Time (level 1) nested within schools (level 2) • Long-format data • Estimates effects of time-varying covariates on:
– Intercepts (year 1 scores) – Slopes (growth in scores over time)
• (Hox, 2010; Raudenbush & Bryk, 2002; Singer & Willet, 2003)
• Growth mixture modeling (GMM)
– To what extent are there statistically significantly different trajectories in proficiency growth?
– Estimate hierarchical linear growth model conditional on trajectory group (latent class).
– Hallinger & Heck, 2011; Muthén, 2004; Shiyko, et al., 2012 Bowers 2014
2001
Intercepts Slopes
Principal Background &
Experience:
Female
Minority
Age
Selective BA
Selective Masters
First Year Principal
Two-Five Years Principal
Assistant Principal
Taught in Same School
Demographics:
Students:
% African American
% Hispanic
% Asian
% Free & Reduced Lunch
% LEP
Enrollment
Mobility
School:
% Inexperienced Teachers
ITAC
Latent
Trajectory
Classes
C
2002 2003 2004 2005 2006
School % met or exceed standard on ISAT
Growth Mixture Model of Illinois ISAT Elem & MS Growth Trajectories
Bowers, A. J., & White, B. R. (in press). Do Principal Preparation and Teacher Qualifications Influence Different Types of School Growth
Trajectories in Illinois? A Growth Mixture Model Analysis. Journal of Educational Administration.
n=3,154 Elementary and Middle Schools
Bowers 2014
Growth Mixture Model of Illinois ISAT Elem & MS Growth Trajectories
Bowers, A. J., & White, B. R. (in press). Do Principal Preparation and Teacher Qualifications Influence Different Types of School Growth
Trajectories in Illinois? A Growth Mixture Model Analysis. Journal of Educational Administration. Bowers 2014
0
20
40
60
80
100
% M
et o
r E
xce
ed
ed
Sta
nd
ard
(IS
AT
)
Asst. Princ. Not Asst. Princ.
Chicago
Principal Previously Assistant Principal
High
Low
High
Low
Year Year
0
20
40
60
80
100
% M
et o
r E
xce
ed
ed
Sta
nd
ard
(IS
AT
) High
Low
High
Low
Non-Chicago Chicago
Asst. Princ. Not Asst. Princ.
Taught in same school Taught in diff. school
Taught in same school Taught in diff. school
Principal Previously Taught in Same School
Non-Chicago
Growth Mixture Model of Illinois ISAT Elem & MS Growth Trajectories
Bowers, A. J., & White, B. R. (in press).
Overview of the Talk
Patterning Student Data to Predict Outcomes & Inform Interventions:
• What are the most accurate “dropout flags”? • Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out? A Review of
the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity. The
High School Journal.96(2), 77-100.
• Are there significantly different trajectories of student grade patterns
that predict high school drop out? • Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories Associated with
Dropping Out of High School: A Growth Mixture Model Analysis. The Journal of Educational
Research, 105(3), 176-195.
• How early can we detect these patterns? • Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of
Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster
Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18.
Understanding Leadership Effects through Subgroup Trajectory Analysis:
• Does principal training influence school achievement trajectories? • Bowers, A.J., White, B.R. (in press) Do Principal Preparation and Teacher Qualifications
Influence Different Types of School Growth Trajectories in Illinois? A Growth Mixture Model
Analysis.Journal of Educational Administration.
• Are there different types of school leadership? • Bowers, A.J., Blitz, M., Halverson H. (in prep) Is There a Typology of Teacher Responders
to the Comprehensive Assessment of Leadership for Learning (CALL) and Do They
Cluster in Different Types of Schools? A Two-Level Latent Class Analysis of CALL Survey
Data
Bowers 2014
• Latent Class Analysis
– Research question: Is there only one kind of respondent across a survey, or are there statistically significantly different “modes” across the multidimensional response data?
– Said another way:
• Is there a single monolithic response group, or are there statistically significant different subgroups that respond or behave differently?
Bowers 2014
Are there different types of school leadership? • Bowers, A.J., Blitz, M., Halverson, R. (in prep) Is There a Typology of Teacher Responders to the
Comprehensive Assessment of Leadership for Learning (CALL) and Do They Cluster in Different
Types of Schools? A Two-Level Latent Class Analysis of CALL Survey Data.
Angela Urick, University of Oklahoma Urick, A., Bowers, A.J. (2014) What are the Different Types of Principals Across the U.S.? A Latent Class
Analysis of Principal Perception of Leadership. Educational Administration Quarterly, 50(1) 96-134.
Mark Blitz
University of Wisconsin-Madison
Richard Halverson
Bowers 2014
Bowers 2014
Bowers 2014
Background & Purpose
• Are responses across the CALL survey at the teacher and principal levels homogenous or heterogeneous? – This study is in response to recent calls to more appropriately
model the complex nature of teaching and leadership in schools (Hallinger & Heck 2011)
• We used Latent Class Analysis (LCA) to address this question. – LCA does not force everyone in a sample to a single “best fit”
line, but rather examines if there are significantly different subgroups/typology
– Barnes, Camburn & Sebastian (2010); Bowers (2012); Henry & Muthen (2010); Urick & Bowers (in press); Urick (2012)
Bowers 2014
CALL Domain Subdomain 1 Subdomain 2 Subdomain 3 Subdomain 4 Subdomain 5
1. Focus on learning School-wide focus on
learning
Formal leaders
recognized as
instructional leaders
Collaborative design
of integrated
learning plan
Appropriate services
for students who
struggle
2. Monitoring Teaching
& Learning
Formative eval of
student learning
Summative eval of
student learning
Formative eval of
teaching
Summative eval of
teaching
3. Building Nested
Learning Communities
Collaborative school-
wide focus on
teaching and learning
Professional learning Socially distributed
leadership
Coaching and
mentoring
4. Acquiring & Allocating
Resources
Personnel practices Structuring and
maintaining time
School resources
focused on student
learning
Integrating external
expertise into
instruction
Coordinate & supervise
relations with families
& external communities
5. Maintaining Safe &
Effective Learning
Environment
Clear, consistent &
enforced expectations
for student behavior
Safe learning
environment
Safe haven provided
for struggling
students
Buffering the
teaching
environment
CALL Survey Leadership for Learning Domains & Subdomains
Bowers 2014
Research questions
• To what extent is there a typology of teacher and principal response patterns to CALL?
• To what extent do teacher and principal response patterns to CALL align (or not) across different types of schools and responders?
Bowers 2014
Sample & Method
• CALL Validation sample: – Level 1: n=3,919 teachers
– Level 2: n=109 schools
• Mean scores across the 21 subdomains
• Latent Class Analysis (LCA) – Person-centered statistic, rather than variable
centered.
– Identifies if there is one, or more than one significantly different type of responder.
• Bowers (2012), Henry & Muthen (2010), Muthen (2004), Urick (2012).
Bowers 2014
Two level latent class analysis (LCA) with three classes at level 1
teacher level (CW) and three classes at level 2 school level (CB)
Bowers 2014
Two level latent class analysis (LCA) with three classes at level 1
teacher level (CW) and three classes at level 2 school level (CB)
Bowers 2014
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Bowers 2014
Focus on
Learning Monitor Teaching
& Learning Building Nested
Learning
Communities
Acquiring &
Allocating
Resources Maintaining Safe
& Effective Learning
Environments
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Bowers 2014
A
A A) Providing appropriate services
for struggling students
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Focus on
Learning Monitor Teaching
& Learning Building Nested
Learning
Communities
Acquiring &
Allocating
Resources Maintaining Safe
& Effective Learning
Environments
Bowers 2014
A
B A
B
A) Providing appropriate services
for struggling students
B) Formative evaluation of
teaching
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Focus on
Learning Monitor Teaching
& Learning Building Nested
Learning
Communities
Acquiring &
Allocating
Resources Maintaining Safe
& Effective Learning
Environments
Bowers 2014
A
B C A
B C
A) Providing appropriate services
for struggling students
B) Formative evaluation of
teaching
C) Personnel practices
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Focus on
Learning Monitor Teaching
& Learning Building Nested
Learning
Communities
Acquiring &
Allocating
Resources Maintaining Safe
& Effective Learning
Environments
Bowers 2014
Focus on
Learning Monitor Teaching
& Learning Building Nested
Learning
Communities
Acquiring &
Allocating
Resources Maintaining Safe
& Effective Learning
Environments
A
B C
D
A
B C
D
E
A) Providing appropriate services
for struggling students
B) Formative evaluation of
teaching
C) Personnel practices
D) Structuring time, resources &
expertise
E) Buffering the teaching
environment
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Bowers 2014
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Bowers 2014
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
1.4 Providing appropriate services for
students who traditionally struggle
Bowers 2014
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
1.4 Providing appropriate services for
students who traditionally struggle
2.1 – 2.2 Formative & Summative
evaluation of student learning
Bowers 2014
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
1.4 Providing appropriate services for
students who traditionally struggle
2.1 – 2.2 Formative & Summative
evaluation of student learning
4.1 Personnel practices
Bowers 2014
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
1.4 Providing appropriate services for
students who traditionally struggle
2.1 – 2.2 Formative & Summative
evaluation of student learning
4.1 Personnel practices
4.2 Structuring and maintaining time
4.3 School resources are focused on
student learning
4.4 Integrating external expertise into
school instructional program
Bowers 2014
3 Leader and 3 Teacher Leadership for Learning (LL) Response Subgroups
Bowers 2014
0
20
40
60
80
100
Pe
rcen
t o
f Te
ach
ers
0
1
2
3
Avg L
ea
der
Ye
ars
Exp
erie
nce
0
10
20
30
Avg T
ea
ch
er
Ye
ars
Exp
erie
nce
Low LL
Schools Moderate
LL Schools
High LL
Schools
0
1
2
3
4
Ave
rage
Su
rve
y R
esp
onse
Leader High Teachers Mod Teachers
LowTeachers
Three School Contexts of Leadership for Learning
Bowers 2014
Conclusions and Implications
Three significantly different groups of teacher & leaders respondents to CALL
Low (31.3%) • Focus on struggling students
• Less formative eval. of teachers
• Not buffered as much from environ
Moderate (43.7%) High (25.0%) • Collab design of integrated learning plans
• Formative eval. of teaching & learning
• Coaching & mentoring
• Acquisition and allocation of resources
Leadership for Learning Teachers
Leadership for Learning Leaders/Schools
Low (41.5%) • Highest % of Low LL Teachers
• Most experienced teachers
• Least experienced leaders
• Lowest Leader responses
Moderate (41.5%) High (17.0%) • Highest % of High LL Teachers
• Least experienced teachers
• Most experienced leaders
• Strong alignment between teachers and
leaders
Bowers 2014
Overview of the Talk
Patterning Student Data to Predict Outcomes & Inform Interventions:
• What are the most accurate “dropout flags”? • Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out? A Review of
the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity. The
High School Journal.96(2), 77-100.
• Are there significantly different trajectories of student grade patterns
that predict high school drop out? • Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories Associated with
Dropping Out of High School: A Growth Mixture Model Analysis. The Journal of Educational
Research, 105(3), 176-195.
• How early can we detect these patterns? • Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of
Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster
Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18.
Understanding Leadership Effects through Subgroup Trajectory Analysis:
• Does principal training influence school achievement trajectories? • Bowers, A.J., White, B.R. (in press) Do Principal Preparation and Teacher Qualifications
Influence Different Types of School Growth Trajectories in Illinois? A Growth Mixture Model
Analysis.Journal of Educational Administration.
• Are there different types of school leadership? • Bowers, A.J., Blitz, M., Halverson H. (in prep) Is There a Typology of Teacher Responders
to the Comprehensive Assessment of Leadership for Learning (CALL) and Do They
Cluster in Different Types of Schools? A Two-Level Latent Class Analysis of CALL Survey
Data
Bowers 2014