informing early childhood policy: an analysis of the sensitivity of a school readiness risk index to...
TRANSCRIPT
INFORMING EARLY CHILDHOOD POLICY: AN ANALYSIS OF THE SENSITIVITY OF A
SCHOOL READINESS RISK INDEX TO CHANGES IN INDICATOR SELECTION
KRISTA S. SCHUMACHER, PHD CANDIDATEAPRIL 3, 2015
2
PURPOSE Examine sensitivity of Oklahoma School Readiness Risk Index (OK SRRI) to changes to the
indicator set
OK SRRI Originally published 2013 by Oklahoma Department of Human Services Revised in 2014 as part of Oklahoma School Readiness Reach-by-Risk Report Ranks counties according to risk for children starting school unprepared using 10 indicators Rankings compared to availability of early education and child care Intended use:
Inform decisions regarding early childhood policies and resource allocation
3
OK SRRI INDICATORS 18 indicators considered
race/ethnicity, family and economic characteristics, health Final set of 10 selected using principal components analysis (PCA) and OLS
multiple regression Indicators organized into domains via PCA results Index scores: mean z-scores for individual indicators, equal weighting
Hispanic Background Family Structure / Economic Distress
Children in Child Welfare
Hispanic ethnicity Poverty (<100% FPL) Abuse/neglect
English-language learners (ELL) Single parent Foster care
Migrant Young maternal age (< age 20)
Low maternal education (no high school diploma)
American Indian race
Excluded indicatorsBehind on vaccinations
Tobacco use during pregnancyLate or no prenatal care
Low birth weight
Black race
Homeless
IEP
Kindergarten retention
4
ALTERNATE INDEX Alternate OK SRRI created based on original set of 18 indicators
Goals for alternate index: 1. Indicators can be addressed through interventions and/or public policy2. Theoretically based: Transactional/ecological theory of child development and cumulative risk model3. Full representation of multidimensional construct of school readiness risk4. Follow formative measurement model
Final indicators selected by analyzing indicator descriptive statistics, correlationsReady Families Ready Communities Ready Services - Health
Single parent Poverty (<100% FPL) Behind on vaccinations
Low maternal education ELL Tobacco use during pregnancy
Young maternal age Late or no prenatal care
Abuse/neglect Low birth weight
Foster care
Excluded indicators
Hispanic ethnicity
American Indian race
Black race
Migrant
Homeless
IEP
Kindergarten retention
5
MEASUREMENT MODELS: REFLECTIVE & FORMATIVE
Reflective Scale development to measure latent variable, e.g.., extroversion, happiness
Latent variable exists independently of its indicators Indicators must be highly correlated for unidimensional scale Error can be extracted
Formative Index development to measure composite variable, e.g., school readiness risk
Composite variable depends on the indicators Indicators form a multidimensional construct Highly correlated indicators = construct underrepresentation Error cannot be extracted (considered disturbance term)
Extroversion
Social
Talkative
Enthusiastic
Risk
Poverty
ELL
Teen mother
6
INDICATOR CORRELATIONS AND OUTLIERS # outliers
American Indian 1Migrant 6Hispanic 7ELL 10Poverty 5Single parent 0Young maternal age 0Low maternal education 3Abuse/neglect 3Foster care 1Vaccinations 1Birth weight 1Prenatal care 2Tobacco use 2
7
METHODS1. Indicator and domain dominance
• Pearson’s r correlations of indicator and domain z-scores with overall index scores• Multiple regression commonality analysis
2. Relative effect of indicators and domains on outcome rankings• Excluded one indicator or domain at a time from index score calculations• Examined rankings (percentiles) on reduced scenarios compared to full index
• ≥ 15 percentiles = moderate shift; ≥ 20 percentiles = significant shift
• Compared rankings across full original and alternate indexes
3. Associations of index rankings with a proxy outcome of school readiness risk• Spearman rank-order correlations of full and reduced indexes to percent of kindergartners
scoring below proficient on pre-literacy assessment
8
FINDINGS: DOMINANCE ANALYSES Correlations
One negative association between indicators and scores on alternate index (vaccinations) One negative association between domains on both indexes Families domain dominated alternate index
Commonality analysis Little unique contributions of indictors to either index
Vaccinations and tobacco use indicators small confounding effect (negative shared effects) Domains
More unique effects than indicators Largest unique and total effects
Families domain (alternate index); Family Structure/Economic Distress domain (original index)
Four indicators had largest correlations and explained variance in both indexes Low maternal education, young maternal age, poverty, single parent
Changes in indicator impact from original to alternate indexes Abuse/neglect, foster care and teen mothers increased Low maternal education decreased marginally ELL decreased considerably
9
FINDINGS: EFFECTS ON RANKINGSRange of median rank shifts on each reduced
scenario
# counties shifting ≥ 15 percentiles on each
reduced scenario
# counties shifting ≥ 15 percentiles across all
reduced scenariosIndicators
Original 1.3 to 3.9 1 to 3 12
Alternate 2.6 to 3.9 3 to 7 26
Domains
Original 5.2 to 7.8 11 to 24 37
Alternate 5.2 to 18.2 10 to 41 53
10
FINDINGS: ORIGINAL & ALTERNATE INDEX RANK COMPARISONS
rs = .88 Median rank shift = 8 percentiles 26 counties shifted by ≥ 15 percentiles, 16 by ≥ 20 percentiles
Even split of number increasing and decreasing Trend of shifts from original to alternate indexes
Increases: Relatively moderate to low rankings on original index Several counties with high scores on at least one health indicator
Decreases: Relatively high rankings on original index Several counties with high scores on racial/ethnic indicators
Changes to risk groupings occurred for 22 counties
11
FINDINGS: ASSOCIATIONS WITH LITERACY SKILLS
Original and alternate indexes moderately correlated with literacy skills rs = .20 and .21, respectively
Associations with indicator-reduced scenarios Original: rs = .17 (single parent) to .24 (Hispanic) Alternate rs = .15 (single parent) to .24 (vaccinations)
Associations with domain-reduced scenarios Original: rs = .08 (Family Structure/Economic Distress) to rs = .26 (Hispanic Background) Alternate: rs = .01 (Ready Families) to rs = .24 (Ready Services – Health)
12
CONCLUSIONS OK SRRI sensitive to changes to indicator set The same indicators can behave differently when used with varying sets of indicators
E.g., abuse/neglect, foster care, ELL Indicators with greatest impact on both indexes
Low maternal education, young maternal age, poverty and single parent Use of highly correlated indicators results in double counting
More critical with indicators of greater variability and numerous outliers Reliance on statistical methods of indicator selection underrepresented the construct
More problematic with population-level data Ecological fallacy Differential behavior of correlations
Domains must be organized conceptually and unambiguously to fulfill intended purpose Identify areas of concern and promote appropriate responses
13
IMPLICATIONS
Choice of indicators matters Less sensitivity of original index to changes to indicator set a result of:
High correlations Fewer dimensions of school readiness risk represented Is sensitivity to changes to indicator set necessarily a bad thing?
Construct of school readiness risk highly dependent on indicators used Different sets of indicators can significantly change construct definition Need consensus regarding construct definition, construct domains and issues of relevance
Selecting indicators should be participatory process Indexes inherently political in nature
Excluding part of construct from index excludes it from consideration regarding policy and resources Indexes have power to direct resources toward or away from counties with particular characteristics
“The act of deciding what to count is value oriented and subjective in nature” (Simpson, 2006, p. 5).