Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules
MODULE 2 SLIDE DECK:LESSONS IN USING (AND MISUSING) CALIFORNIA’S CHILD WELFARE DATA
2Module 2: Lessons in Using Data 2
Instructor Notes for Module 2 This module exposes students to data
concerning California’s child welfare system, its purpose is to: Provide a broad overview of California’s child
welfare system through visual displays of data Introduce state and federal child welfare
indicators for tracking agency performance (with a more technical module for optional use)
Promote critical thinking in the context of basic statistical concepts through the review of popular press examples based on actual child welfare data
Module 2: Lessons in Using Data 3
Understanding California’s Child Welfare System through Data
Module 2, Section 1
4Module 2: Lessons in Using Data 4
The “big” picture in 2011… 9,992,333 children under the age of 18 471,790 children reported for maltreatment
(47.2 per 1,000 children) 90,472 children with a substantiated
allegation(9.1 per 1,000 children)
31,431 children entered foster care(3.1 per 1,000 children)
On any given day, roughly 59,484 children in foster care
(6.0 per 1,000 children)
5Module 2: Lessons in Using Data 5
The Iceberg Analogy
Maltreated children not known to child protective services
Maltreated children known to child protective services
Module 2: Lessons in Using Data 6
Module 2: Lessons in Using Data 7
Module 2: Lessons in Using Data 8
Module 2: Lessons in Using Data 9
Module 2: Lessons in Using Data 10
Module 2: Lessons in Using Data 11
Module 2: Lessons in Using Data 12
Module 2: Lessons in Using Data 13
Module 2: Lessons in Using Data 14
Module 2: Lessons in Using Data 15
Tracking child welfare performance through federal and state outcome measures
Module 2, Section 2
16Module 2: Lessons in Using Data 16
Trends over the last two decades Increased (and improved) data collection Increased emphasis on accountability
Observed across government agencies Shift from measuring processes, to
performance outcomes What matters is where you end-up…
promotes innovation But what “outcomes” should we measure?
And how can we best “measure” these outcomes?
1717
Lesson #1: Any One Measure Will Not be Enough…
counterbalancedindicators of system
performance
permanencythrough reunification,
adoption, orguardianship
lengthof stay
stability of care
rate of referrals/substantiated referrals
home-based services vs.
out of home care
positive attachments to family, friends, and neighbors
use of leastrestrictive
form of care
Slide Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill: Jordan Institute for Families,
reentry to care
1818
Federal and State Outcome Measures
Federal Child and Family Services Review (CFSR)
State Accountability Act AB 636
Went into effect in California on January 1, 2004. This new system holds the state and counties
accountable for improving outcomes for children through the establishment of improvement goals, public reporting of outcomes and county-specific improvement plans that must be approved by county boards of supervisors and submitted to the state
No goals or standards. Rather, objective is continuous, quality improvement within each county.
19Module 2: Lessons in Using Data 19
Lesson #2: Measuring Outcomes Can Get Complicated (quickly)…
20Module 2: Lessons in Using Data 20
What you will find reported for California
2121
Website view example…
Reunification composite
Module 2: Lessons in Using Data 22
Optional/additional performance outcome information for instructor use
Module 2, Section 2.1
23Module 2: Lessons in Using Data 23
Children and Families Service Reviews(more details than most will want, but truly useful to understand!)
Federal Child and Families Service Reviews (CFSR) Transition from individual “measures” to safety
indicators and composite measures or permanency and stability
National standards for both the indicators and composites are based on the 75th percentile of state performance in 2004
Although national standards have been set for the composites rather than individual measures… The goal is to improve State performance on all
measures (every improvement reflects a better outcome for children)
Improvement on any given measure will result in an increase in the overall composite score
Analogous to Academic Achievement Test Scoring…
24Module 2: Lessons in Using Data 24
Principal Components Analysis (PCA)(the “black box” version)
black box of fancy statistical tools
Timeliness of Reunification
Timeliness of AdoptionPermanency of
ReunificationPlacement Stability
Median Time in CareRecurrence of Maltreatment
Abuse in Foster Care
Emancipating from Care Component #1Component #2Component #3
A bunch of measures… Three components based on related measures!
25
Z-Scores? Before dumping all of the measures into the PCA “Black
Box”, they were transformed into standard scores (z-scores)
A z-score serves two purposes:Puts measures in the same “range”
Sets measures to the same “system”
2626
And an Example… A researcher interested in measuring
“success” in high school. Collects the following measures for each
student:Athletic AbilityGood Grades Physical AttractivenessInterest in SportsChess Club Membership Science Club MembershipSocial LifePrincipal Components Analysis…
Module 2: Lessons in Using Data 27
Interest in Sports Athletic Ability
Good Grades Chess Club Member Science Club Member
Physical Attractiveness Active Social Life
Reduces the number of individual measures:
VERY HIGHLY ASSOCIATED!!
Explores the contribution of each part to the whole:
Jock Component =
Brainiac Component =
Popular Kids Component =
Structures the data into independent components: Athletic
AbilityInterest in Sports
Good Grades
Chess Club Member
Physical Attractiveness
Active Social Life
28Module 2: Lessons in Using Data 28
Measure Contributions to Composites
C1.1 (22%)
C1.2 (21%)
C1.3 (12%)
C1.4 (46%)
0%
20%
40%
60%
80%
100%
Composite 1
Reunification Within 12 Months (Exit Cohort)
Median Time To Reunification (Exit Cohort)
Reentry Following Reunification (Exit Cohort)
Reunification Within 12 Months (Entry Cohort)
Note: Measures may not sum to exactly 100% due to rounding.
29Module 2: Lessons in Using Data 29
C2.1 (15%)
C2.2 (19%)
C2.3 (22%)
C2.4 (18%)
C2.5 (26%)
0%
20%
40%
60%
80%
100%
Composite 2
Adoption Within 24 Months (Exit Cohort)
Median Time To Adoption (Exit Cohort)
Adoption Within 12 Months (17 Months In Care)
Legally Free Within 6 Months (17 Months In Care)
Adoption Within 12 Months (Legally Free)
Note: Measures may not sum to exactly 100% due to rounding.
Measure Contributions to Composites
30Module 2: Lessons in Using Data 30
C3.1 (33%)
C3.2 (25%)
C3.3 (42%)
0%
20%
40%
60%
80%
100%
Composite 3
Exits to Permanency (24 Months In Care)
Exits to Permanency (Legally Free At Exit)
In Care 3 Years Or Longer (Emancipated/Age 18)
Note: Measures may not sum to exactly 100% due to rounding.
Measure Contributions to Composites
31Module 2: Lessons in Using Data 31
C4.1 (33%)
C4.2 (34%)
C4.3 (33%)
0%
20%
40%
60%
80%
100%
Composite 4
Placement Stability (8 Days To 12 Months In Care)
Placement Stability (12 To 24 Months In Care)
Placement Stability (At Least 24 Months In Care)
Note: Measures may not sum to exactly 100% due to rounding.
Measure Contributions to Composites
32Module 2: Lessons in Using Data 32
C4.1 (33%)C3.1 (33%)
C2.1 (15%)C1.1 (22%)
C4.2 (34%)C3.2 (25%)
C2.2 (19%)C1.2 (21%)
C4.3 (33%)C3.3 (42%)
C2.3 (22%)C1.3 (12%)
C2.4 (18%)
C1.4 (46%)
C2.5 (26%)
0%
20%
40%
60%
80%
100%
Composite 1 Composite 2 Composite 3 Composite 4Note: Measures may not sum to exactly 100% due to rounding.
Measure Contributions to Composites
Module 2: Lessons in Using Data 33
Module 2: Lessons in Using Data 34
Module 2: Lessons in Using Data 35
Module 2: Lessons in Using Data 36
Popular press examples of data use/misuse (aka, numbers gone wild)
Module 2, Section 3
37Module 2: Lessons in Using Data 37
Public Data: Putting It All Out There
PROS: Greater performance accountability Community awareness and involvement,
encourages public-private partnerships Ability to track improvement over time,
identify areas where programmatic adjustments are needed
County/county and county/state collaboration
Transparency Dialogue
38Module 2: Lessons in Using Data 38
Public Data: Putting It All Out There
CONS: Potential for misuse, misinterpretation,
and misrepresentation Available to those with agendas or looking
to create a sensational headline Misunderstood data can lead to the wrong
policy decisions “Torture numbers, and they’ll confess
to anything”(Gregg Easterbrook)
Module 2: Lessons in Using Data 39
There are three kinds of lies:
Lies, Damned Lies and StatisticsMisused
Statistics
^
40Module 2: Lessons in Using Data 40
1)Compare Apples and Oranges2)Use ‘snapshots’ of Small Samples3)Rely on Unrepresentative Findings4)Logically ‘flip’ Statistics 5)Falsely Assume an Association to be
Causal
Five Ways to Misuse Data (without actually lying!):
41Module 2: Lessons in Using Data 41
Two doctors in Anytown, CA… Doctor #1 Doctor #2
What if the populations served by each doctor were very different?
2/1000 20/1000
1) Compare Apples and Oranges
Doctor of the Year?
Module 2: Lessons in Using Data 42
“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”
SF Chronicle, “Accidents of Geography”, March 8, 2006
43
Different families and children served?
Different related outcomes?First entry rates in Fresno are consistently lower
Re-entries in Fresnoare also lower…
“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”
44Module 2: Lessons in Using Data 44
Number of Crimes Period 1: 76Period 2: 51Period 3: 91Period 4: 76
Crime jumped by 49%!!No change.Crime dropped by 16%Average = 73.5
Crime in Anytown, CA…
2) Data Snapshots…
Module 2: Lessons in Using Data 45
“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area...”
SF Chronicle, “No refuge. For foster youth, it’s a state of chance”, November 15, 2005
46Module 2: Lessons in Using Data 46
Abuse in Care RatePeriod 1: 1.80% Period 2: 1.64% Period 3: 0.84% Period 4: 0.00%
Responsible use of the data prevents us from making any of these claims (positive or
negative). The sample is too small; the time frame too limited.
100% improvement! 0 Children Abused!
= 2/111
= 0
= 2/122= 1/119
“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…”
47Module 2: Lessons in Using Data 47
Survey of people in Anytown, CA…
90% of respondents stated that they support using tax dollars to build a new football stadium.
The implication of the above finding is that there is overwhelming support for the stadium…
But what if you were then told that respondents had been sampled from a list of season football ticket
holders?
3) Unrepresentative Findings…
Module 2: Lessons in Using Data 48
“Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.”
“My Word”, Oakland Tribune, May 25, 2006
49Module 2: Lessons in Using Data
“…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” Oakland Tribune
Factually true? Yes.
Misleading? Yes.
This was a survey of emancipated foster youth
Emancipated youth represent a distinct subset of the foster care population
This “accurate” statistic misleads the reader to conclude that one-third of foster children have been maltreated in care…
50Module 2: Lessons in Using Data
4) Logical “flipping”…Headline in The Anytown Chronicle:
60% of violent crimes are committed by men who did not graduate from high school.
“Flip” 60% of male high school drop-outs
commit violent crimes?
Module 2: Lessons in Using Data 51
“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”
Bath and Haapala, 1993 as cited in “Shattered bonds: The color of child welfare” by Dorothy Roberts
52Module 2: Lessons in Using Data 52
In reading statistics such as the above, there is a tendency to want to directionally “Flip” the interpretation
But the original and flipped statements have very different meanings!
75% of neglect cases involved families with incomes under $10,000
DOES NOT MEAN
75% of families with incomes under $10,000 have open neglect cases
Put more simply, just because most neglected children are poor does not mean that most poor children are neglected
“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”
53Module 2: Lessons in Using Data 53
A study of Anytown residents makes the following claim:
Adults with short hair are, on average, more than 3 inches taller than those with long hair.
Finding an association between two factors does not mean that one causes the other…
Hair Length Height
Gender
X
5) False Causality…
Module 2: Lessons in Using Data 54
“The study, conducted by researchers at the University of California at Berkeley, shows that foster children consistently scored lower in state English and math tests, even when factors such as income, race and learning disabilities were taken into account. ” As reported in USA Today, September 24, 2010
55Module 2: Lessons in Using Data 55
“Foster children struggle to learn…”
56Module 2: Lessons in Using Data
Response to Data Misuse? Continued efforts to frame the data,
educate interested media, policymakers, and others what do these findings mean? how can these data be used to gain
insight into where improvements are needed?
Agencies/child welfare workers must be proactive in discussing both the “good” and the “bad” (be first, but be right). be transparent if not playing offense…playing defense
Using Publicly Available Data to Engage IV-E Students in Research and Statistics: Instructional Modules
QUESTIONS? PLEASE CONTACT: