evaluating complex systems of care a logic model approach to ehdi program evaluation sherry spence,...
TRANSCRIPT
Evaluating Complex Systems of CareA Logic Model Approach to EHDI Program Evaluation
Sherry Spence, MA
Oregon Department of Human Services
Health Services, Office of Family Health
National EHDI ConferenceAtlanta, Georgia, March 3, 2005
Supported in part by funding from the Maternal and Child Health Bureau (Title V, Social Security Act), Health Resources and Services Administration, and by the Centers for
Disease Control and Prevention, US Department of Health and Human Services
Opinions stated do not represent those of CDC, HRSA, or USDHHS
Acknowledgements EHDI Staff at CDC and HRSA EHDI Data Committee Oregon’s EHDI Team
• Claudia Bingham
• Al Ferro
• David Laszlo
• Maureen McNamer
• Amy Rosenthal
Acknowledgements
FamilyNet Executive Sponsors• Katherine Bradley
• Lorriane Duncan Office of Family Health
Epidemiologists and Research Analysts
FamilyNet Project Teams and . . .
And Oregon’s Families
Foreword
“If you torture data sufficiently, it will confess to almost anything.”• Fred Menger (1937- ), chemistry
professor• From A Word A Day (AWAD) -
Foreword
Corollary: “Sometimes you can coerce data by threatening deletion.”• Arthur Epstein, professor of chemistry
and physics (from daughter, Melissa, responding in AWADmail in response to the Menger quote)
Foreword
Conclusion: “There are lies, damn lies, and statistics.”• Mark Twain (1835-1910), author
and humorist
What is our goal for today?
Disprove the Twain/Menger-Epstein hypothesis
Working hypothesis:• It is possible, without false
testimony, torture, or coercion, to develop an evaluation plan using our program data.
What is our goal, really?
What is our goal for today? Define a logical framework for
evaluating EHDI programs Discuss whether our logic model
approach avoids the Twain/ Menger-Epstein condition.
Figure out where our model fits in identifying newborn hearing screening as a best practice.
What are we currently evaluating?Logical framework for EHDI evaluation
HospitalsDiagnostic
FacilitiesEarly Intervention
Facilities
Parent
DHS
Healthcare Provider
Within 10Days of
Screening, Testing or
Referral
6 weeksafter
“Refer”
2 weeks after
Dx Hearing loss if noEI referral
ChildIs
6 weeks of age
Parent
Healthcare Provider
Local Public Health
Parent
Healthcare Provider
Local Public Health
Reminder letters / program contact only when child has not completed the desired EHDI process by the desired time.
Participating facilities, HCP and LPH able to return updated contact information
Monthly reports, program statistics back to facilitiesDemographics, screening, testing and referral information to DHS
DHS Response When:
Screening Facilities
What are we evaluating?
Input
Output
Intermediate Outcome
Intermediate Outcome
Resources & needs
Protocol measure
Performance Measure
Program Measure
EHDI staffing & funding
Report & follow-up completeness &timeliness
Diagnosis, intervention, & risk follow-up
Child served can communicate & learn
What are we trying to prove? Success in achieving EHDI goals
• Early screening of everyone
• Early identification of some
• Early preventive or treatment intervention
• Acquisition of language and communication skills
What else do we need to show?
Success in achieving EHDI goals
Success in achieving health care goals & optimizing resource use
Success in all children achieving their full health and development potential
What are the components of an EHDI logic model?
Input
Output
Intermediate Outcome
High Level Outcome
Resources & needs
Performance measure
Program Measure
Population Measure
EHDI staffing & funding
Screen & Dx completeness & timeliness; follow-up
Children served can communicate & learn
All children communicate & are ready for school
What are the components of an EHDI logic model?
Children interacting and ready to learn
Language and communication skills
Intervention to assure skill acquisition
Early and complete identification and risk monitoring
High level outcomes - population attributes
Intermediate outcomes - client/patient attributes
Outputs - program activities and interventions
What data do we have?
Input
Output
Intermediate Outcome
High Level Outcome
Resources & needs
Performance measure
Program Measure
Population Measure
4.0 FTE, 2 Grants
93% screened 90% screened and linked 70% diagnosed within 3 mos. 7 of 49 in EI (per reports)
NA
NA
What is the approximate timing of this monitoring? All children ready to
interact and learn Language and
communication skills Intervention to assure
skill acquisition Early and complete
identification and risk monitoring
4-5 years after birth
12-36 months after birth
3-6 months after birth
1-3 months after birth
So we won’t know for 4-5 years if this was a good idea?
How can we use this logic model in program evaluation? Conduct research
Conduct research
Monitor program outcomes
Monitor program protocols
4-5 years after birth?
12-36 months after birth?
3-6 months after birth
1-3 months after birth
Will system integration help?
Earlier screening and identification More complete screening and
identification More timely follow-up Less loss to follow-up
Why do we think this is so? About 40% of hearing loss occurs with
other conditions or syndromes, resulting in additional disabilities (Gallaudet Research Institute, 2003)• Additional issues increase likelihood of other
public health program involvement. In Oregon, about 50% of women giving
birth are enrolled in WIC and will enroll their babies• The WIC program keeps track of its highly
mobile population.
How does the logic model work for systems?
Input
Output
Intermediate Outcome
High Level Outcome
Resources & needs
Protocol measure
Performance measure
Performance result
EHDI system require-ments
Strategic plan, design, system works
Facilitation of diagnoses, referrals, and follow-up
Increases number who communicate & are ready to learn
What are our intermediate system outcomes? COORDINATION
CONTINUITY
EFFICIENCY
TIMELINESS
ACCURACY & APPROPRIATENESS
How does this fit with the logic model?
Population success
Program success
Process success
Program Logic Model
System Logic Model
Conclusions
One purpose of a system of services is to assure the success of the individual programs and processes it involves.
One purpose of a data system is to facilitate and enhance the success of the programs that use it.
What do we have so far?
Solid framework for measuring program processes
Solid framework for measuring system processes
So what?
What’s the key to all this?
The logic model works best if the program intervention is an established, evidence-based best practice.
What best practice evidence to we need? Gold standard Controlled
studies by programs
Expert testimony by professionals
Repeated measures and testimony
Immoral or unethical Good evidence from
which we can generalize?
Good evidence or bias?
Eliminates bias and demonstrates generalizability
What evidence do we have?
Prospective and retrospective studies by programs and researchers?
Expert testimony by speech, hearing, and language professionals?
Repeated measures of all kinds?
Thank you.
Thanks to
Amy Rosenthal, Oregon EHDI Coordinator for her participation and to
Scott Grosse, PhD, NCBDDD/CDC, for his participation and for noting an error in slide
32.