Scaling up to a population health

intervention:

A readiness assessment framework

Duyen Thi Kim Nguyen, MSc (Psychology)Doctoral student (Population & Public Health)

Department of Community Health Sciences d.t.nguyen@ucalgary.ca

PHIRNET Student Webinar

Purpose of this webinar

To share my study purpose, research questions, background, and methods with the PHIRNET community

To obtain constructive feedback from trainees and experts in the field of population health, intervention research, evaluation, and scale up

To provoke discussions regarding knowledge gaps in scaling up and how to advance PHIR methods

Purpose of my dissertation

To develop a readiness assessment framework to assist stakeholders to determine if a health intervention is ready to scale up to a population-level intervention.

ready

stakeholder

health

intervention

scale up

population-level

Key terms:

Research Questions

1) How are health interventions initially scaled up successfully to population-level interventions?

2) How are health interventions sustained successfully as scaled up population-level interventions?

3) What are the key elements of successfully scaled up population-level interventions ?

Phase 1: Scoping Review

4) What can we learn by piloting the readiness assessment framework on an existing community-level intervention?

Research Questions cont’d

Phase 2: Case study evaluation (CSE) Case = Stardale

Phase 3: Test and refine the framework

Background

“Canada is a country of perpetual pilot projects” (Bégin et al., 2009, p. 1185)

Not necessary for all interventions to be scaled up (e.g., programs treating individuals)

Interventions using upstream approaches should be considered

PHI have potential for great health impact Root causes Reducing inequities and incidence

Types and paths of scaling up (Uvin, 1995)

Scaling up

Mangham & Hanson (2010) literature search in June 2008 PubMed: ‘scaling up’ & ‘scaling-up’ 1989-2001 = 2 citations re. scaling up & health interventions 2001-2008 = 51 citations July 2008-Feb 2012 = 132 citations

Scaling up Canada and globally

Conference to Advance the State of the Science and Practice on Scale-up and Spread of Effective Health Programs (2010)

Phase 3: Test and refine the framework

-Assess its utility & practicality (Stardale)-Feedback & assess validity (Experts)

Phase 3: Test and refine the framework

-Assess its utility & practicality (Stardale)-Feedback & assess validity (Experts)

Phase 2: CSE

-Provide evaluation report -Greater understanding of

Stardale’s processes, outcomes, and more

Phase 2: CSE

-Provide evaluation report -Greater understanding of

Stardale’s processes, outcomes, and more

Phase 1: Scoping Review

-Develop a Readiness Assessment Framework

-Expert feedback and assess validity

Phase 1: Scoping Review

-Develop a Readiness Assessment Framework

-Expert feedback and assess validity

Readiness assessment framework to assist stakeholders to determine if a health intervention is ready to scale up to a population-level intervention

Scoping Review: 101

Systematic review Focused question

Hypothesis-testing

Systematic & explicit method

I/E Criteria defined a priori

Study filters

Formal quality assessment

Synthesize the data to provide evidence in support/rejection of the research hypothesis

Scoping review Broad question

Assessing scope & breadth

Less constrained & iterative

I/E Criteria defined post hoc

Typically no study filters

Quality assessment (optional)

Examine extent, range, & mapping; Value of a sys review; Summarize & disseminate; Gaps

Phase 1: Conceptual Framework

Scoping: Arksey & O’Malley (2005); Levac et al, (2010)

Readiness Assessment Framework: Population health approach

Reducing health inequities and improving health among population groups (aggregate)

Focus on broad range of underlying conditions (PHAC, 2001)

Complex Adaptive Systems “a collection of individual agents”; “not always

predictable”; interconnected (Plesk & Greenhalgh, 2001, p. 625)

Individ properties ≠ Collect. properties (Matheson et al., 2009)

1) Database – PubMed, Medline, EMBASE, Global Health, Canadian Health Research Collection, CINAHL, PsycINFO, Social Services Abstract, Social Work Abstract, Cochrane Database of Systematic Reviews, ERIC, EconLit, & WHO Library and Information Networks for Knowledge Database (WHOLIS)

2) Grey literature - Scopus, select conferences regarding scale up (e.g., Conference to advance the state of the science and practice on scale-up and spread of effective health programs) and scale up studies and evaluations of organizations, companies, and government initiatives (e.g., ExpandNet)

3) Authors will be contacted for clarification, data, or further studies

4) Experts and research teams identified during the review process will be contacted for information regarding their knowledge of ongoing or unpublished studies or other consultation

5) Bibliography list of identified relevant publications; Select journals

Quality Assessment Tools

Downs & Black (1998) Checklist Randomized & non-randomized studies Scores fro overall study quality, quality of reporting, power, internal

validly (bias & confounding), and external validity

Critical Appraisal Skills Programme (CASP) appraisal checklist (Public Health Resource Unit, England, 2006)

10 questions regarding rigour, credibility, and relevance

Quality in Qualitative Evaluation (Spencer et al, 2003)

Focus on qualitative evaluation research

Search on the Cochrane Qualitative Research Methods Group

Categorization for organization & analysis

Research Questions:

Underlying condition(s)Underlying condition(s)

Population characteristics

Population characteristics

NVivo 9

Health outcome(s)

Health outcome(s)

ContextContext

ComplexityComplexity

Characteristics of Stardale

Characteristics of Stardale

Theory behind the

intervention

Theory behind the

intervention

Sustainability processes

Initial processes

Key elements

Key elements

Expert feedback

Experts (academic, practice) Population health Intervention research Scaling up

Surveys and/or semi-structured interviews Face validity Content validity (main domains present) Feedback re. strengths, weaknesses, gaps

Phase 2: Case Study Evaluation

Stardale: Honouring the girls stories

Non-profit organization and registered charity

Honouring the Girls Stories (est. 2005) in Calgary, AB

Provides life skills, literacy education, and advocacy to urban Aboriginal female youths (10-17 years)

Organized activities afterschool (e.g., arts, tutoring, sporting events, dance, acting, modeling)

2/wk from Sept-June; 6:00p-8:00p; with supports all year-round

Participatory Approach

Doing research “with”, not “on” the participant (Minkler & Wallerstein, 2003)

Participatory = research design & implementation (Merriam, 2009), developing sense of ownership, create meaning, and increase likelihood of using the results (Patton, 2002)

Principles: Flexible design, building theory based on participants perspectives, simplicity, agreed topics, analysis reflects the detail and diversity, & share results with participants for informed conclusion (Rothe, 2000)

Aboriginal Research/Evaluations

Tri-council guidelines re. Research involving the First Nations, Inuit and Métis peoples of Canada (Gov of Canada, 2011)

Aboriginal advisory committee

Key components (Johnston, 2008): Unique way of knowing Explaining in relevance to Aboriginal culture In-person, verbal communication Acknowledge Aboriginal core values Time

Case Study Approach

Case study method ideal when: addressing “why” and “how” questions; lack of control over events; contemporary phenomenon (Keen & Packwood, 1995; Yin, 2009)

CSE Framework (Yin, 1992; 2009): Develop understanding of the program & combine with

literature Define units of analysis; Create timeline Define measures, protocols, & procedures Data collection, analysis, and synthesis; Compose a report

Reflection: Myself and Stardale stakeholders (i.e., staff)

Phase 3: Test and refine the framework

Research Question:4)What can we learn by piloting the readiness assessment framework on an existing community-level intervention?

Pilot by Stardale staff and stakeholders

Semi-structure interview for framework feedback Practicality Function Comprehension Feedback re. strengths, weaknesses, gaps

Establishing Validity

Experts (different from study 1) Population health Intervention research Scaling up

Semi-structured interviews Face validity Content validity (main domains present) Feedback re. strengths, weaknesses, gaps

Phase 2: Case Study Evaluation

-Greater understanding of a community-level

intervention, Stardale

Phase 2: Case Study Evaluation

-Greater understanding of a community-level

intervention, Stardale

Phase 1: Scoping Review

-Develop a Readiness Assessment Framework

-Expert feedback and assess validity

Phase 1: Scoping Review

-Develop a Readiness Assessment Framework

-Expert feedback and assess validity

Readiness assessment framework to assist stakeholders to determine if a health intervention is ready to scale up to a population-level intervention

Phase 3: Test and refine the framework

-Assess its utility and practicality (Stardale)-Feedback & assess validity (Experts)

Phase 3: Test and refine the framework

-Assess its utility and practicality (Stardale)-Feedback & assess validity (Experts)

Select References

Arksey, H. & O’Malley, L. (2005). Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, 8(1):19-32.Edwards, N. (2010). Scaling-up health innovations and interventions in public health: a brief review of the current state-of-the-science. Draft Paper Commissioned for the Conference to Advance the State of the Science and Practice on Scale-up and Spread of Effective Health Programs. Available from: http://ihiscaleupconference10.blogspot.com/2010/06/june-25-introductory-call.htmlJohnston, A. L. K. (2008). Aboriginal ways of knowing: Aboriginal-led evaluation. The Canadian Journal of Program Evaluation, 23(1), 1-6.Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science; 5:69. Available from: http://www.implementationscience.com/content/5/1/69Mangham, L. J. & Hanson, K. (2010). Scaling up in international health: What are the key issues? Health Policy and Planning, 25(2): 85-96. doi:10.1093/heapol/czp066Matheson, A., Dew, K., Cumming, J. (2009). Complexity, evaluation and the effectiveness of community-based interventions to reduce health inequalities. Health Promotion Journal of Australia, 20(3): 221-226.Uvin, P. (1995). Fighting hunger at the grassroots: Paths to scaling up. World Development, 23(6): 927-939.Yin, R. K. (2009). Case study research. California: Sage.

Literature reviews:Bambra, C. (2011). Real world reviews: A beginner’s guide to undertaking systematic reviews of public health policy interventions. J Epidemiol Community Health, 65, 14-19.Grant, M. J. & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91-108.Jackson, N. & Waters, E. (2005). Criteria for the systematic review of health promotion and public health interventions. Health Promotion International, 20, 367-74.Riaz, M., Sulayman, M., Salleh, N., & Mendes, E. (2010). Experiences conducting systematic reviews from novices’ perspective. Avail from: http://www.cs.auckland.ac.nz/~norsaremah/EASE2010.pdf

Evaluation research:Hawe, P., Degeling, D., & Hall, J. (1990). Evaluation health promotion: A health worker’s guide. Australia: Elsevier.Weiss, C. H. (1998). Evaluation, 2nd ed. New Jersey: Prentice Hall.

Qualitative research:Denzin, N. K. & Lincoln, Y. S. (Eds). (2011). The SAGE handbook of qualitative research. Thousand Oaks: SAGE. Merriam, S.B. (2009). Qualitative research: A guide to design and implementation. San Francisco: Jossey-Bass.Stake, R. E. (2010). Qualitative research: Studying how things work. New York: Guilford Press.

Helpful Resources re. PHIR Methods

Thank You!Supervisors: Drs. Lynn McIntyre & Lindsay McLarenSupervisory committee and members of PHIRC at UofC

Funding for my dissertation has been provided by the CIHR Population Health Intervention Research Centre PhD Scholarship.

Funding for my internship at the WHO has been provided by the CIHR Population Health Intervention Research Network Internship Scholarship.

Acknowledgements

A CIHR Centre for Research Development in Population Health