creating national performance indicators that are relevant to stakeholders: participatory methods...
TRANSCRIPT
Creating National Performance Indicators that are Relevant to Stakeholders:
Participatory Methods for Indicator Development, Selection, and Refinement
Demia L. Sundra, M.P.H., Margaret Gwaltney, M.B.A., Lynda A. Anderson, Ph.D., Ross Brownson, Ph.D.,
Jennifer Scherer, Ph.D.
Evaluation 2004November 3, 2004
Contributors to Project DEFINE [Developing an Evaluation Framework: Insuring National Excellence]
Evaluation Advisory GroupRoss Brownson, Ph.D. (co-chair)
Alan Cross, M.D.Robert Goodman, Ph.D., M.P.H.
Richard Mack, Ph.D.Randy Schwartz, M.S.P.H.
Tom Sims, M.A.Avra Warsofsky
Carol White, M.P.H.
CDCLynda A. Anderson, Ph.D., Robert Hancock, Demia Sundra, M.P.H.
COSMOS CorporationJennifer Scherer, Ph.D., Margaret Gwaltney, M.B.A. (Currently with
Abt Associates), Thérèse van Houten, D.S.W.,Cynthia Carter
Concept Systems, Inc.Dan McLinden, Ed.D., Mary Kane
Goals of Presentation
Describe the context in which the performance indicators for the CDC’s Prevention Research Centers (PRCs) program were developed
Review the participatory methodology utilized to develop and select national indicators
Discuss the benefits, challenges, and lessons learned from developing performance indicators in an established, diverse, national program
Background and Context
Features of CDC’s Prevention Research Centers Program
33 academic-based extramural research centers across United States
Academic centers partner with community and state organizations to develop, conduct, and disseminate prevention research through participatory methods
Diversity across centers: When founded (newly funded to 18 yrs) Community setting and partners Focus of research
Context for Developing a National Evaluation
National evaluation planning project initiated in response to: Institute of Medicine (IOM) report on the PRC program Support for evaluation at CDC Growth of program and related need for accountability
Project DEFINE Goals (Planning Phase) Engage stakeholders, develop a logic model and
performance indicators, and draft an evaluation plan Maintain a participatory and utilization-focused
approach throughout
Intended Purposes of Performance Indicators
Individual data on each PRC Evaluation Monitoring Technical assistance needs
Cross-center summary data Accountability Program improvement Information sharing and communications with internal
and external stakeholders
Anticipated Challenges in Developing PRC Indicators
Centers strive to achieve diverse health outcomes Program had few previous cross-center
requirements Centers are at various stages of growth and
maturity Interests of diverse stakeholders had to be
considered Concern existed about how performance
indicators would be used Indicators had to be meaningful and impose
minimal burden on PRCs in terms of time and cost
Methodology
Basis of Project DEFINE
Concept Mapping Gained national and community perspectives on
PRC program through 2-tiered approach Engaged diverse stakeholders in brainstorming
statements describing PRC program Statements analyzed to create
visual maps of concepts Concepts used to
build draft logic models
Community and national logic models combined
Engage the Community
Diversity & SensitivityRelationships & Recognition
Active Dissemination
Technical Assistance
Training
Research Methods
Research Agenda
Core Expertise & Resources
Development of Draft Indicators
More than 70 indicators first drafted
Mapped to all components of program logic model
Some indicators dropped, others refined based on input received at regional meetings and PRC contextual visits
52 candidate indicators remained on the list
National, Regional, or Local Health
Priorities and Health DisparitiesEngage
theCommunity
Establisha Research
Agenda
Conduct Core and
Other Research
Using SoundResearch Methods
ProvideTraining, Technical
Assistance, orMentoring
• Researchers• Practitioners• Students• Community
Members
PRC Community Committee
PRC Capacity • Human resources:
core expertise, diversity, sensitivity
• Facility• Communication and
data systems• Administrative
capacity• Evaluation expertise
Relationships with Partners• State and Local Health Departments• Community Partners• University Partners• Other PRCs• CDC
Motivating Conditions for Developing and Maintaining Relationships (e.g., Trust)
Motivating Conditions for Developing and Maintaining Relationships (e.g., Trust)
Research andEvaluation FindingsCommunicated and
Disseminated
• Publications • Presentations• Media• Reports
Research andEvaluation FindingsCommunicated and
Disseminated
• Publications • Presentations• Media• Reports
Trainees andTechnical Assistance
Recipients
Programs and Interventions
OUTCOMES
Improved Community
andPopulation
Health and
Elimination of Health
Disparities
Improved Community
andPopulation
Health and
Elimination of Health
Disparities
EnhancedCommunity
Capacityfor
Prevention
Translation of Research
to Practice and Policy
Widespread Use of
Effective Programs and
Policies
Skilled Public HealthProfessionals
INPUTS ACTIVITIES OUTPUTS
Evaluation
ExpandedResources Recognition
CONTEXTUAL CONDITIONS(e.g., health services and service gaps, socioeconomic conditions)
Stakeholder Recommendations from Regional Meetings
Select a limited number of indicators focused on features common across Centers
Collect data on some components of logic model in other ways as part of the national evaluation, rather than through indicators
Develop indicators through an iterative process, with multiple opportunities for input
Link the performance indicators to the PRC Information System
Stakeholder Selection of the National Performance Indicators
52 indicators listed in structured feedback tool (workbook) All stakeholder groups provided feedback and comments
PRCs, Community, State, and CDC Planned on having core and optional indicators
Results of Performance Indicator Feedback
100% response rate received on workbooks Comments from workbook summarized within
each stakeholder group 3 of the 4 stakeholder groups recommended 8
indicators 2 of the 4 stakeholder groups recommended an
additional 11 indicators
Resulting National Performance Indicators
Evaluation advisory group selected and refined 13 indicators based on Results and feedback of workbook Map of indicators across the logic model Cross-walk of recommended indicators with IOM
report recommendations
Indicators correspond to various logic model components, e.g. Community input on selecting health priorities
(input) Existence of explicit research agenda (activity) Evidence of peer-reviewed publications (output)
Collecting the Information
Performance indicators were integrated into the information system that was in development Conceptualized from the beginning Reinforced through stakeholder feedback
Fields were created in the information system for each performance indicator
Information system was developed and reviewed by: Evaluation Contractors Centers and partners (usability and pilot tests) CDC staff Evaluation Advisory Group PRC Steering Committee
Core and Optional Indicators
Only core indicators developed through Project DEFINE. Consensus allowed us to: Focus on 13 indicators Not use resources for optional indicator development and
integration into information system
11 out of 28 PRCs developed center-specific indicators on their own Topics areas such as
community satisfaction with partnership; funding generated; web site hits; infrastructure measures; research methods appropriate for minority population
PRC Performance Indicators: Summary
Specific component requirements across all grantees
Indicators reflect both process and outcome measures, with focus on process Initial requirements as part of new funding cycle Prospective evaluation
Assess general information across PRCs rather than specific health topics Defining common outcomes, e.g. community capacity
Indicators will be refined during evaluation implementation
Challenges, Benefits, and Lessons Learned
Current Challenges with Performance Indicators
Requests for more guidance on how to further define and collect data
Development of summary reports and provision of feedback to all stakeholders
Need to increase specificity of indicators over time
Balance between participatory processes and program requirements
Buy-in, support, and ownership of indicators Evaluation advisory group was critical for trust
and support from larger stakeholder groups
Community voice is reflected in indicators
Perspective of the PRCs’ staff and partners reflected in utility and feasibility issues surrounding indicators and information system
Benefits of Participatory Approach for Performance Indicator Development
Lessons Learned and Recommendations
Utilize participatory methods for selecting and refining indicators to increase stakeholder support
Build sufficient time into schedule to allow multiple opportunities for stakeholder input
Acknowledge inherent challenge in developing indicators for an established program
Include community input in indicator development to increase accountability to partners
For more information on the Prevention Research Centers Program
http://www.cdc.gov/prc/
Click on “about the program” to view the Conceptual Framework (logic model) and narrative
Contact information: Demia Sundra: [email protected]
PRC IS: General Information Page
PRC IS: Community Committees
PRC IS: Health Priorities