susan g. komen continuum of care evaluation course ... · susan g. komen continuum of care...
TRANSCRIPT
Susan G. Komen Continuum of Care Susan G. Komen Continuum of Care Evaluation Course Presentation Evaluation Course Presentation
Janet A. Phoenix, MD, MPHJanet A. Phoenix, MD, MPH Senior ConsultantSenior Consultant
October 13 and 20, 2011October 13 and 20, 2011
The Grant Group LLCThe Grant Group LLC
. . 342 Orchard St. NW Suite 201, Vienna, VA 22180342 Orchard St. NW Suite 201, Vienna, VA 22180--41414141..
Corporate HQ Tel: 301.325.8850 Corporate HQ Tel: 301.325.8850 ●●
www.thegrantgroupwww.thegrantgroup--llc.comllc.com 1
Evaluation OverviewEvaluation Overview
Definition: Definition:
Program evaluation is carefully collecting Program evaluation is carefully collecting information about a program or some information about a program or some aspect of a program in order to make aspect of a program in order to make necessary decisions about the program. necessary decisions about the program.
2
•Needs Assessment•Cost Benefit Analysis•Formative•Summative
3
Examples
Evaluation GoalsEvaluation Goals
Understand and increase the impact of products Understand and increase the impact of products or servicesor services
Improve delivery mechanisms to be more Improve delivery mechanisms to be more efficient and less costlyefficient and less costly
Verify that the program is achieving intended Verify that the program is achieving intended goals and objectivesgoals and objectives
4
Essential ElementsEssential Elements
FundingFundingConsumer ResponseConsumer ResponseUnmet needUnmet needImpact on design of future Impact on design of future
programs/activitiesprograms/activitiesCapacityCapacity
5
6
Considerations
•Purposes•Audience•Information needed for decisionmaking•Information needed to better understand program/process•Sources •Data collection methods•Timeframe•Resources for evaluation
AudiencesAudiences
FundersFunders
Board of DirectorsBoard of Directors
ManagementManagement
Program StaffProgram Staff
ConstituentsConstituents
7
8
Sources of Information
•Surveys•Interviews•Case Studies•Focus Groups•Existing Materials
Data Collection MethodsData Collection Methods
Questionnaires/SurveysQuestionnaires/Surveys
Quick and nonthreatening; may be able to use Quick and nonthreatening; may be able to use existing data; easy to compareexisting data; easy to compare
Not open ended; lacks complete informationNot open ended; lacks complete information
InterviewsInterviews
Gives comprehensive views; Gives comprehensive views;
Costly; hard to administer; interviewer biasCostly; hard to administer; interviewer bias
Document ReviewDocument Review
DoesnDoesn’’t interrupt the programt interrupt the program
Can take a lot of time; inflexibleCan take a lot of time; inflexible
9
Method Selection CriteriaMethod Selection Criteria
1. What information is needed to 1. What information is needed to make current decisions about a make current decisions about a product or program?product or program?
2. Of this information, how much can 2. Of this information, how much can be collected and analyzed in a lowbe collected and analyzed in a low--
cost and practical manner, e.g., using cost and practical manner, e.g., using questionnaires, surveys and checklists?questionnaires, surveys and checklists?
10
Method Selection CriteriaMethod Selection Criteria
3. How accurate will the information be?3. How accurate will the information be?
4. Will the methods get all of the needed 4. Will the methods get all of the needed information?information?
5. What additional methods should and could be 5. What additional methods should and could be used if additional information is needed?used if additional information is needed?
11
12
6. Will the information appear as credible to decision makers, e.g., to funders or top management?
7. Will the nature of the audience conform to the methods, e.g., will they fill out questionnaires carefully, engage in interviews or focus groups, let you examine their documentations, etc.?
8. Who can administer the methods now or is training required?
9. How can the information be analyzed?
Process Based EvaluationProcess Based Evaluation
How do employees and/or the customers decide How do employees and/or the customers decide that products or services are needed?that products or services are needed?
2. What is needed from employees in order to 2. What is needed from employees in order to deliver the product or services?deliver the product or services?
3. How are employees trained about how to 3. How are employees trained about how to deliver the product or services?deliver the product or services?
4. How do customers or clients come into the 4. How do customers or clients come into the program?program?
13
Process Based EvaluationProcess Based Evaluation
5. What is required of customers or 5. What is required of customers or client?client?
6. How do employees select which 6. How do employees select which products or services will be provided products or services will be provided to the customer or client?to the customer or client?
14
Process Based EvaluationProcess Based Evaluation
7. What is the general process that customers or 7. What is the general process that customers or clients go through with the product or program?clients go through with the product or program?
8. What do customers or clients consider to be 8. What do customers or clients consider to be strengths of the program? strengths of the program?
9. What do staff consider to be strengths of the 9. What do staff consider to be strengths of the product or program?product or program?
15
16
10. What typical complaints are heard from employees and/or customers?
11. What do employees and/or customers recommend to improve the product or program?
12. On what basis do employees and/or the customer decide that the product or services are no longer needed?
17
Australian Breast Cancer National Accreditation Standards
Program AIMS
1. To ensure that the Program is implemented in such a way that significant reductions canbe achieved in morbidity and mortality attributable to breast cancer.2. To maximise the early detection of breast cancer in the target population.3. To ensure that screening for breast cancer in Australia is provided in dedicated andaccredited Screening and Assessment Services as part of the BreastScreen Australia Program.
18
Australian Breast Cancer National Accreditation
StandardsProgram AIMS
4. To ensure equitable access for women aged 50−69 years to the Program.
5. To ensure that services are acceptable and appropriate to the needs of the eligible population.
6. To achieve high standards of program management, service delivery, monitoring and evaluation, and accountability.
19
Australian Breast Cancer National Accreditation StandardsProgram OBJECTIVES
1.
To achieve, after five years, a 70 per cent participation rate in the BreastScreen Australia Program by women in the target group (women aged 50−69 years) and access to the Program for women aged 40−49 years and 70−79 years.
2. To rescreen all women in the Program at two-yearlyintervals.
20
Australian Breast Cancer National Accreditation StandardsProgram OBJECTIVES
3. To achieve agreed performance outcomes which minimise recall rates, retake images, invasive procedures, ‘false negatives’, and ‘false positives’, and maximise the number of cancers detected, particularly the number of small cancers.
4. To refer to appropriate treatment services and collect information about the outcome of treatment.
21
Ways that the Breast Screening Centers Differ
•Structure of the Administration•Quality Assurance Procedures•Protocols for Women with Family History•Use of Clinical Staff•Data Collection Procedures•Forms•Manner of Issuing Results
22
Challenges Addressing the Needs of Symptomatic Women
(1)BreastScreen is not resourced to investigate women with symptoms, all of whom require clinical breast examination (and often ultrasound and biopsy) in addition to mammography.
(2) Women and their general practitioners may be falsely reassured by a negative result and not pursue the essential further investigation.
23
Challenges Addressing the Needs of Symptomatic Women
(3) Cancers in symptomatic women may be incorrectly reported as ‘screen detected’, artificially increasing the reported cancer detection rate of BreastScreen.
(4) ‘Screen-detected’
symptomatic cancers inflate the Breast-Screen average tumour size, as these cancers are likely to be larger than screen-detected asymptomatic cancers, making it more difficult to achieve accreditation standards requiring a high percentage of cancers to be small.
24
What Group of Women Should Not be Receiving Breast Screen
Services?
•High Risk Women due to Family History•High Risk Women with Genetic Mutations•Women with a History of Breast Disease•Women with Previous Breast Cancer (due to need for annual mammography and clinical breast examination
25
Colorectal Cancer Screening Guidelines
•Colonoscopy
•Every 10 years
•Flexible Sigmoidoscopy
•Every 5 years
•Fecal immunochemical test
•Every year
•Guiaic fecal occult blood
•Every year
26
DIFFERENCES BETWEEN TRADITIONAL COLONOSCOPY ANDCT COLONOSCOPY
•No need for sedation for CT colonoscopy•Both require bowel preparation•Faster to perform than traditional
colonoscopy•Lower in cost than traditional colonoscopy•Radiation risks in CT colonoscopy
27
Outcome Based Evaluation: Why Perform One?
•To determine whether your program is making a difference for clients
•To supplement a process based evaluation
•To examine and quantify the impact and/or benefits for clients
•To relate investment in the program to benefits for participants
28
Outcome Based Evaluation Definition and Components
Definition:
Outcome based evaluation examines the effect on clients of programs and services by describing the short term, intermediate and long term impacts.
Components:
•Inputs•Activities•Outputs•Outcomes•Outcome Targets•Outcome Indicators
29
Outcome Based Evaluation Components
Inputs:
Materials and resources that the organization uses to carry out the program:
•Equipment•Staff•Volunteers•Facilities•Donations
30
Outcome Based Evaluation Components
Activities:
Processes that the program uses to meet client’s needs and achieve the program objectives:
•Screening•Education•Counseling•Making Referrals•Collecting data
31
Outcome Based Evaluation Components
Outputs:
Quantifiable units of services that are provided by your program:
•Number of clients screened
•Numbers of clients who received CBE
•Numbers of patients referred to an oncologist
32
Outcome Based Evaluation Components
Outcomes:
The actual benefits or changes that program participants experience during or afterwards:
•Number of patients who have lesions identified•Number of patients who are referred to an oncologist•Number of patients who begin treatment as a result of the program•Number of patients whose breast cancer is identified at an early stage
33
Outcome Based Evaluation Components
Outcome Targets:
The number and/or percentage of participants that you have set as a goal to receive the outcome:
•80% of women eligible will receive a mammogram•60% of women who have cancer identified will be at stage one•90% of women will receive clinical breast examination during routine visits
34
Outcome Based Evaluation Components
Outcome Indicators:
Observable and measurable milestones that mark progress along the path to the outcome:
•Number or percentage of participants who receive a referral to an oncologist
•Number or percentage of participants who receive diagnostic imaging services
•Number of patients who return for a post treatment follow up visit
35
Susan G. Komen Evaluation Guidelines
Evaluation(limit –
3,500 characters)
•
Describe in detail how the organization (s) :
will measure achieving project goals/objectives &how the impact of the project on the priority selected be assessed.
•
Describe the evaluation expertise that will be available for this purpose.
•
What resources are allocated for evaluation in the project budget?
36
Susan G. Komen Evaluation Approach and Components
Approach: Measure both the quality and quantity of strategy implementation and outcomes:
Components:
Impact Evaluation: Assesses the changes that can be attributed to a particular intervention, such as a project, program or policy. Impact Evaluation helps us to answer key questions such as, what
works, what doesn’t, where, why and for how much?
Process Evaluation: Assesses the delivery of programs. Process evaluation verifies what the program is and whether it is being implemented as designed. It answers the questions of what is delivered in reality and where are the gaps between program design and delivery?
37
TIPS FOR COLLABORATIONAND TEAM BUILDING
Begin With Co-Hosting
•Manage a stakeholder process with sponsorship from more than 1
organization
38
Create a Game Plan and Group Covenants
•Discuss how much time people can devote to the group process
•Communicate about alternates, attendance and logistics
39
Concentrate on Relationships First
•Let people get to know each other as individuals
•Share a meal
•Have people explain the impact of decisions or agreement on them and their
community or organization
40
Be Transparent About Decision Making
•Clarify who will make decisions
•Determine how representation will be
established•Ask stakeholders to identify
when they are speaking officially or unofficially
41
Pay Attention to Power
•Establish who has the authority for the group to control what information gets considered
•Who has the authority for the group to decide what’s not
important
42
Create Rituals
•Give members a sense of identify through small habits
•E.g. Songs, Homemade foods, celebrating birthdays
43
Talk About Values
•Talk about the values participants bring to the tables
•Do this before talking about problems, data or potential
solutions
44
Acknowledge Different Kinds of Knowledge
•Acknowledge and support differences in the group in
communication and ways of knowing
45
Generate Multiple Problem Definitions
•All definitions of the problem need to be on the table
•
No definition is wrong because they all reveal issues and hopes
46
Explore Validity and Accuracy With Care
•All information should be questioned as to its validity, accuracy, authenticity and
reliability•This is true regardless of the
source whether scientific, technical, traditional or cultural