results from the cancer alliance of texas 2015 evaluation august 20, 2015
TRANSCRIPT
Results from the Cancer Alliance of Texas 2015 EvaluationAugust 20, 2015
Roadmap
• Objectives• Methods• Results• Recommendations• Lessons Learned
Background and Evaluation Methods
Charles Shumate, MPH, CHESResearch Specialist
Office of Surveillance, Evaluation and Research
2015 Strategic Planning Focus Groups
4
Objectives:
1. Provide follow-up measurements for domains of coalition functioning.
2. Explore member beliefs on coalition successes, experiences and sustainability.
3. Offer recommendations to coalition and evaluation.
Coalition Domains
5
Empowering Leadership
Value Added Collaboration
Effective Communication
Shared Decision Making
Dedicated Staff
Diversified Funding
Priority Work Plans
2015 Interview Guide
6
Questions
1. What is your role in CAT? How long have you been in this role?
2. In what ways is the Cancer Alliance of Texas (CAT) fulfilling its mission statement?
3. What, if any, expectations did you have when you first joined CAT? How have your expectations changed?
4. If you could change anything about CAT, what would you change?
5. How would you describe your experience as a CAT member/partner? Can you provide examples?
6. Are you engaged in one of the three new priority workgroups? If so how would you describe how these priority workgroups are functioning?
7. Do you have suggestions about how CAT could improve the likelihood of sustainability of its membership? How can we grow and expand our organization?
8. Do you have suggestions about how CAT can diversify its funding base?
9. Do you have suggestions about how CAT could improve the likelihood of its sustainability in its processes or organizational culture?
2015 Coalition Member Survey
7
Evaluation• Planned
– Member survey– Key informant interviews (KII)
• Recruitment– Staff were emailed in February 2015
• Member Roster
– Calls scheduled ~ 30 minutes
• Analysis– Thematic – Summed scores, averages
2015 Coalition Member Survey
8
Evaluation
– Overall survey participation 17% (n=23) • CAT Meeting (n=16)• Online (n=7)
– KII participation 57% (n=21)
– Analysis• Quantitative
– Percentages vs. summed scores• Qualitative
– No change to analysis plan
Participant Characteristics
9
Results
2015 Coalition Member SurveyResults
11
Coalition leaders are actively involved in coalition decisions and efforts
Our coalition efforts are controlled by just a few people
Our coalition maintains clear roles, responsibilities, and procedures
9
13
4
30
4
4
13
17
39
26
43
30
4
22
13
13
13
Reponses(%): Empowering LeadershipStrongly Disagree Disagree Neutral Agree Strongly Agree No response
Empowering Leadership
2015 Coalition Member SurveyResults
12
Our coalition’s efforts do not translate into meaningful influence in the larger community
Our coalition has successfully maintained or increased its credibility
Coalition members distrust one another and / or the leadership
Members believe the coalition works on value-added efforts - things that wouldn’t happen if the coalition didn’t work on them
30
48
39
4
17
9
13
26
17
17
9
22
35
9
13
9
13
35
9
26
Responses (%): Value-Added Collaboration
Strongly Disagree Disagree Neutral Agree Strongly Agree No Resonse
Value-Added Collaboration
2015 Coalition Member SurveyResults
13
Effective Communication
There is little and irregular communication from the coalition leadership to its members
Several modes of communication (such as websites, emails, newsletters and calls) are used to communicate to members
Members feel their voice is heard and have channels to communicate with coalition leadership
Meetings are didactic and mostly focus on reports
26
4
13
48
4
39
9
4
13
22
4
26
22
13
13
13
13
52
48
13
Responses (%): Effective Communication Strongly Disagree Disagree Neutral Agree Strongly Agree No Response
2015 Coalition Member SurveyResults
14
Shared Decision Making
Decisions are made by only a few people
It is understood how decisions are made
Decisions are made based on personal preferences and the loudest voices
13
4
17
35
13
26
13
26
26
26
26
13
13
17
17
13
Responses (%): Shared Decision MakingStrongly Disagree Disagree Neutral Agree Strongly Agree No Response
2015 Coalition Member SurveyResults
15
Dedicated staff is sufficient and effectively coordinated to make progress
The roles and responsibilities of staff are unclear/ not communicated
13
48
43
9
30
4
13
1717 4
Responses (%): Dedicated StaffStrongly Disagree Disagree Neutral Agree Strongly Agree No Response
Dedicated Staff
2015 Coalition Member SurveyResults
16
Our coalition members are motivated and inspired
Members are recruited haphazardly
Our coalition works largely in isolation of the community
New members are welcomed and effectively oriented to the group
Our coalition develops specific roles and responsibilities for members based on their interests and skills
Our coalition structure (workgroups, team leads, etc.) has not changed since our planning days
9
43
43
9
26
4
17
9
13
17
26
35
4
48
48
4
35
4
26
9
17
17
17
13
17
22
17
26
22
Responses (%): Flexible StructureStrongly Disagree Disagree Neutral Agree Strongly Agree No Response
Flexible Structure
2015 Coalition Member SurveyResults
17
Coalition meetings are perceived as unproductive
The roles of members and workgroups are unclear and not communicated
Accountability is valued and evident because the vast majority of members follow through with assignments and meet deadlines
26
22
48
39
9
4
13
17
4
9
39
17
17
2213
Responses (%):Clear Roles and AccountabilityStrongly Disagree Disagree Neutral Agree Strongly Agree No Response
Clear Roles and Accountability
2015 Coalition Member SurveyResults
18
Coalition priorities from the Texas Cancer Plan are set and periodically reviewed and revised
Coalition members lack a clear understanding of the priorities of the coalition
The coalition takes actions that are not related to priorities of the Texas Cancer Plan
The coalition’s efforts are focused and broad enough that members find a place for their interests and skills
Members are involved in developing plans to implement strategies and these plans are written with tasks, timelines, and responsibilities
9
13
13
13
4
57
13
43
48
17
22
13
22
13
4
48
43
13
Responses (%): Priority Work PlansStrongly Disagree Disagree Neutral Agree Strongly Agree
Priority Work Plans
2015 Coalition Member SurveyResults
19
We have a diversified funding base
The resource needs for our implementing our coalition efforts are not known
We have resource plans and seek out new funding opportunities
13
9
13
9
17
48
39
39
4
35
17
22
17
17
Responses (%): Diversified FundingStrongly Disagree Disagree Neutral Agree Strongly Agree No Response
Diversified Funding
2015 Coalition Member SurveyResults
20
Other Issues
The coalition is addressing the disparity in cancer control services in Texas
My “home” organization has benefited from the Cancer Alliance of Texas
9
13
9
13
48
26
21
35
13
94
Responses (%): Other IssuesStrongly Disagree Disagree Neutral Agree Strongly Agree No Response
21
Take Aways From KIIs
Question Themes Illustrative Quotes
Fulfilling Mission • Products• Awareness• Connections• Networking• Texas Cancer Plan
“It [CAT] has helped me stay focused to make decision about the direction of my own work so that [it] is better aligned with the priorities set forth in the plan [TCP].”
Expectations • Need for awareness
• Networking opportunities
• Work requirement• Inherent value
“To learn what others were doing and opportunities to collaborate, to help foster efforts towards to cancer control.”
Changes to CAT
• Transparency• Location• CPRIT• Increase
engagement• Topics
“[How are] meetings scheduled. There are no doodle polls. [It is like] ‘this is when they are’. That is frustrating.”
“I do wish they [CAT] were a bit friendlier at remote attendance. [You] can’t always make it in person and to attend in remotely, not conference, but being able to watch slides on a computer and be more [engaged].”
22
Continued
Question Themes Direct Quotes
Member Experience
• Rewarding • Tension
“Very fulfilling, glad to be engaged and good to work with people. None of these people are paid people.”
“It kind of lacks engagement sometimes. [Occasionally] there is a meeting; it doesn’t make me want to go. I missed the last meeting. There is no hook to get me to go. That is how it has impacted my participation.”
Benefits of Priority Work Groups (PWGs)
• They are working• Needs
improvement
“It’s going well given the time crunch because everybody is a volunteer. We don’t get paid.”
Benefits of PWGs
• Implement TCP• Opportunity to
Contribute• Produce
deliverables
“The idea is to move the model. To implement the plan.”“Get some concrete time where there can be discussion and planning on how to move the needs and actually strategize the use of CATs resources to accomplish [things].”“Again, providing additional tools for people, to get best practices to patients and technical assistance for their home organization.”
23
Continued
Question Themes Direct Quotes
Membership • Provide meaningful engagement
• Quality time• Not a problem,
membership is strong
“Everybody has to make hard decision about spending time. If I am getting something out of things and my contributions is important, [I am] more likely to attend and more likely to stay.”“Need to go back to dedicated time for members. Not just business updates. I don’t know what that would look like [maybe a] once a year kind of thing.”“[This topic is a] broken record. We have sustained interest by good people who are doing their part. I don’t think we are losing.”
Funding • 501c3• DSHS• Grants
“[This is an ongoing] discussion topic. Unless we choose to be something that we are not. Should we become 501c3? Should we have a foundation? Once you get into that, that Jockeying between doing the mission and seeking funding. We need to decide who we are and what we can do. Form follows function. We would have to have a big change in overall perceptive [regarding 501c3 status].”“Department of State Health Services needs to give more money. DSHS should have a bigger buy in.”
Culture • Transparency• Maintain status
quo
“And yeah would not hurt to review bylaws. You know the roles different seats [executive committee] hold. Often times they [members] don’t want to run and are afraid to nominate someone since they don’t know right person to nominate to run. Those kinds of things.” “How reflective how the organization is, how will CAT learn from this evaluation of the organization. How and what extent will the organization look at results and make changes related to the results? That is very important in terms of sustainability.”
Lessons Learned
Results Support
• CAT is fulfilling its mission and provides members with important networking opportunities and information on cancer control efforts at the state and national level.
• CAT needs to increase coalition transparency. Idea was expressed by leadership and rank and file membership alike. Transparency was woven throughout interviewee responses.
25
• CAT members identified many accomplishments in both the survey and interviews, it became apparent that CAT lacks a process to document its successes, promote itself to the external community, or internally to new members.
• CAT should increase opportunities for rank and file members to have a voice in meeting activities, diffusion of workgroup responsibilities, and sustain post quarterly meeting engagement using the existing communication channels.
26
Results Support
27
• Review CAT mission, goals and bylaws
• Provide members “member time” at CAT meetings
• Document and share CAT success/accomplishments
• Utilize breakout sessions to address member driven topics
Results Support
28
Evaluation Process
• Focus the member and partner survey to a few domains that are of CAT and DSHS interest.
• Explore evaluation of CAT networking strength
Limitation Solutions to minimize
Data accuracy Assess reasonableness of the dataUnknown data elements Focused on percentage rather than
changes in absolute valuesSelection bias Ensuring the sample was appropriateConfidentiality Obtain de-identified data Small sample size Focus on data as illustrative of case
study vs. in-depth/inferential statistics
Limitation and mitigation to existing data validity threats
Discussion Points
• What surprises you about results?
• What doesn’t surprise you?
• What other information about the evaluation would be useful to you?
29
