process and outcome evaluations zha blong xiong, ph.d. associate professor department of family...
TRANSCRIPT
Process and Outcome Evaluations
Zha Blong Xiong, Ph.D.Associate Professor
Department of Family Social ScienceUniversity of Minnesota
Process Evaluation Design
• Data for the process evaluation were based on a cross-sectional design using a mixed method approach. Specifically, observation, survey, face-to-face interview, and personal journal were used to collect data from the advisory team and staff to determine STEEP’s strengths and weaknesses.
The Dandelion Effect Approach
• The dandelion effect approach is based on the idea that to prevent tobacco use in the Southeast Asian community it is essential to:– build the capacity of the community by providing on-going
partnership opportunities, mentoring, and support for members of the community or volunteers to deliver the tobacco education;
– use innovative but culturally-tailored, multi-approached education to bring awareness to the people and agencies in the SEA community; and
– engage the community to bring change to socio-cultural norms and practices.
Strengths
• Members of the Collaborative– have participated in various local and national
conferences, training, and workshops on tobacco control, coalition building, program sustainability, and financial accountability.
– are passionate about the collaboration and continue to maintain a strong commitment to serve the community.
– are willing to look beyond their differences to work for a common goal.
– willing to share resources and sacrifice for the community.
Strengths
• Staff– Have four full-time Southeast Asian staff whose work focuses
exclusively on tobacco education and engagement. They are passionate about the work, motivated to change community norms and policies, and committed to help each others.
– Data show that the staff’s confidence level has increased from a “moderate” level to a “high” level. • Their ability to teach about tobacco issues has increased from a
score of “low” to a score of “moderate” to “high.” • On average, they evaluated the effectiveness of the training
program as “quite effective” in regards to tobacco knowledge and presentations.
Strengths
• Feedback data from program participants found the educators to be knowledgeable about the subject matter (mean = 9.02; SD = 1.10).
• Program participants also reported that the educators responded very well to the audience’s questions (mean = 9.26; SD = .92).
• The majority of the program participants rated the training favorably as to how it was presented (mean = 9.20; SD = .90) and how valuable the training was for their work (mean = 8.85; SD = 1.39).
Limitations and Challenges• Collaborative Board:– Not everyone has a chance to participate in all of
the conferences, training, and workshops.– Concept of time is viewed differently by different
collaborative members. • Showed up late at meetings
– Financial accountability and reporting• Management practices – reallocating STEEP money to
cover other projects for sustainability reasons.• Accounting practices – the practice of simplifying the
accounting system without complying to the fiscal agent’s requirements.
Limitations and Challenges
• Collaborative Board:– Different levels of commitment from the collaborative
members.• Hurried life, limited time.• Unequal staff size and experience between organizations.• Lack of funding for smaller agencies.• “A new problem we are facing now is that each partnering
organization has faced its own financial problems. Not [only the] tobacco budget, but also budget from our funders. All of us have to cut something, and we have to work harder. If you have that kind of problem, you tend to have less commitment to STEEP. For agencies with fewer staff, EDs have to work a lot harder in their own agencies.”
Limitations and Challenges
• Staff– The challenge of training and retaining volunteers
without any monetary incentives.– The challenge of record keeping and documenting
information to demonstrate the levels of penetration and impact in the community.
– The challenge of seeking alternative funding sources to help sustain the collaborative beyond the current funding stream.
Outcome Evaluation Design• Based on a cross-sectional evaluation study using a retrospective
design to follow up with volunteers, program participants, and partners who have been involved with STEEP in the past.
• A sample of 190 (53.7% male, 44.7% female) where 28% were Cambodian, 26% were Hmong, 32% were Lao, 6% were Vietnamese, and the remaining 8% were from other ethnic groups.
• Assessed five areas: – tobacco knowledge, – beliefs about tobacco use, – norms around tobacco use, – visibility of STEEP in the community, and – effect s STEEP has on the participants.
How much have we been penetrating the SEA communities?
• STEEP’s staff have attended 65 events where Asians were present (or where there were a high concentration of Asians in attendance).
• The educators have delivered their messages to 40,000 people in three major target Southeast Asian locations: Twins Cities, Rochester, and St. Cloud.
• The educators have recruited and trained 120 volunteers.– Of the 120 volunteers, 20 have done some education in
their respective community.
How much have we made a difference in the SEA communities?
• All collaborative agencies have adopted a policy that all agency sponsored activities be tobacco-free.
• All collaborative agencies have adopted a policy not to accept money from tobacco companies.
• Worked closely with St. Paul Parks to adopt a policy not to smoke in parks if youths under the age of 18 are present.
• As a result of STEEP’s involvement, a new policy has been signed into law in Minneapolis prohibiting people from smoking at play grounds, swimming areas, and other designated areas.
• The Planning Committee of the World Refugee Day has verbally agreed for two years (2009 – 2010) to have a smoke-free event.
• The Dragon Festival Planning Committee has verbally agreed for 2 years (2009-2010) to have a smoke-free event.
How much do the people in the community know about STEEP?
Have heard people talking about STEEP's workin community (n=183)
16.3
28.4
42.1
9.5
0
1020
3040
50
No Once awhile
Quite often Every time
How much have we made a difference in the SEA communities?
7.9 14.7
92.183.2
0
20
40
60
80
100
STEEP changedpersonal thoughts
about tobacco(n=190)
STEEP has changedcommunity thoughts
about tobacco(n=186)
No
Yes
How much have we made a difference in the SEA communities?
Have seen changes in community aboutsmoking in past two years (n=186)
18.9 22.6
48.9
7.4
010
203040
5060
No A little Some what Very much
How much have we made a difference to the people we educate?
Pre-test Assessment Post-test Assessment
Question N Mean SD N Mean SD
How important is tobacco as a health issue? 39 8.69 2.20 41 9.41 .92
How much do you know about tobacco? 39 7.21 2.45 40 8.47 1.52
How much do you want to tell others about
tobacco?
39 7.72 2.53 41 8.34 1.76
How much more did you learn about tobacco
today?
40 8.58 1.53
How much have we made a difference to the people we educate?
The same now vs. before you knew STEEP A little different now vs. before you knew STEEP
Much different now vs. before you knew STEEP
Answer to these questions be... (N=180)
0
10
20
30
40
50
60
70
11.1
64.7
18.9
How much have we made a difference to the people we educate?
Same as now vs. before you knew STEEP A little different now vs. before you knew STEEP
Much different now vs. before you knew STEEP
Answer to these questions be… (n=188)
0
10
20
30
40
50
60
70
17.9
57.4
23.7
How much have we made a difference to the people we educate?
Same as now vs. before you knew STEEP A little different now vs. before you knew STEEP
Much different now vs. before you knew STEEP
Answers to these questions (Harmless, weight, stress) (n=190)
0
10
20
30
40
50
60
70
17.9
57.9
24.2
How much have we made a difference to the people we educate?
(n=187) (n=107)Have rules about not smoking inside home Have rules before exposure to STEEP
0
10
20
30
40
50
60
70
80
60.5
72.2
36.3
20
1.6 0.9
yesnonot sure
How much have we made a difference to the people we educate?
Post signs Tell them to smoke outside if they smoke
Let children tell them there is no smoking in the house if they
smoke
Other0
10
20
30
40
50
60
70
80
90
12.2
79.1
3.5 1.7
How to let others know about no smoking (n=111)
How much have we made a difference to the people we educate?
13.2
38.9
11.1
22.6
8.9 12.1 15.3
34.2
79.5
50
81.1
60.5
82.173.2 71.6
48.9
5.3 5.8 4.2 8.4 6.8 8.9 10 11.1
0
1020
30
4050
60
7080
90
(n=186) (n=180) (n=183) (n=174) (n=186) (n=179) (n=184) (n=179)
Nowadays A yr ago Nowadays A yr ago Nowadays A yr ago Nowadays A yr ago
Inside restaurants Indoor worksites Inside publicbuilding
Inside funeralhomes
yes
no
not sure
How much have we made a difference to the people we educate?
8085.3
60 62.6
47.9 51.6 51.1 52.6
7.92.1
12.68.9
36.3 33.237.4
31.6
7.9 5.3
23.7 22.1
11.1 8.4 9.5 11.1
0
10
20
30
40
50
60
70
80
90
(n=182) (n=176) (n=183) (n=178) (n=181) (n=177) (n=186) (n=181)
Nowadays A yr ago Nowadays A yr ago Nowadays A yr ago Nowadays A yr ago
Inside cars Outside thetemples
At communityevents
At the flea markets
yes
no
not sure
How much have we made a difference to the people we educate?
13.218.9
3.713.2
33.2
55.8
67.957.9
80.5
69.5
54.7
25.3
14.2 16.8 13.2 12.1 8.914.2
0102030405060708090
(n=181) (n=178) (n=185) (n=180) (n=184) (n=181)
Nowadays A yr ago Nowadays A yr ago Nowadays A yr ago
Inside common areas Inside churches Inside bars/night club
yes
no
not sure
Ethnicity• The Cambodian group indicated greater knowledge gained than did the
Hmong group.• The Lao group indicated greater change in their beliefs compared to the
Hmong group.• When asked the question “Do you think smoking should be used at all
times, some of the times, or not at all during family gathering events and celebrations?” three of the four ethnic groups (Hmong, Lao, and Vietnamese) tended to think smoking should not be used at all during family gatherings, whereas the Cambodian group had more respondents who still believed smoking should be used.
• When asked, “In the past two years have you seen any changes in your community about smoking?” the Hmong group tended to see fewer changes in their community compared to Cambodian, Lao, Vietnamese, and other Asians groups.
Involvement
• “In the past two years have you seen any changes in your community about smoking?” – Both the partner and volunteer groups saw more
changes in the last two years than did the participant group.
– Furthermore, the partner group tended to report seeing more changes than all the other groups.
Age
• “In the past two years have you seen changes in your community about smoking?”– The middle-aged group (age 46 – 54) reported seeing the most
changes in their community about smoking in the last two years, while the young group (age 12 -25) reported seeing the least amount of change.
• “Do you think smoking should be used at all times, some of the times, or not at all during family gathering events and celebrations?” – The young group (age 12 – 25) overwhelmingly felt that smoking
should not be used at family gatherings. Whereas the middle-aged group (age 46 – 54) continued to hold some beliefs that smoking should be used at family gathering events and celebrations.
Do you have any suggestions to help STEEP do a better job of educating more people in the community about the dangers of tobacco use? (n=129)
• Continue and increase STEEP involvement. Comments such as “Keep sharing over time, the hope is some people will change” and “They won’t come to you, tell them about the harm” indicate the urgency respondents feel to the issue of continuing the educational process within their communities.
• Involve and empower ethnic communities. A very strong theme of continuing STEEP education in the SEA communities and the value of empowering culturally specific methods was urgent as seen in the following comments, “I want STEEP staff to continue working on the issue so one day people can stop using it”, “Keep educating the community, if STEEP stops the project – who will do it?”
• Strategies for outreach to target areas. Respondents have many, many ideas for how to outreach into target areas. Outreach includes geographic locations; not yet reached such as rural areas. “Reach wider”, go to events, and a very specific direction “Go to the funeral home in South St. Paul.” These are just a few comments that can guide future outreach in the next phase of STEEP projects.
Recommendations
• Target new places of community congregation (i.e., funeral homes, multi-unit housing).
• Target youths both as volunteer participants. • Target the middle-aged group (age 46 – 54) since
respondents continued to hold some beliefs that smoking should be used at family gathering events and celebrations and.
• Target supporting the existing pool of volunteers.