Process Evaluation PlanPrepared for the Heart and Stroke Foundation of

Saskatchewan

December 6, 2010

Prepared By University of Saskatchewan MPH Students:Sunisha Neupane, Stuart Lockhart, Heather Davis, Paul Duong, Alim

Gillani, Prachi Bandivadekar

Table of Contents

1. Introduction.............................................................................................................................1

2. Purpose..................................................................................................................................3

3. Proposed Evaluation Design..................................................................................................5

3.1 Evidence-based Indicators for C.A.R.E...................................................................................73.1.1 Community Action..........................................................................................................83.1.2 Advocacy....................................................................................................................... 93.1.3 Research...................................................................................................................... 103.1.4 Education..................................................................................................................... 11

4. Data Sources and Data Collection Methods........................................................................12

4.1 Data Sources........................................................................................................................ 134.2 Data Collection Methods.......................................................................................................144.3 C.A.R.E Process Evaluation Strategies................................................................................15

4.3.1 Community Action........................................................................................................164.3.2 Advocacy..................................................................................................................... 174.3.3 Research..................................................................................................................... 184.3.4 Education..................................................................................................................... 19

5. Data Analysis.......................................................................................................................20

5.1 Analyzing Qualitative Data....................................................................................................215.2 Analyzing Quantitative Data..................................................................................................22

6. Evaluation Resources..........................................................................................................24

6.1 Budget.................................................................................................................................. 256.1 Human Resources................................................................................................................25

7. Conclusion............................................................................................................................26

7.1 Summary.............................................................................................................................. 277.2 Recommendations................................................................................................................28

Appendix 1 Evaluation Plan Checklist..............................................................................................30

Appendix 2 Evaluation Plan Checklist..............................................................................................31

2

1. Introduction

Established in 1956, the Saskatchewan Heart Foundation (a provincial division National

Heart Foundation of Canada) founded by a visionary group of Canadians who had a dream to

turn the tide on heart disease. Later renamed to the Heart and Stroke Foundation (HSF) with an

effort to more accurately reflect the scope of the foundations vision and hopes for the future.

Growing both nationally and provincially the HSF continues to endeavor to put heart health on

the public agenda, to elevate researchers, alleviate the burden of heart disease, and translate

the knowledge emerging from research to education for Canadians about their hearts.  Since

the establishment of the HSF more than $1 billion has been raised and invested into

innovationative heart and stroke research. Moreover, in an effort to address need where it is

most required more than 80 per cent of donations directly support research and education

programs in the province where the funds are raised. With the increasing burden of heart and

stroke-related diseases increases to impact the lives of so many Saskatchewan residents the

Heart & Stroke Foundation of Saskatchewan (HSFS) continue to receive impressive financial

support striving to reach new milestones in research support, health promotion, and advocacy.

These inspiring results are only made possible by the ongoing and valuable efforts of volunteers

and donors, and the success of HSF programs across this province and the country from coast

to coast (Heart & Stroke Foundation Canada, 2010).

The HSFS with the support and guidance of visionary doctors and researchers, staff

and, the corporate community and the community at-large, plays a relevant leadership role in

the study, prevention and reduction of disability and death from heart disease and stroke in

Saskatchewan. Heart disease continues to be an important public health problem in

Saskatchewan that requires a focus on solutions at the population-level. The HSFS is dedicated

to reducing the risk factors that are associated with heart disease and stroke in addition to

improving outcomes and instruction for survivors (Heart & Stroke Foundation Saskatchewan,

2010).

Heart disease and stroke are two of the three leading causes of death and the leading

cause of disability in Canada. Canadian baby boomers are now moving into their middle years

which are expected to result in a large increase in death and disability due to heart disease and

3

stroke. (Heart & Stroke Foundation Saskatchewan, 2010). This underlines the importance of

primary, secondary and tertiary prevention strategies. With Saskatchewan ranking high in its

smoking and obesity rates, it is as pertinent as ever to push residents to engage in healthy

eating, physical activity and tobacco cessation (Buller, 2010). Research has shown that

negative lifestyle choices such as sedentary lifestyle, smoking, porr nutritional choices, etc. play

a synergistic role in the development of heart and stroke related-diseases, and via education we

can change these negatives into positives and endeavor for a more heart smart Saskatchewan.

Collaboration between governments, researchers, health care practitioners, communities, and

individuals are necessary to extend the reaches of heart healthy programming and education

across the province (Buller, 2010) (Heart & Stroke Foundation Saskatchewan, 2010).

The HSFS is currently developing a strategy in health promotion that will benefit

communities throughout the province. Their aim is to eliminate heart and stroke related disease

and reduce their impact through the advancement of research and apply researched based

knowledge to the advocacy and promotion of healthy heart smart living. In order to determine

the efficacy of the programs within this model and their adherence to the initial guidelines set

forward, it is necessary to evaluate the Health Promotion process. The goals set forth in these

programs are:

To keep people healthy so that they do not suffer from heart or stroke related-disease in

the first place.

To prevent progressive, more debilitating disease in those affected by heart disease,

stroke and ischemic attack.

To reduce the incidence of death and disability from cardiovascular disease and stroke.

Focusing primarily on prevention, this document will lay the foundation for the evaluation

steps involved in the Heart and Stroke Foundation's health promotion programs. The

Community Action, Advocacy, Research and Education (C.A.R.E.) approach is a strategy that is

being used to achieve the previously stated goals. Although, these elements are interrelated, a

separate focus on each allows us to better describe the role that each can play in health

promotion programs. With defined target communities outlined by HSFS’s Community Action

Plan (Kaar, 2010) C.A.R.E. aims to give the communities ownership of the prevention programs

and processes. By engaging the community and providing a sense of ownership, HSFS can

4

identify strong leaders and companions situated within the community, leveraging the most it

can out of local resources. This will allow all parties to efficiently and effectively work within the

organization capacity and build for the future while continuing the ongoing process of

community development. C.A.R.E. will provide each community a support system, skills and

knowledge aligning with HSFS mission and vision. This will only lead to strong networks and

partnerships for sustainable programs for the short and long term.  Working with individual

communities HSFS can develop programs that will best involve the local health issues and

prioritize common health issues found throughout Saskatchewan.  With success in mind the

C.A.R.E. programs we outline clearly articulated goals mechanisms for decision making and

evaluations based on predefined outcomes and indicators to be achieved specific to each

section of C.A.R.E. (Kaar, 2010) (Issel, 2009).

5

Figure 1.1 Heart & Stroke Foundation Saskatchewan Health Promotion Logic Model. (Adapted

from: Mission in Action. Heart & Stroke Foundation Saskatchewan (Buller, 2010).

6

2. Purpose

Process evaluations are an important tool when it comes to the implementation of all

program types. They help us to understand why or why not a program was successful and they

can determine the effectiveness of a program prior to the investment of valuable resources

(Saunders, et al, 2005). This makes process evaluations beneficial from a financial standpoint

as well as an operative stance. Program implementation can be quite variable depending on

many factors so it is constructive to measure program performance in many regions and

communities throughout the province. To implement this strategy and standardize the C.A.R.E.

approach to health promotion, we have generated a process evaluation plan for the activities

and methodologies that will be carried out to improve the heart health of the people of

Saskatchewan.

The framework that we used for this evaluation follows the format of the Centers for

Disease Control and Prevention (CDC), Division for Heart Disease and Stroke Prevention

(CDC, 2006). Within this structure we took specific steps to investigate the quality,

completeness, exposure, satisfaction, participation rates, recruitment and context of the

programs that are going to be used in the C.A.R.E. approach. Looking to the literature for

guidance, we found evidence for key indicators that can be used to look at varying aspects of

community action, advocacy, research and education. For each indicator that we found, we

generated questions that are necessary to measure the achievement of the activities performed.

The answers to these questions can be found from many different sources which is why we

identified appropriate locations to find this data. While some of the necessary data is already in

existence through administrative records, logs, etc, others must be collected directly from

communities as primary data. The collection techniques and methodologies are also defined in

this report. Finally, the analysis of the data must be done in order to understand the results of

the data and to draw conclusions that can impact recommendations for revision.

7

3. Proposed Evaluation Plan

3.1 Evidence-based Indicators for C.A.R.E

3.1.1 Community Action

In developing criteria to evaluate the measure of success in implementing aspects of the

Community Action component of HSFS’s program strategies, it was determined that due to

measurement limitations most criteria would be based in self-reflection and would not be

quantifiable. In addition, because the Community Action Coordinator (CAC) plays such a direct

role in this part of the overall strategy, many of the criteria are directed at the performance of

this person. The performance of these people will stem directly from their ability to lead

effectively. The term best describing the two persons - who will work directly with communities

in enabling them, inspiring them, and mobilizing them - is leader. From this viewpoint, their

performance as leaders should be monitored.

Measuring Success of Group Interaction Sessions

The indicators that have been chosen for this portion of the C.A.R.E. strategy are

directed at measuring the success of community sessions from the viewpoint of community

members in attendance. The reason their input is valued more than the that of HSFS employees

is because community action is about grassroots involvement and must ultimately shape to the

mould of the community. In this sense, community action would be better termed community-

based action.

Measuring Coordinators’ Success as a Leader

The role of the Community Action Coordinator is to facilitate the community action

process. This involves communicating with community leaders to begin the community

engagement and action process; facilitating sessions of problem identification, solution

formulation, and monitoring and evaluation; and providing resources and support to

communities initiating projects. The HSFS’s presence at community action sessions will also

have an education component. Teaching community members about other community action

projects in other parts of the province will help to motivate the community about the potential

8

that it has to create a heart healthy community. A renowned theorist about critical education,

Paulo Freire of Brazil, articulated the ideas of empowerment education that has contributed so

much to discussion about engagement within communities.  “Education should have as one of

its main tasks to invite people to believe in themselves. It should invite people to believe they

have the knowledge (Bernstein & Wallerstein, 1988).” Empowerment education should be an

essential component of the CAC in delivery of community action sessions, though evaluation of

the CAC should be based on a more complete framework.

The criteria recommended to evaluate the leadership performance of the Community

Action Coordinator will be based upon the Kouzes and Posner Leadership Model. The criteria

used are derived from four of the Five Practices of Exemplary Leadership: Model the Way,

Inspire a Shared Vision, Enable Others to Act, Encourage the Heart. This leadership model is

built upon years of research by Jim Kouzes and Barry Posner about what it is that is essential to

get extraordinary things done in organizations. The research was based on qualitative accounts

of thousands of “personal best” stories.

3.1.2 Advocacy

All the programs are benefited from proper alliance, and it is necessary to evaluate it in

the context of the health program. The HSF in its program has recommendations for

development and implementations of tobacco control programs, food banks, government

agencies to implement various programs and policies etc. It also needs various collaborations

for delivery of services to various smaller communities at the local level. Within C.A.R.E. the

HSFS will use advocacy to form collaborations or partnerships for mutual benefit and the

common objective reduction of heart and stroke related-disease. Partnerships will be forged

with a person or an organization. Moreover, HSFS may form mutual strategic alliances at the

government, partner organizations, community or local levels to benefit wider range of

residents. With HSFS mission the involvement of partners from the public and private sectors is

needed to enhance sustainable health capacity of a community, and remove barriers that

enable the programs of C.A.R.E to work effectively (Heart & Stroke Foundation Canada, 2010).

The goals of health promotion programs are to provide lasting change in the community

that will lead to reduction of heart and stroke related-disease. Schwartz et al. (1993) report that

advocacy is facilitated at two levels. The first level involves formation of partnerships with

9

community organizations and citizens committing to advance program efforts. The second

involves environmental and policy interventions that are likely to remain and to result in positive

permanent change (Elder et al. 1993) (Schwartz et al. 1993).

Private Partnerships

Creating partnerships with members and organizations that are established in the

community will attract new stakeholders which are vital to sustaining programs. Thus, it further

engages the community and respected members of the community, not only legitimizing the

programs efforts, but furthering the ownership of the success of the program and the

community’s role in them. HSF has a base of long lasting partnerships already, however many

of these partnerships are in the larger centres, C.A.R.E. will be providing programs throughout

the province making the establishment of new local and provincial partnerships essential. In

smaller communities support by local organizations will only further aid in program success.

However, evaluating the support and satisfaction of partnerships is essential for maintaining

existing partnerships and forming long lasting new partnerships (Elder et al. 1993) (Schwartz et

al. 1993).

Public Partnerships

Formation of partnerships with any level of government could lead to successful policy

interventions for the reduction of heart and stroke related-disease. A successful partnership

would have a foundation of evidence such as data provided by evaluations. Moreover,

successful government partnerships would use HSF input within the context of social-planning

model, so that the development and implementation of policies are relevant to health

behaviours. Whether they are school or worksite policies or community and provincial

legislature, policies are often implemented to suppress undesirable or unhealthy behaviour.

Heart smart health promotion involves getting people to eat, drink, smoke less or not at all and

be more physically active. To some extent, individuals will always retain the “right to choose” to

be unhealthy. However, in many cases it is not a choice, but an intrinsic cycle perpetuated by

lack of education and social supports, only furthering the critical role of public partnerships

(Elder et al. 1993) (Schwartz et al. 1993).

10

3.1.3 Research

Bridging the gap between research findings and practical applications is essential to

population health intervention strategies. Many health promotion programs have failed because

evidence-based knowledge was not optimally transferred to practice. Zapka et al. cited three

types of potential program failure: (a) theory failure (the wrong strategy, messages, or vehicle

chosen for the intervention); (b) program implementation failure (the right strategy, messages,

and medium were planned but not implemented); and (c) measurement failure (the right theory

for the program, which was implemented as planned, however, the measurement methods were

not sensitive enough or not valid or reliable) (Zapka, Goins, Pbert, & Ockene, 2004). In order to

avoid these pitfalls, the Heart and Stroke foundation recognizes the importance of knowledge

transfer and exchange. As such, they have dedicated their efforts to support “the development

and application of policy- and practice- relevant research in a way that draws on the contribution

of both researchers and knowledge users” (Heart & Stroke Foundation of Canada, 2009).

Evaluating knowledge transfer and exchange activities is necessary to determine which

aspects work and areas that can be improved upon through modification and refinement. After

all, effective knowledge transfer and exchange efforts treat knowledge as a means to improve

practice and situations by having positive impacts, rather than as an end in itself (Sperling, Von

Sychowski, & Zarinpoush, 2007). Keeping this in mind, we developed a framework to facilitate

the evaluation of knowledge transfer and exchange activities. These activities include:

distribution and accessibility, dissemination, presentations, training sessions, workshops,

conferences, seminars, and meetings. By evaluating these activities, we can measure the

change in: user’s capacity to apply knowledge, integration of evidence into decision-making

process, collaboration between knowledge producers and users, cultural shift within the

organization or a community of practice.

Capacity to use knowledgeConducting effective knowledge transfer empowers participants to apply the gained

knowledge to real-life situations and problems in their own contexts. Thus, knowledge transfer

and exchange activities that focuses on the application of knowledge – rather than merely the

provision of knowledge – can lead to an increase in the capacity of users to interpret and apply

what they have learned (Sperling et al., 2007). The Saskatchewan Heart Health Program

11

(SHHP) is a good example of building individual capacity. A capacity-rich individual was defined

as: One who clearly articulates their health promotion values and principles, enhances skills of

self and others, are computer literate, partners within and across Health Districts and sectors,

records and shares experiences, steps out of “comfort zones,” engages in reflective practice,

and learns continuously were considered (Ebbesen et al., 2001). The program conducted an

annual survey (Health Promotion Contact (HPC) Profile) in the fall of 1998 to get an

understanding of individual capacity in each of the Health Districts. They found that prominent

resources contributing to individual capacity were personal commitment, enthusiasm and

energy, along with management support for and understanding of health promotion. Key

barriers to individual health promotion capacity included: lack of time; conflicting responsibilities;

lack of shared understanding of health promotion; reluctance to attempt more challenging

approaches for fear of previous work being de-valued; and the long time frame implicit in health

promotion (Ebbesen et al., 2001).

Integrating evidence into decision-makingThe knowledge that is produced and exchange should help practitioners make well

informed decisions about the various policies, programs, and projects in which they are involved

(Sperling et al., 2007). Hence, it is important to know if knowledge transfer and exchange efforts

have influenced decision-making process. Once more, the SHHP can be used as a reference

for evidence-based decision making. Participants of this program spoke most often of using

research findings to encourage reflective practice through the design and process of their data

collection (Haalboom, Robinson, Elliott, Cameron, & Eyles, 2006). Facilitated sessions were

held for researchers and practitioners to make sense of research findings and develop action

plans tailored to the local and provincial context (Riley et al., 2009).

CollaborationActivities that support collaboration among researchers and practitioners can promote

the creation of knowledge that is current, relevant, and readily applied by users (Sperling et al.,

2007). Therefore, measuring the level of collaboration created through knowledge transfer and

exchange is important. Perhaps the best example of successful collaboration would be the

Canadian Heart Health Initiative (CHHI). The CHHI was groundbreaking in its attempt to bring

together researchers and public health leaders to jointly plan, conduct, and act on relevant

12

evidence (Riley et al., 2009). The data from this project helped the coalitions to recognize each

other’s weaknesses and strengths, highlighted opportunities to build rural community capacity

and contributed to their evolution to broader wellness coalitions (Haalboom et al., 2006). For

example, in Prince Edward Island (PEI), research had an influence on heart health

implementation where feeding back research results contributed to better quality and more

comprehensive programming based on data from evaluations (Haalboom et al., 2006).

Cultural shiftCultural differences between researchers and practitioners can significantly affect

knowledge exchange. Thus, it is desirable to foster a culture that encourages ongoing dialogue

between knowledge producers and users which can produce an environment whereby both

groups will better understand each other’s perspectives, experiences, languages, and needs

(Sperling et al., 2007). Again, evidence from the CHHI provides a good understanding of how

the relationship between knowledge producers and users can lead to positive results. In the

CHHI, leadership from well-positioned, influential people (scientists and public health

professionals) within the public health system was especially critical to mobilizing a productive

and deeply engaged pan-Canadian community (Riley et al., 2009). The conference of Principal

Investigators brought together provincial and federal CHHI leaders to facilitate the exchange of

plans, ideas, and tools (Riley et al., 2009). This allowed researchers and practitioners to draw

on each other’s experiences and expertise during proposal development, implementation,

analysis, and write-up phases.

3.1.4 Education

An important component to any health promotion strategy is education of the community

and its individuals about initiatives and programs that are available in the community. To convey

the messages of heart healthy living and risk factor prevention, workshops, seminars, training

events and programs can be used to bring people together, provide helpful information and

create a positive learning environment. Sharing primary prevention tools with community

members allows them the opportunity to make decisions to take control of their health and make

improvements if needed.  Media messages are another part of education that play an important

role in communicating with the public. They increase awareness and provide healthy living

messages at a population level.

13

Marketing Tactics

In order to understand whether the educational tools, programs and marketing

techniques are being achieved successfully for a specific community, we can use a set of

indicators to determine their performance in the context of this health promotion strategy. We

can evaluate these indicators based on their adherence to implementation as intended, dosage

or amount of education provided, satisfaction and context.

In this evaluation, we must determine whether media messages put forth by the Heart

and Stroke Foundation are simple and easy to understand and how often they are carried out.

From the North Karelia project in Finland, it has been shown that media messages must be

simple and run as frequently as possible for them to be retained and comprehended (McAlister,

1982). The appropriateness of the venue for communication must also be assessed on a per

community basis. Saskatchewan is very expansive geographically and as such, extra thought

must be taken into the approach for conveying heart healthy living messages. If the right

communication techniques are being utilized, then community awareness of programs offered

by the Heart and Stroke Foundation for primary, secondary and tertiary prevention should most

likely be at a high level. This can be measured to evaluate marketing success (Issel, 2009).

Program Dosage

An important indicator to consider for educational programs, workshops, seminars and

events is the amount of sessions that are run and the number of participants who attend (Issel,

2009). This data can help guide the evaluation team in understanding the usage of existing

programs and determine whether there is a need to expand operations or to scale back due to

lack of participation. This can implicate future funding opportunities if attendance is outside of its

predetermined threshold range. If advertising of sessions is inadequate, then that aspect may

have to be addressed as well. Quality and completeness of programs should also be looked at

as an indicator in each community. The completeness of the intended program or educational

marketing tool can determine whether it will be successful or not.

Since every city and town differs in size, geography, population composition and health

needs, we must determine which programs are required in each community. Understanding the

needs of a specific community is essential to find out whether the services that are being offered

14

are in fact the right ones. Also, we would like to measure whether the program diversity

provided is adequate.

Program Satisfaction

Satisfaction with educational materials, advertising and programs will be evaluated from

a staff, volunteer and participant perspective. These indicators are extremely valuable to know

due to the significant impact that they have on the program as a whole. If those involved are not

satisfied with the program, they are not likely to become engaged in the educational messages

that they are being given. This is also important from a funding perspective because funding

agencies are interested in knowing the satisfaction level of the participants in the programs they

are funding (Issel, 2009).

These indicators as a whole provide insight into the success or lack thereof, the activities

being carried out to accomplish the goals initially set out. By measuring them, we can

understand which areas need more focus and how we can attempt to make improvements in

the education component of the health promotion strategy.

4. Data Sources and Data Collection Methods

4.1 Data Sources

The data, which is collected and analyzed by someone else other than HSF staff, is

secondary data for HSF. Secondary data should always be obtained from a reliable source so

that it is already validated and ready to use. It is essential to be careful to select only relevant

and representative data, which fits in the research that is being done. Here is a list of sources

that might be helpful for HSFS to obtain the secondary data depending on what type of

information is required: 

Saskatchewan health insurance data files provided by Statistics Canada.

The Canadian Community Health survey, which provides summary every two years

(Neudorf, 2008).

15

Data from local hospitals regarding heart disease

File and document review of HSFS program documents, proposal, research materials

related to the program.

Public health surveillance data

The health plan employer data and information set (HEDIS). Along with other health

indicators, HEDIS measures controlling high blood pressure (Neudorf, 2008).

Meeting minutes (participants, issues discussed, follow-through)

4.2 Data Collection Methods

Primary data is collected to explain and validate the need of interest (Issel, 2009). Data

collection is required to answer all the questions raised in the indicators table. Few of the data

collection methods that are suggested in the table are discussed in further depth.

Key Informant Interview

Key informant interview provide significant information through in-depth questions and

probing. Detailed and rich data can be gathered in an easy and inexpensive way. This method

is useful when the information from specific people or specific group in the community is

required. For example, HSF is suggested to use this method to get the information on the group

who are not accessing the programs being offered. The key informants for HSF can also be the

people who are at risk for heart disease. HSF can use this method to build and strengthen the

relationship with important key informants and stakeholders in the community. A few

experienced, analytical, knowledgeable and informative individuals should also be interviewed

(McKillip, 1987). The researcher should try to recognize the understanding and motivation of the

community through the interview. However, while analyzing the data, special attention needs to

be paid on not evaluating the program solely based on key informant interview because

sometimes it might not represent the whole community. If the resources allow H&S to hire a

researcher to conduct the interviews it can be well structured, consistent (as one person does

all key informants) and easy to analyze the information received. As the question arises how

many interviews to conduct, it should be enough to make sure that different community groups

and people with different experiences are adequately represented (McKillip, 1987).

Community Survey

16

Surveys can be easy to administer and cheaper for Saskatoon acknowledging that the

communities are quite spread out. It is convenient if the participants wish to remain anonymous

as well. Community surveys work perfectly in getting diverse and specific information.

Step 1 (Developing questionnaire): The questions for community survey needs to

descriptive and objective. Too many questions at a single time might discourage community

members to participate in the survey. Before developing the survey questions, it is necessary

that HSF is clear on who the target population are and are the findings going to be useful?

There are four types of question structures that can be followed. The types and examples are

given below from Dillman, 1978.

i.       Open ended (example: What should be done in order to improve access to Heart  and

Stroke Foundation’s programs?

ii.     Close-ended with ordered response (example: How often do you go to program/workshop

offered through HSF?

iii.    Close ended with unordered response (example: Which programs do you attend most

frequently? List answers for participants to circle)

iv.    Partially close-ended (example: Programs offered at which site of HSF is easiest to

access?)

Step 2 (Sample selection): Once the questions are ready, population needs to be sampled. Key

informant already will have some sort of criteria to select the participants so surveys can be

done by random sampling. 

Step 3 (conducting survey): Three ways that are suggested for HSF to conduct their community

surveys are:

i.       Online survey: Online survey can reach large number of population in easy and

inexpensive manner. SurveyMonkey (http://www.surveymonkey.com/) is a free and reliable

website for online survey.

17

ii.     Mailing survey: Mailing out survey works good as well. The issue can arise when not many

participants mail back their survey. Hence, to mitigate that problem, HSF can partner with

grocery stores in the city and ask help in collecting back the survey. If the grocery stores agree

on holding a drop off box for the surveys, there would be a bigger participant rate.

iii.    Booth in malls: If few booths are run in a busy mall, it is possible that large number of

people can be drawn to fill out the survey questionnaire. Few small incentives (candy, pen,

water bottle) can also be distributed to attract more people.

Attendance Log

Attendance log will serve the purpose of keeping track of how many people are

attending the offered workshops and programs. It can also help in identifying few key informants

based on attendance who might be able to answer questions such as

Why does someone attend one workshop in four months?

What motivates someone to attend all the workshops?

Here is an example of the attendance log that can be used:

Attendence log for Heart and Stoke information session       Facillitator:   Time:  Site:   Date:         

NameDo you always come to this site?

Do you like attending programs offered by HSF?

Can you contact you if we have any questions regarding participant satisfaction?

       Kim Crowe No Yes Yes       

18

Session Evaluation Forms

After every session, an evaluation should be done to understand participant’s level of

satisfaction. Separate evaluation form for the program staff and the facilitator is suggested. This

way it is participatory and meaningful to both participants and facilitators. The only motive

behind evaluation form is to enrich the participant’s and staff’s experience. There should be

session (or program) specific evaluation form, which will help in improving the specific program.

More general evaluation form might not serve the same purpose. Example of an evaluation form

is given in the appendix.

Concluding Discussion

Concluding discussion also plays a similar role as evaluation form, but is more in depth.

Discussion can sometime take more time than expected; this method can be used when there is

no time restriction. This is a great method to collect feedback. Concluding discussion is also

beneficial in summarizing the session and making a list of lessons learnt. 

4.3 C.A.R.E Process Evaluation Strategies

4.3.1 Community Action

4.3.2 Advocacy

In order to understand if advocacy is being successful and meeting goals, the use of a

set of indicators will be used to determine the effects of advocacy on the HSFS health

promotion program. Evaluation based on these indicators will allow HSF to know if Advocacy

was implemented as it was intended, if its meeting its objective, if there be any modifications

and the outcome (Heart and Stroke Foundation of Saskatchewan, 2010)(Issel, 2009).

In this evaluation, HSFS’s C.A.R.E. program needs to determine the quality and

quantities of collaborations that the HSFS has made for its program, the benefit that the HSFS

and the alliance agency is achieving and if it is overall benefit to the program. It is very important

to determine if the alliance is proper, and if it is meeting its objective. We can determine this by

knowing the kind of collaborations the HSFS has, with what kind of agencies, big or small,

19

government or private etc. the appropriateness of the alliance agency is also important for

determining the proper outcome of the program (Elder et al. 1993) (Schwartz et al. 1993).

Some of the Questions that can be asked to determine these indicators are

Do HSFS existing partnerships/alliances/collaborations support the strategic plan is the

alliance sustainable?

Is there adequate efficacy, efficiency and effectiveness?

Are the community board members adequately involved?

Benefit from the alliance to the HSFS and the partner agency?

HSFS also needs to ensure that the mission and goals of C.A.R.E are met, while also

considering the impact of partnership on the potential participant outcome and how collaboration

could benefit participants/program, and at what cost (Karr, 2010).

Some of the indicators to determine the success of a collaboration suggested by the

Public Health Agency of Canada are: determination of the proposed project, benefits of each of

the organizations of entering into the collaborations, timeline and if specific objectives are met

(Public Health Agency of Canada, 2010).

HSFS also needs to evaluate if their collaborations have a positive effect on the

community through the C.A.R.E. program. Factors such as the effectiveness of the program on

the community’s public health, HSFS will need to take into account the effects of advocacy.

This means if advocacy is meeting its objective, reaches the community and is valued by the

community. HSFS also needs to determine participation of community at the local level in

implementing strategies if the strategies are broad enough to reach each aspect of the plan, if

the advocacy is centered on developing community needs and enhancing the participation from

the community in a specific program or to all programs within the scope of C.A.R.E. (Klein &

Adelman, 2005) (Borden & Perkins, 1999).

4.3.3 Research

Given the wide range of knowledge transfer and exchange activities and large

geographical area of Saskatchewan, it is important to use multiple data collection methods. In

20

order to collect process data for measuring indicators from these activities, we suggest the

following methods: surveys/questionnaire, interview, organizational records, activity logs, and

focus groups.

Pre- and post-test questionnaires/surveys and Interviews will be useful for assessing

participants’ capacity to use knowledge and their ability to integrate evidence into decision-

making. Questions that need to be considered include:

1. Are the participants able to apply the gained knowledge to real-life situations and

problems in their own local contexts?

2. Do practitioners use the evidence-based findings to make well informed decisions about

the various programs and projects in which they were involved?

3. Does the activity provide enough opportunities for knowledge exchange?

4. Are there any follow-up actions that participants should do?

5. Do participants have resources that are accessible?

6. How satisfied are the participants?

The questionnaire/survey can be conducted before and immediately after the activity has

taken place to gauge the participant’s attitude and receive feedback. A follow-up in-depth

interview (face-to-face or telephone) can be conducted after 1 month. This gives the participants

time to practice what they have learned. The questions should be open-ended prompts about

the participants’ overall goals as well as specific goals related to heart disease and stroke health

promotion program if any, suggestions for improvement, enthusiasm level and attitude since the

activity, and anecdotal examples of integrating evidence into practice.

When it comes to monitoring the cultural shift and collaboration between researchers

and practitioners, organizational records, activity logs, and focus groups can be used. Some

guiding questions that need to be addressed are:

1. Is there ongoing dialogue between knowledge producers and users?

2. Are researchers open to practitioners’ inputs to ensure that the results of research are

relevant to the current context of heart and stroke health promotion?

21

3. Do researchers and practitioners embrace the shift from their traditional roles to one that

is collaborative, dynamic, and reciprocal?

4. Is there sufficient investment of time in developing relationships?

5. Is there actual implementation of study recommendations?

6. Are there any barriers to knowledge transfer and exchange?

Organizational records and activity are valuable data sources to establish baseline

information. They will be able to provide information such as meeting minutes and notes,

attendance, resource allocation, participation etc. When baseline data has been gathered,

conducting key informant interviews and then a focus group will help determine if the activities

were successful in meeting their objectives. Success should be judged by whether the existence

of knowledge brokers increased exchange between partners, increased decision maker uptake

of new evidence in health promotion, and helped the province accomplish their goals (CIHR,

2006). In addition, if it was found that the multidisciplinary group that assembled for the activities

continued to communicate and meet, it would be a good indicator of success.

4.3.4 Education

The indicators for education tools, as described previously in the report, are extremely

valuable in conducting this process evaluation. To measure them however, we designed a plan

to attain the proper data to evaluate them. 

We developed a set of questions pertaining to each indicator to get a better idea of what

information is needed. For example, when looking at the marketing of educational messages,

we found that it would be important to use indicators such as simplicity of media messages,

frequency of media messages and methods of communication. In each of these examples we

identified key questions that needed to be answered:

Are media campaigns simple, attractive and well-placed?

Are advertising methods recognized by members of the community?

Are advertisements run with sufficient frequency?

Are the methods of communication/advertising appropriate for the community given its

resources, size, population, etc.?

22

Are the methods of communication reaching the target population?

Is it a setting where the target population can be effectively reached?

The answers to each of these questions lie in different places which is why we must

consult multiple sources to find the data that we need. In each of these questions there is likely

no existing data meaning that it must be collected directly. For the qualitative data such as the

comprehension, attractiveness and location of media, a survey should be conducted amongst

the target population in the community. To make the survey resources efficient, questions

addressing a variety of indicators should be asked at the same time. Therefore the community

survey should address not only marketing questions, but others as well such as:

Are community members aware of the programs being offered?

Is the target population properly identified and are messages customized for them?

Is there an appropriate number of events /seminars/ workshops being run to address

different demographics in community?

These surveys can be conducted in a written, telephone, or online manner depending on

the resources that are available in the specific community. If staff resources are available to

distribute the written and telephone surveys in the community, then those can be used. If not

however, a central location with additional staff can administer the online survey. Participants

can be selected randomly or written surveys can be distributed at local gathering sites such as

grocery stores, Some basic guidelines to write survey questions state that easier questions

should be placed first, words that will provoke bias or an emotional response should be avoided,

and that similar questions should be placed together in a logical order (Community Tool Box,

2010). Examples of questions that can be asked in one of these surveys are provided in

Appendix II.

With respect to the educational programs operating in the community, it is valuable to

know the dosage or amount that is being completed. Questions we are interested in for this

component of the evaluation include:

Are there an appropriate number of program sessions that a community needs?

o How many?

o How many participants are attending each session?

23

o Is this number in compliance with the acceptable rate?

If attendance is low, how come?

o How many different types of programs in the community are there to address

multiple heart healthy strategies?

These questions can be addressed using data sources that should be available from

program staff. Tracking program logs will determine the number of participants and the number

of sessions that are being run. A threshold number for each is set in advance to determine

whether the program standards are being met. These are helpful pieces of information to gather

for future planning in terms of expansion or contraction of the program (Issel, 2009). We also

need to find out through administrative records and program logs, how many different types of

programs are in the community. This data is useful to know which risk factors are being targeted

and to see if there is a need for other programs to be implemented.

Satisfaction by participants, staff and volunteers is essential to understanding if the goals

and objectives of the programs, marketing strategies, events, seminars or workshops are being

met. The questions to be focused on for these indicators are:

Do program sessions meet their specific goals or objectives?

Is the program what participants were expecting?

How satisfied are participants with the program?

Did participants accomplish the goals of the session and did they increase their

knowledge of heart healthy strategies?

Do participants plan on using this knowledge and applying it to their lives?

Would participants recommend this program to others?

Are there enough training sessions?

Do training sessions develop the ability to teach/run the program in a way that meets the

program objectives?

Collection of this data will need to be done directly by staff or administration for the

program itself. This is done by conducting surveys of staff, participants and volunteers either

randomly or on a voluntary basis. Again, surveys can be written, telephone or online depending

on resources and staff available. Measurement of program satisfaction is extremely important

24

and is often necessary for future funding opportunities. This being the case, care must be taken

to develop specific questions that are going to be asked. Staff, volunteers and participants will

be answering similar questions but surveys should be tailored specifically for each role. If

desired, the Likert rating scale can be used. This involves ranking satisfaction level on a scale

between 1 and 5 with a score of "5" representing "very satisfied" and "1" being " not satisfied"

(Issel, 2009). Qualitatively focused questions such as "Would you recommend this program to

others?" and others listed above, are very beneficial to establish the success of the program,

event, seminar or workshop. Satisfaction surveys should be completed after every session to

ensure that each one is meeting the goals and expectations that were originally set out.

A full summary table of the methods described above, is provided in the Appendix.

5. Data Analysis

To come to a conclusion, it is essential that the data collected are managed, processed

and analyzed. The whole process evaluation and data collection for it is done to achieve

valuable feedback on how to make the programs better and accessible to larger number of

population in the community. Data analysis can be done in many different ways following

various techniques. There are numerous software to make the data analysis easier. In this

section qualitative and quantitative data analysis methods will be discussed for the collected

data. Issel suggests pay special attention to avoid fallacies such as Ecological fallacy during the

data analysis process. Ecological fallacy is “the assumption that a group characteristic applies

to all individuals within that group” (Issel, 2009).

5.1 Analyzing Qualitative Data

The data collected as comments and feedback through evaluation forms, concluding

discussions, open ended survey questions and key informant interviews need to be analyzed in

a qualitative manner. Qualitative approach gives opinion both literally and metaphorically to

stake holders and makes them feel valued (Issel, 2009). There are four elements that need to

be present in qualitative data, which are credibility (internal validity), transferability (external

validity), dependability (reliability) and conformability (findings truly from respondents) (Issel,

25

2009). Researcher should be aware which data collection method requires qualitative analysis

before starting the data collection as it is important to get answers that have all four elements.

Qualitative analysis should be done by a HSF staff who are familiar with all the programs and

workshops being offered. This is essential to ensure the data collected is completely

understood. Quantitative data can be analyzed by an outsider where as this method cannot be.

Three steps to go about for qualitative analysis that Issel suggests are

Overview of analytic process: The step includes

Developing a format and constructing a consistent structure for all collected qualitative

data.

Understanding and interpreting the meaning of what was said by the participants

Coding data (developing codes for certain type of answer)

Generating categories to compare data

Defining the categories (makes easier to separate which data belongs to which

category)

Linking the data

Software: There are a few common computer software used in qualitative analysis such

as NUD*IST, Ethnograph and ATLAS-ti. There are many software programs that can organize

and manage the data; however, staffs still need to be responsible to analyze it.

Issues to consider: Data can get messy and confusing. It is also a tedious task so the

evaluator needs to be patient. It requires trained evaluator(s) and data collector(s) so that the

process runs smoothly. Trained personnel can also ensure unbiased, consistent and reliable

data results (Issel, 2009).

The figure from Dey (1993) demonstrates a diagrammatic format for the steps involved in

qualitative data analysis.

26

Figure: The steps involved in qualitative data analysis.

5.2 Analyzing Quantitative DataNumerical data enables to do quantitative data analysis. It is crucial to do quantitative

analysis so that HSF can compare the results from different programs in a measurable way.

Data collected from all three survey methods needs to be stored in a computer database to

make the analysis faster and more efficient. Microsoft word excel spreadsheet is an example of

computer program that enables the data analyst to organise and analyse the data efficiently.

Alternative software for analyzing data are Statistical Package for the Social Sciences (SPSS)

or Power analysis software. SPSS is widely used to compare the data and to do statistical

analysis of the data. If the resources allow, HSF should consider hiring a graduate student to do

the data analysis using SPSS.

The software that HSF wishes to use for data analysis should be decided before the

data entry begins (Issel, 2009). Quantitative analysis result in findings such as the participant’s

rate, which HSF site is most visited, how many programs are useful and attended, how many

more programs to offer etc. The data needs to be organized, correct and up to date. Errors in

data entry process can lead to false conclusion and more difficulties. The researcher should

check for outliers, which are the values that are outside of the normal range (Issel, 2009).

27

Outliers should be discarded because they prevent from giving a representative result. Deciding

on removing or keeping the outliers can also be done on a case by case basis.

It is helpful to have the research plan in front of you while doing the analysis; it acts as a

guide and reminds of the goal. Here is an example of a research process to show where the

quantitative data falls (Bryman & Duncan, 1990).

Figure: Showing where data analysis falls in the research process.

28

6. Evaluation Resources

6.1 BudgetThough each individual program within C.A.R.E. will have their own specific budget,

variance between communities of similar sizes should be compared on an ongoing basis. This

will determine the extent how well each communities C.A.R.E. programs are performing with the

projected program expenditures.  Also, issues such as locations for programs and variations in

schedules may also impact budget as well as effectiveness of programs budget variance may

be important in distinguishing differences between communities. Sufficient variance is an alert

that the programs are not being delivered as planned. Outside salary of staff members contract

to carry out the interviews and focus groups for evaluation, there are no budgetary constraints

(Issel, 2009).

6.2 ResourcesWhen implementing the C.A.R.E. evaluation HSFS must realize that the staff, volunteers

and there quality and training are its most valuable asset.  HSFS staff and volunteers have a

history of commitment to the reducing the effect of heart and stroke related-disease.  The

magnitude of the C.A.R.E evaluation will call for an expanded volunteer and staff base to

adequately provide viable results.  Due to the fact that the programs are province wide, with

each target community varying in size HSFS community action coordinators will have to be

vigilant in order to ensure C.A.R.E. is being implemented efficiently, effectively and with a level

of efficacy.  To evaluate C.A.R.E a number of focus groups and key informant interviews will be

undertaken, these will require committed staff members that are trained with competency,

knowledge, and skills accompanied with the correct attitude to carry out and mediate

discussions to acquire appropriate data for the evaluations success.  These, staff members and

the data received are essential in verify the programs sustainability.  C.A.R.E.’s wide reach

throughout the province would call for varying numbers of staff, small communities under

10,000residents (such as Lloydminster, Kindersley, Estevan, etc.), will only require one staff

member to conduct interviews and mediate discussions. Whereas, larger communities up to

35,000 residents (such as Prince Albert, Moose Jaw, the Battlefords, etc.) will require three staff

29

members.  Metropolitan centres such as Saskatoon and Regina will require five trained staff

members (Issel, 2009). 

The majority of data on C.A.R.E will be received from surveys and questionnaires which

can largely be collected by volunteers.  The range of volunteers will also vary depending on

community size.  However, many of these surveys and questionnaires can be delivered during

the programs themselves, or via existing events and fundraisers put on by HSFS.  By using

existing events such as the Saskatoon dragon boat festival, Hoops for hearts and fundraisers

like SaskEnergy Big Bike a C.A.R.E questionnaire can be attached to the participant

registration.  Though this will not be a representative sample; however, the data will still be

relevant.  Annual events such as the door to door fundraising campaign in February will allow a

vehicle to distribute a short survey and will be more representative of the provincial population. 

Moreover, another vehicle for data collection would be jump rope for heart where numerous

schools across the province participate, with the participating children a survey can be sent

home for the parents.  Using existing programs and events will allow for data to be collected

using the existing pool of volunteers.  Not only will this reduce the number of volunteers needed

but using existing programs will allow HSFS to engage all relevant stakeholders in the

evaluation process (Issel, 2009).

All the data collected (primary and secondary) will need to be analyzed by a consultant

as the qualitative data will need to be transcribed and the quantitative data subjected to

statistical analysis.  The HSFS Memorandum of Understanding with University of Saskatchewan

may allow for the analysis to be done by volunteer students, in an effort to further the

partnership between the University of Saskatchewan and HSFS (Issel, 2009).

7. Conclusion

7.1 SummaryThe proposed process evaluation for the HSFS has been developed by specifically

applying concepts of evaluation to the C.A.R.E. health promotion program. The process

evaluation will allow HSFS to see how the C.A.R.E. programs are performing, a basis for

30

explanation of differences between communities, and areas that can be improved. To gain the

most accurate results a baseline of data should be collected before or at the beginning of the

programs and be periodically compared to data collected throughout the life of the programs.

The analysis of the data will show if the programs are performing as expected, as well as the

satisfaction of all stakeholders involved, and highlight disparities between communities. The

collection of this data and its analysis is vital to the success of the evaluation. C.A.R.E. has the

potential to change many lives and reduce the public burden of heart and stroke related-disease

on Saskatchewan, when it is implemented and its programs are used to their full potential.

7.2 RecommendationsDuring program development organization resources have been identified that are

integral in implementing C.A.R.E. but to insure the success and sustainability of the programs

HSFS must use the data to hold the programs accountable.  In order to maintain accountability,

baselines or preliminary data needs to be collected to evaluate the initial opinions and statistics

before program implementation, thus to truly see the effect of C.A.R.E.  In an effort to see

positive changes and evaluate if the programs are doing their intended purpose periodic

evaluations must be done.  The data from these periodic evaluations will allow HSFS to see if

the programs need to be changed or if certain programs are excelling, and why.  From the

obtained information reports must be formally made to HSFS to be used in evaluations and

program implementation.  The regular reports will also allow a format for information to be

passed to stakeholders, volunteers, and members of the community; which is key for continued

support and the sustainable success of C.A.R.E. (Issel, 2009).

31

Appendix A: Summary Tables

Appendix B

Appendix C

References

Bernstein, E., & Wallerstein, N. (1988). Empowerment education: Freire's ideas adapted to health education. Health Educ Behav; 15: 379.

32

Borden, L.M. & Perkins. D.F (1999). Assessing Your Collaboration: A Self Evaluation Tool. Journal of Extension. 37(2).

Bryman. A. & Duncan, C. (1990). Quantitative Data Analysis: For Social Scientist. New York: Routledge.

Dillman, D. A. (1978). Mail and telephone surveys: The total design method. New York: John Wiley and Sons.

Ebbesen, L. S., Woodard, G. B., McLean, S., Butler-Jones, D., Green, K., Reeder, B. A., Steer, S., & Feather, J. (2001). The saskatchewan dissemination story. Promotion & Education, Suppl 1, 35-39.

Elder, J.P., Schmid, T.L., Dower, P. & Hedlund, S. (1993). Community Heart Health Programs: Components, Rationale, and Strategies for Effective Interventions. Journal of Public Health Policy, 14(4): 463-479.

Haalboom, B. J., Robinson, K. L., Elliott, S. J., Cameron, R., & Eyles, J. D. (2006). Research as intervention in heart promotion. Canadian Journal of Public Health, 97(4), 291-295.

Heart and Stroke Foundation of Canada. (2009). Knowledge transfer and exchange in research: A guide for applicants and reviewers Canada:

Issel, L. M. (2008). Health program planning and evaluation: a practical and systematic approach for community health. 2nd ed. Toronto: Jones and Bartlet.

Klein, G.L. & Adelman, L. (2005). A Collaboration Evaluation Framework. Submitted to: International Conference on Intelligence Analysis. Retrieved from: http://abbyxmix.com/abby/ATK%20Collab%20Framework.pdf

Kouzes, J., & Posner, B. (2005). The leadership challenge. San Francisco, CA: John Wiley & Sons Ltd.

McKillip, J. (1987). Need analysis: Tools for the human services and education. Newbury Park, CA: Sage.

Neudorf, C. (2008). Sources of health data and health status monitoring (Powerpoint slides). Saskatoon Health Region.

Riley, B. L., Stachenko, S., Wilson, E., Harvey, D., Cameron, R., Farquharson, J., Donovan, C., & Taylor, G. (2009). Can the Canadian heart health initiative inform the population health intervention research initiative for Canada? Canadian Journal of Public Health, 100(1)

Sperling, J., Von Sychowski, S., & Zarinpoush, F. (2007). Effective knowledge transfer & exchange for nonprofit organizations: A framework Canada: Imagine Canada.

Schwartz, R., Smith, C., Speers, M.A., Dusenbury, L. J., Bright, F., Hedlund, S., Wheeler, F., Schmid, T.L. (1993). Capacity Building and Resource Needs of State Health Agencies to

33

Implement Community-Based Cardiovascular Disease Programs. Journal of Public Health Policy. 14(4):480-494.

Zapka, J., Goins, K. V., Pbert, L., & Ockene, J. K. (2004). Translating efficacy research to effectiveness studies in practice: Lessons from research to promote smoking cessation in community health centers. Health Promotion Practice, 5(3), 245-255.

34