1

CES Conference, June 4, 2006

Workshop facilitated by Harry Cummings, PhD., University of Guelph and Harry Cummings and

Associates, Guelph, ON

Results Based Approaches to Program Logic Models

Table of Contents 1. Workshop Objectives ............................................................................... 1 2. What is a Program Logic Model? ............................................................. 2 3. How do you prepare a Program Logic Model?......................................... 2 4. Case Study 1: Resource Jump Teams .................................................... 6 5. Case Study 2: Second Chance Alternative School Program.................... 15 6. Program logic Model Worksheet .............................................................. 19 7. Bonne Bay Primary Health Care Program Logic Model ........................... 20 8. What is an Evaluation Matrix?.................................................................. 27 9. Evaluation Matrix Worksheet ................................................................... 34 10. Bibliography and References ................................................................. 22 List of Tables Table 1: Program Logic Model with Definitions and Hypothetical Example.. 4 Table 2: Narrative Summary of Program Logic Model – HCA example ....... 5 Table 3: Evaluation Matrix with Definitions and Hypothetical Example ........ 29 Table 4: Evaluation Matrix – HCA example.................................................. 30

1

1. Workshop Objectives

• To give participants a greater understanding of Program Logic Models and how to develop them

• To link program logic models to customer satisfaction surveys in the rural

health environment

• What are you hoping to take away from this workshop?

2

2. What is a Program Logic Model (PLM)? A PLM (refer to Table 1) is a tool to help design and evaluate programs. It is a “picture” of the logical cause and effect relationships among four program components: inputs, activities, outputs, and outcomes. • Inputs are all resources which contribute to program activities • Activities are events or deliverables supported with program inputs • Outputs are the observable products of each activity • Outcomes are the short, medium, or long-term changes taking place as a

result of the program. Long-term outcomes are often referred to as impacts. The logical relationships can be understood as follows. The inputs must be made available if the activities are to be done. Activities must be completed for the outputs to be produced. Outputs must be produced and used if the outcomes are to be realized. The description of these program components makes up the first row of the PLM and is called the narrative summary. The next row is a row of indicators of performance or success, which is often the focus of monitoring and evaluation efforts. A row that lists assumptions and risks involved in running a program is the final row of the PLM. The PLM in Table 1 contains explanations of what is to be included in each column for each row and gives an example for clarity. The example used in Table 1 is for an imaginary poverty reduction program. Although Table 1 does not include a numbering system, it is often helpful to include one in order to show, for example, which activity is related to which output. Table 2 is an actual narrative summary portion of a PLM that was done as part of the Framework for Evaluating the Quality Assurance Programs of the Colleges of Health Professions in Ontario written by HCA. 3. How do you prepare a PLM? The Narrative Summary A typical place to start when making a PLM is with the narrative summary row (Row A in Table 1). The narrative summary in each column contains only words and does not attempt to indicate quantity or quality. Additional pages can be attached to a PLM when more space for elaboration is required. Performance Indicators The indicators row (Row B in Table 1) is more complicated to complete. Indicators seek to measure results and to provide evidence that progress is being

3

made toward the achievement of the goal. They are often the focus of monitoring and evaluation efforts. Indicators should consider quantity, quality, and time. Quantity refers to the number (e.g. the number of households being helped). Quality refers to characteristics of the beneficiaries (e.g. low-income households). Time refers to the time period in which the level of the indicator is to be achieved (e.g. household will benefit in year 1). A popular way to remember the characteristics of good indicators is the word SMART. S = Specific – is the indicator specific and clear? M = Measurable – will you be able to collect the data? A = Actionable – can you take action based on the findings of the indicator? R = Relevant – is the information provided by the indicator necessary for decision making? T = Timely – will the indicator tell you what you need to know at the right time? Indicators must be chosen carefully. There can be a limitless number of indicators for any program. There are six criteria that can be used to select which indicators to use. 1. Does the indicator measure the result as directly as possible? 2. Is the indicator a consistent measure over time? 3. Is the indicator sensitive enough to reflect changes in the attributes being

measured? 4. Is the information easy to collect and analyze 5. Will the information be helpful in decision making? 6. Are there sufficient funds to collect and analyze the data? Assumptions and Risks The last row of a PLM is for Assumptions and Risks. Assumptions refer to the external conditions that must exist for the cause and effect relationships expressed in the PLM to behave as expected. Risk refers to the probability that the assumptions will not hold true. Risk is rated as low, medium, or high. Assumptions and risks may apply to more than one column.

4

Table 1: Program Logic Model with Definitions and Hypothetical Example Outcomes Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact

(beyond life of the program)

A Narrative Summary

A listing of the human, financial, and physical resources used or required to implement the program E.g. Participating households

Events supported with program inputs Major deliverables A description of how the inputs are used in an organized manner E.g. Job searching seminars

Completion of an activity The directly observable products of each activity No attempt to show changes E.g. Participation of households in job seminars

Short term changes taking place as a result of the program Shows the link to program objectives E.g. Improved access to employment through increased skills

Medium term changes taking place as a result of the program Shows the link to program objectives E.g. Reduction in level of unemployment for participating households

Long term societal changes taking place as a result of the program Shows the link to program goal or vision Eg. Poverty reduction in City X

B Performance Indicators

The units used to measure the quantity, quality, and timeliness of each input E.g. # of low income households registered by month

The units used to measure what the inputs have been used for E.g. # of job searching seminars held by month

The units used to measure the quantity, quality, and timeliness of each output E.g. # of households participating in job seminars by month

The units used to measure short term changes E.g. # and type of jobs for participants by month by socio-economic status

The units used to measure medium term changes E.g. % of graduates still working by type of job 2,3,and 4 years post program

The units used to measure long term impacts E.g. # of jobs created as % of total jobs in City X

C Assumptions and Risks

Assumptions: The necessary conditions external to the program that must exist if the cause and effect relationships between program components is to be as expected Risks: The probability that the assumptions will not hold true E.g. households have time to participate, medium risk E.g. job seminar leaders are available and committed, low risk E.g. no unexpected changes in employment opportunities, high risk

HCA’s recommended model: adapted from various sources

5

Table 2: Narrative Summary of Program Logic Model – HCA Example Inputs Activities Outputs Short-term

Outcomes Long-term Outcomes Impacts

Narrative Summary

• Public involvement in QAP development (though consultation and participation on the QAC)

• Human (staff, committee members, consultants) and financial resources for QAP development, promotion, implementation and maintenance

• Research, consultation and literature search undertaken to develop and mange data collection)

• Facilities and equipment to develop and manage the QAP (computer equipment, software, office space)

• Knowledge and capacity to carry out QAP monitoring and evaluation

• RHPA Requirements: • Regulations to direct

and enforce the QAPs • Functioning QAC

• Development of management and implementation plans for the QAP, including resource allocations and time lines

• Development of QAP components (peer assessment or review; continuing education and professional development; practice enhancement and remediation, etc.)

• Member education and promotion of the QAP

• Development of QAP materials for members

• Implementation of the QAP • Development of

procedures and administrative systems for QAP implementation

• Development of procedures and data collection systems to establish baseline data on the profession

• Development and distribution of practice standards

• Collaboration / consultation with other Colleges

• Member approval and support for the QAP

• Members participating in the QAP components

• Practice standards guidelines applied in the QAPs

• Self assessment profiles completed

• Peer reviews conducted

• Continuing education requirements fulfilled

• Data from members’ participation in QAP is collected and analyzed

• Data collected and analyzed to establish base line performance of the profession

• Increased awareness of QAP function and process by members

• Members’ have increased awareness of their learning needs and styles

• Members aware of and compliant with practice standards and guidelines

• Key issue areas in practice are identified across the professions and appropriate remediation mechanisms applied members’ incompetence is identified and addressed by Colleges

• Improved knowledge and skill of members

• Improved practice relative to practice standards

• Improved standards of practice across the professions

• Increased bottom-line performance of professions as a whole

• Colleges have increased knowledge of effective QA activities

• Improved public access to competent practitioners

• Improved quality of health care provision (safe, ethical, client-focused)

• Improved population health status

• Improved patient outcomes

6

4. Case Study 1: CONTEXT OF THE HURON HEALTHKICK PROJECT As described in the HealthKick Huron project proposal (February 2005), healthcare in rural Ontario is at a crossroads. Rural communities throughout the province are facing chronic challenges in meeting the healthcare needs of the resident population. Central to this issue is the persistent and critical problem of attracting and retaining health care providers. While it is generally accepted that the medical profession has specialized to the point where personnel such as transplant surgeons, radiation oncologists or pediatric cardiologists need to work in larger cities, most of rural Ontario is faced with a lack of healthcare personnel to deal with basic, common health problems – family physicians, nurses – even pharmacists. The acute and persistent shortages of health care providers in rural Ontario and the never-ending struggle to attract and keep them in rural communities has been well acknowledged for quite some time. Many small communities that had doctors 40 years ago have none today. Other communities that had two or three now have one, and are concerned about their chances of replacing those professionals should they retire or choose to leave. The shortage of health care providers in rural Ontario has received considerable attention from researchers, policy-makers and health care planners. Yet recruitment and retention tools have to date been narrowly focused on providing financial incentives such as signing bonuses, and more recently the establishment of community-owned and operated turn-key health clinics. While these recruitment initiatives are now viewed as necessary prerequisites, and may work in some communities, capital is difficult to raise in small communities. At best, bidding wars are costly and hard to win, even for larger population centres. Furthermore, despite these efforts, the problems of attraction and retention of health care professionals have not been resolved.1 1 In 2003, The Ontario Rural Council (TORC) commissioned a series of reports as part of a Comprehensive Rural Health Human Resource Strategy for Ontario. One of these reports – A Review and Synthesis of Strategies and Policy Recommendation on the Rural Health Workforce by Raymond Pong and Noreen Russell makes several useful recommendations for rural communities facing the challenge of ensuring the existence of adequate health care professionals. The points as summarized in the original SHARP proposal are presented in Appendix A.

‘Health care is important to rural residents, but it also

has an impact on the ability of rural communities to retain and attract investment. We want to make rural communities attractive places for relocation by skilled

workers and professionals, such as badly needed health care professionals. We’ll be looking to support

innovative projects that retain and enhance rural health services. We’ll support strategies for attracting and

retaining health care professionals.’ John Gerretsen

Minister of Municipal Affairs & Housing Rural Ontario Municipal Association Conference

February 2004

7

The challenges described above point to the need for exploring alternative and innovative efforts towards recruitment and retention of healthcare professionals. A well-designed and comprehensive healthcare attraction and recruitment strategy is needed if rural communities are to have access to adequate services in the future. The HealthKick Huron project is responding to this opportunity and using Huron County as a case study to assess the performance and impact of the initiative. The next section of the report introduces the evaluation approach that is being used for the HealthKick Huron project.

8

The healthcare worker shortage is not exclusive to doctors. Registered nurses account for approximately 35% of the entire healthcare workforce. In 2000, there were 11,855 Registered Nurses practicing in rural Ontario. This represents an 8% drop from 1991. 75% of healthcare employers in Bruce, Grey Huron & Perth state that they have a great deal of difficulty recruiting nurses. Furthermore, the Canadian Nursing Association is suggesting that recruiting and retaining rural nurses may become more challenging in the future given the reduced number of individuals entering and graduating from nursing programs. To compound the issue of a drop in rural healthcare professionals, the need for healthcare services in rural Ontario is expected to increase. Rural populations continue to grow at a rate of half a percentage point a year. But given demographic trends common to many rural areas (youth out-migration, baby boomer in-migration), the rural population will age at a significantly faster rate than the province as a whole. In 15 years, the median age of rural Ontarians will be 55. This in turn will have major implications for the demands placed on the healthcare system in those communities. For the purposes of the pilot, the area under examination is Huron County, located in southwestern Ontario north of the City of London and west of Toronto (Figure 3).

9

Figure 3: Huron County in Southwestern Ontario

Source: www.hurontourism.on.ca/PAGES/INFORMATION%20PAGES/findingus.html As outlined in the February 2005 HealthKick Huron (SHARP) project proposal, Huron County has many characteristics that make it a prime example of the state of rural healthcare in Ontario today:

• Huron County’s current population stands at 60,000 people widely distributed across a patchwork of small towns, villages, hamlets and rural municipalities.

• Population centres include Clinton (pop 3,000); Exeter (pop. 4,500); Goderich (pop. 7,500); Seaforth (pop 2,500); and Wingham (pop. 3,000). Smaller communities include Bayfield (pop. 700); Blyth (pop. 1,100); Brussels (pop. 1,000); Hensall (pop. 1,200) and Zurich (pop. 600).

• With a population density of just 17.7 residents per square kilometer, and approximately 55% of County residents living in non-urban areas, the population of Huron County is one of the most “rural” in Ontario.

• Huron County has a strong agricultural sector with over 18% of the local labour force employed in agriculture and resource based industries – this is 4 times greater than the provincial average.

10

• As with many rural areas Huron County has an older than average population. 15% of Huron County residents are over the age of 65, compared to a provincial average of just 13%. Furthermore, the seniors’ population in Huron is expected to grow by 3% over the next decade.

• Huron County’s population is served by 5 community hospitals located in each of the larger centers. The area has been designated under the Province’s Underserviced Area Program. When the project proposal was drafted Huron County had 15 vacancies for family physicians.2

• At the same time, a workforce crisis is looming in the nursing sector. According to the Grey Bruce Huron Perth District Health Council, 74% of Huron County’s Registered Practical Nurses are over the age of 50. With a large numbers of impending retirements in the next decade, the nursing sector is concerned about being able to replace the current workforce, especially given that there has been a noticeable decline in the number of new graduates.

• To make matters worse, the region’s healthcare sector was recently delivered a severe blow with the announced departure of 6 doctors in the Goderich area, resulting in 4,800 newly “orphaned” patients. Combined with an estimated 2,000 orphaned patients in other parts of the County, this leaves over 10% of the local population stranded without the services of a family physician.

2 The most recent List of Areas Designated as Underserviced for General/Family Practitioners shows a total of 11 vacancies in 6 different communities across Huron County: Clinton (1 vacancy), Exeter (1), Goderich (1), Huron East (2), Wingham/Listowel and Area (5), Zurich (1). Source: Ontario Ministry of Health and Long-Term Care, Underserviced Area Program, April/May/June 2006.

11

There are five major healthcare centres in Huron County:

• Clinton Public Hospital • South Huron Hospital Association (Exeter) • Alexandra Marine and General Hospital (Goderich) • Seaforth Community Hospital and Health Centre • Wingham and District Hospital

The nearest secondary and tertiary referral centres are located in London and Stratford. Additional details on each of the five healthcare centres is provided in Appendix B. 4.2 Goals and Objectives of the HealthKick Huron Project The goal of HealthKick Huron is to develop local capacity in Huron County for creating communities of choice for healthcare professionals in rural Ontario. As part of this process, HealthKick Huron is testing innovative strategies for rural healthcare recruitment. The initiative also proposes to develop a new partnership of health care agencies, community economic development interests and community volunteers. Specific objectives and related strategies of the HealthKick Huron project are as follows: 4.2.1 Youth Engagement Objective: To encourage local youth in the pursuit of health-related careers by exposing them to a full range of employment opportunities found in the healthcare sector.

ISSUE STRATEGIES PARTNERS TIMING

Youth Engagement

Rural Health Career Exploration

1. Rural youth have become disinterested in health related careers given a perception of poor working conditions and low pay – particularly in the rural sector.

i. Provide relevant career information and mentoring to at least 1000 high school students in order to educate them about the range of rewarding careers found within the rural healthcare sector. This will be accomplished by developing web-based materials for use by guidance counselors, libraries; and presentations to

Avon Maitland District School Board; Huron Perth District Catholic School Board; Foundation for Enriching Education – Huron Perth

2005 Develop website; attend career fairs 2006 Attend career fairs 2007 Attend career fairs

12

students.

ISSUE STRATEGIES PARTNERS TIMING

Youth Engagement

Rural Health Career Placements

2. Individuals that have been raised in a rural area and have positive work experiences in their communities are much more prone to be connected and locate there permanently.

ii. Provide opportunities for at least 30 rural youth to experience working conditions in the local health care sector through increased summer job placements & internships. Conduct MapQuest Camp with approximately 24 rural high school students from Huron and Perth Counties.

Local Healthcare Employers (Clinics, Pharmacies, CCAC’s, LTC); HRC Centres for Students Schulich School of Medicine at the University of Western Ontario. Huron County Emergency Management Committee

2005 Work with partners to establish ten appropriate summer placement positions; conduct follow-up survey of work placement students and employers at completion of placements 2006 Work with partners to establish ten appropriate summer placement positions; conduct follow-up survey of work placement students and employers at completion of placements; conduct survey of MedQuest Camp students at completion of program (July 2005) 2007 Work with partners to establish ten appropriate summer placement positions; conduct follow-up survey of work placement students and employers at completion of placements

13

Youth Engagement Activities to Date Career Information HealthKick Huron has collected a variety of healthcare career brochures, videos and posters from a variety of professional healthcare organizations. Interviews have been conducted with local healthcare service providers and currently there are 11 Huron County Healthcare Career Profiles on the HealthKick Huron website with an additional 14 profiles to be added in the next few months. Project staff attended the Huron Perth Career Expo in Stratford in October 2005 where over 2,000 students attended. Provided a healthcare career presentation to St. Mary’s grades 7 and 8 with a nurse co-presenter. Student Work Placements Student work placements completed in 2005 included 3 placements with pharmacies in Huron County for part time students and 5 placements with local Chamber of Commerce in four communities to develop healthcare recruitment materials. Proposals are currently being reviewed for the 2006 round of rural healthcare work placements. Up top 12 placement proposals will be approved to receive funding of up to $3,000 each to hire a student in a healthcare related summer job. MedQuest The Schulich School of Medicine and Dentistry’s Southwestern Ontario Medical Education Network (University of Western Ontario) has developed MedQUEST: a strategic collaborative program that encourages students from Southwestern Ontario, an area with a substantial rural population, to apply in greater numbers to medical school. MedQUEST is a one-week summer program for Grade 10 and 11 students where local physicians, nurses and medical/nursing students deliver a curriculum to regional high school students. Students are nominated by their schools for demonstrating academic skills, leadership qualities, and interest in the field of medicine.

MedQUEST is designed to provide students with realistic experiences in the field of medicine using state-of-the-art advanced simulator training, peer workshops, and the opportunity to watch physicians/nurses in a clinical setting. Over the course of the week students learn to deliver robotic babies, read x-rays, splint and cast fractures, create a professional resume and gain an understanding of the training requirements needed to become a physician. Early exposure to the practice of medicine can help influence students' career plans and ultimately

14

increase the chance of successfully entering medical school. Similarly, these types of exposures can influence students to choose a career in nursing.

A MedQuest Camp has been confirmed for Huron and Perth Counties for July 10-14, 2006. The Camp will be conducted in Seaforth. Approximately 24 grade 10 and 11 students from Huron and Perth Counties will participate. Meetings were conducted with guidance counselor staff at both school boards to outline student application procedures. Plans for a mock disaster exercise in 2006 are also being coordinated with the Huron County Emergency Management Committee. Schulich has developed a number of evaluation processes that they will conduct in relation to the activities offered at the MedQuest Camp including daily questionnaires that are completed by the program participants.

15

5. Case Study 2: MICAH (MICRONUTRIENT AND HEALTH PROGRAM FOR AFRICA) The recent World Development Report which focused on ‘attacking poverty’, broadened the definition of poverty to include the Health, Nutrition and Population (HNP) dimensions and emphasized reducing exclusion, lessening vulnerability and providing opportunity (World Bank, 2000). Adequate nutrition is a critical component of human health. It is particularly critical for children under the age of five and mothers. This study focuses on infant and child health, and nutrition, which are among the priority topics in the MDGs and PRSPs. The Hidden Hunger Conference in 1991 and the International Nutrition Conference in 1992 determined that the micronutrient supplementation and fortification of foods was recognized as the most cost effective strategy in combating malnutrition Micronutrient And Health Program for Africa (MICAH) is a program that intervenes to reduce micronutrient malnutrition and thus improve overall health for all family members. MICAH targets women and children, who are the most vulnerable to micronutrient deficiencies because of their increased nutrient needs. MICAH, implemented under the coordination of World Vision Canada and funded by CIDA, has produced positive results in terms of its goals and has reduced the burden of malnutrition. However, a question that remains unanswered is did MICAH’s interventions actually benefit the most disadvantaged groups? If so, how and if not, why not? How can MICAH programming be more effective in this regard? The MICAH results have not been analyzed with respect to poverty and this study provides an ideal opportunity to carry out an incidence analysis. This study will assess the reach of micronutrient and health programs to the most disadvantaged groups in Ethiopia and Malawi, two of the five MICAH countries. An important reason for this study is to improve on MICAH’s programming activities for the second phase of its program, which begins later this year. This study will determine how well MICAH health and nutrition programs have reached the poorest and most disadvantaged groups in Ethiopia and Malawi and the reasons for their success or failure. PROGRAM/POLICY DESCRIPTION The Micronutrient And Health Program for Africa (MICAH) was established in 1995 to reduce micronutrient malnutrition through integrated health and nutrition interventions. MICAH’s goal was to improve the nutrition and health status of women and children through the most cost effective and sustainable interventions. The objectives and activities of MICAH include: Increasing intake and bio-availability of micronutrients (Iodine, Iron and Vitamin A) through supplementation, fortification, promotion of exclusive breast feeding, dietary modification Reduce the prevalence of diseases that affect micronutrient status (diarrhea, parasitic and vaccine preventable) through water and sanitation education and infrastructure improvement, immunization, malaria control, and treatment of worms and parasites.

16

Build local capacity for delivery systems to improve micronutrient statistics by – equipping laboratories, developing the MICAH guide, training staff and influencing national policy or compliance to policy. MICAH was implemented in Malawi, Ethiopia, Ghana, Senegal and Tanzania and completed its first Phase in 2000. The projects have had a profound impact on the target communities in these countries. Some of these results, for the two countries selected, are discussed below. From the baseline survey of 6004 households, the micronutrient needs of 20 districts representative of Ethiopia were identified. Through MICAH’s interventions including the provision of Vitamin A capsules, community education and community gardens, VAD was significantly reduced. A follow-up survey of 6573 households showed Bitot’s spots decreased from 6.4% to 1.4% in pre-school children (MICAH, 2002). MICAH Ethiopia supported diversification of food sources through promotion of backyard gardens, provided training to health workers in Micronutrient and other health issues, promoted construction of latrines, de-worming campaigns and provided access to safe water. MICAH Ethiopia covered 20 project sites, worked with 6 partner agencies with a budget of CAD$10 million. MICAH Ethiopia provided a national representation with 1.5 million beneficiaries. Harry Cummings coordinated the mid-evaluation plan, which recommended that efforts be improved with respect to poverty targeting. The baseline and follow up survey contain a variety of indicators that could be used for incidence analysis. This analysis remains to be done. Malawi was MICAH’s second largest program with a budget of CAD$8.2 million and benefited 1.8 million people (almost 20% of the total population). Baseline and follow-up surveys, available to the researchers (approximately 2300 MICAH households surveyed at baseline and follow-up as well as 2300 non-MICAH households), suggested the MICAH interventions reduced stunting (low height-for-age) from 56% to 40%. The baseline and follow up survey contain a variety of indicators that will be used by this proposed research to analyze poverty incidence. MICAH Malawi reduced anemia caused by iron deficiency through supplementation, fortification and treatment of parasitic infections. Furthermore, new water sites were provided, pit latrines constructed and children were treated for hookworms and schistosomiasis. MICAH Malawi advocated for monitoring and enforcing of iodized salt legislation through provision of equipment and capacity building, and installed fortification facilities in 6 community flourmills. MICAH’s program in Malawi (and Ethiopia) combined with the data from surveys and field checks could be used to analyze the impact on the poor compared to the better off. 3. Literature Review/Hypothesis Statement Malnutrition hinders the achievement of other human development goals and is closely linked with poverty. The correlation between malnutrition and extreme poverty has been demonstrated utilizing a region specific analysis. Just as low income is a contributing factor to poor health and malnutrition; poor health, malnutrition and large family size are key reasons for the persistence of poverty. Malnutrition is the manifestation, cause and

17

consequence of poverty, and has intergenerational effects. In Tanzania and Pakistan, the poorest income quintiles had the highest rate of underweight and stunting. Factors associated with income poverty also contribute to malnutrition – e.g. female illiteracy, poor sanitation, lack of access to health care, and food insecurity. A study in Lagos, Nigeria, demonstrated a high prevalence of protein-energy malnutrition in children whose families had low incomes and lived in single room dwellings with limited access to water. Furthermore, low literacy rates of mothers was also another factor that attributed to high malnutrition levels. Anthropometric indicators such as stunting and wasting can be correlated with poverty. These measurements are standardized, objective, inexpensive to measure and easily comparable. The anthropometric indicators and poverty indicators have been collected for the MICAH project and will be correlated as part of the research proposed here. Nutritional status can further be used to assess development programming needs and can be used as a targeting mechanism as demonstrated in Ecuador for poverty mapping and in Honduras for a food coupon program. This study will help assess the reach of HNP programs to the poor, their impacts and the reasons for their successes or failure. Evidence from Mexico suggests that programs that are more carefully targeted toward the extreme poor are more successful, even though the targeting may cost more. Simulated transfer schemes using household data for Venezuela, Mexico and Jamaica indicated that targeting of interventions to narrowly defined geographic regions helped reduced poverty. This was compared with transfer schemes involving no targeting, such as general food subsidies. Resources and capacities are limited and hence it may be necessary to target. Poverty occurs at household and community levels hence it may be necessary to assess the reach of interventions at these levels. Household surveys and discussions with the community may assist in determining why interventions do not reach the extreme poor. Allocative efficiency and equity are important considerations for channeling resources to the poor. People with more assets such as access to land and livestock (both variables collected in MICAH surveys) may have greater access to nutrition programming activities. Health and nutrition programs are usually targeted toward the poor or most disadvantaged groups but for many reasons, the most disadvantaged do not take full advantage. This study will correlate poverty status (land ownership, household assets, livestock, housing and other socio economic characteristics, anthropometric indicators) with MICAH nutrition programming initiatives (community gardens, health and nutrition education, micro nutrient supplementation initiatives) and seek to determine the reasons for the correlations observed. There are several reasons why HNP interventions do not reach the poor. In a study conducted in rural Ethiopia, heavy workloads, lack of access to health services, traditional practices, poverty, social status and decision-making power were the factors that affected the health of women in Bujatira. In Malawi, the burden of AIDS is impinging upon household resources thus making the poor more vulnerable. One of the reasons for poor reach or inadequate impact of HNP program on the poor, is that many programs target children who go to school. The school attendance group is biased toward the “relatively better off” families.

18

The landless or those who are marginalized tend to get neglected from development interventions. Furthermore, within the beneficiary groups there is a wide range in the income status of those reaping the benefits. In India benefit-incidence analysis demonstrates that government health service expenditures benefit the rich considerably more than the poor. This study will seek to determine how well health and nutrition interventions are reaching the disadvantaged groups and if they are not the reasons for this. The research hypothesis to be tested is that MICAH programming participation and impact is correlated with household poverty status. The sub research hypothesis is that poor households are less likely to benefit from MICAH initiatives than better off households. The analysis is to be completed using the baseline and follow-up survey data from Phase 1 MICAH for MICAH and non-MICAH areas as well as focus groups with selected participants from field sites in Ethiopia and Malawi.

Key Analysis

Baseline and Final Survey Data of MICAH and Non-MICAH Areas

Correlate Data as per Hypotheses Focusing on Poverty Indicators and MICAH Participation and Impact Indicators

Data from National Statistics and other available data.

Phase 1

Focus Group Discussion with MICAH participants – Determine characteristics of poverty. Identify poor and better-off groups. Key Informant Interviews

FGD with Non-Micah group

FGD with MICAH participants - Better-off

FGD with MICAH participants – Poor

P H A S E 2

19

6. Program Logic Model Worksheet Use either case study to fill out this worksheet

Outcomes Inputs Activities Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact

(beyond life of the program)

A Narrative Summary

B Performance Indicators

C Assumptions and Risks

HCA’s recommended model: Adapted from various sources.

20

Page 1 of 7 7. Bonne Bay Primary Health Care Project: Project Logic Model (PLM) – September 2004

Bonne Bay Primary Health Care Project: PLM Narrative Summary (1) Outcomes

Inputs

Activities

Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact • Dept. of Health and Community

Services - Office of Primary Health Care (OPHC)

• Primary Health Care Advisory Council

• Evaluation Advisory Group • Working Groups (Wellness, Scope

of Practice, etc.) • Western Health Care Corporation • Health and Community Services

Western Region • Western Regional Wellness

Coalition • Local Medical Advisory

Committee • Building Better Tomorrows • Community profile information

(Community Health Needs and Resources Assessment, Profile of Youth in Western Region, Community Accounts, etc.)

• PHC Project Coordinator • Health Care service providers

(physicians, nurse practitioners, community health nurses, dieticians, physiotherapists, mental health counsellors, addictions counsellors, child youth and family services social workers, etc.)

• Primary Health Care Network • Other key stakeholders (Personal

Care Homes, School District 3, Tree House Family Resource Centre, RCMP, Private Ambulance Operator, etc.)

• Community Members/Partners • Clients/Patients • Health Care Facilities, equipment,

supplies, etc. • Traveling clinics • Financial Resources

• OPHC consultations with Project Coordinator and local PHC working group

• Develop PHC Team • Conduct monthly PHC Team meetings • Develop inventory of job descriptions

for PHC Team members • Develop practice protocols and referral

processes • Develop conflict resolution processes • Develop and conduct PHC orientation

sessions for new employees • Conduct PHC Team building activities • Assess interdisciplinary training needs

(e.g. ISSP training for PHC Team members working with children and youth under the age of 21; education and training for ambulance services)

• Consultations between PHC Team and PHC Network

• Develop Internet based inventory of various programs, services and referral processes of the PHC Team

• Develop working groups to address coordination of specific services and initiatives (e.g. youth services, diabetes management, cervical screening)

• Develop Terms of Reference for working groups

• Establish the Community Advisory Committee (CAC)

• Develop CAC Terms of Reference • Develop PHC orientation and team

building activities for CAC members • Conduct monthly CAC meetings • Establish communication structure

between CAC and PHC Team

• PHC Team and PHC Network enhanced • Participation of PHC Team members in

monthly team meetings • Participation of PHC Team members in

team building conferences • Inventory of job descriptions for PHC

Team members • Formal practice protocols and referral

processes • Formal conflict resolution processes • Participation in PHC orientation sessions

for new employees • Participation of PHC Team members in

interdisciplinary training sessions • Participation of PHC Team members and

PHC Network members in consultations • Internet based inventory of various

programs, services and referral processes of the PHC Team

• Working groups to address coordination of specific services and initiatives

• Terms of Reference for working groups established

• Reports/recommendations submitted by the working groups

• CAC established with Terms of Reference • PHC orientation sessions and team

building activities provided to the CAC • Participation of community members in

monthly CAC meetings • Input, feedback, and recommendations

from the CAC to PHC Team

• Increased understanding of PHC Team goals and objectives

• Increased understanding and knowledge of the role and ability of each PHC Team member

• Increased provider participation in PHC planning, implementation and evaluation processes

• Increased support provided to individual Team members

• Improved communication between PHC Team members

• Improved coordination of intervention services

• Enhanced scope of practice for PHC Team members

• Increased involvement of community and intersectoral partners in planning and delivery of programs.

• Increased community

participation in PHC initiatives/programs

• Increased participation by client/patient in decisions related to self, family, and community programs

• Enhanced satisfaction of PHC professionals

• Increased efficiency of

health care system • Increased participation by

client/patient in decisions related to self, family, and community programs

• Increased community

satisfaction with health care access and quality of health care

• Improved health status for the residents of the Bonne Bay region

• Increased self-reliance

among community members in regards to health care

21

Page 2 of 7

Bonne Bay Primary Health Care Project: PLM Narrative Summary continued (2) Outcomes

Inputs

Activities

Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact • Develop PHC promotional materials and

public awareness campaign for the community

• Develop Terms of Reference for Local Wellness Coordinator

• Develop workshop to establish local wellness coalition

• Identify wellness initiatives (e.g. physical, mental, youth, seniors health programs)

• Identify funding sources to support wellness initiatives

• Recruit volunteers to assist with delivering wellness initiatives

• Introduction of Broadband communication capacity

• Installation of video conferencing equipment

• Register the PHC Project population with the PHC Team

• Meditech and Client Referral Management System (CRMS) is made available to all relevant service providers

• Development of electronic client/patient record

• Articles written and published in local newspapers highlighting PHC programs and health issues

• Portable PHC information board developed and displayed at community events

• Public information sessions on the PHC Team conducted by PHC Team members/Project Coordinator

• All correspondence formally identifies project areas as part of PHC

• Local Wellness coordinator hired • Participation of PHC Team members and

community partners in a one-day workshop to develop a local wellness coalition

• Local Wellness Coalition established • Wellness initiatives identified, planned and

implemented • Funding secured to support wellness

initiatives • Volunteer base for Wellness initiatives

established • Broadband communication available in all

PHC sites • Video conferencing equipment available in

all PHC sites • Bonne Bay region population registered

with the PHC Team • All relevant service providers have access

to Meditech and Client Referral Management System (CRMS)

• Electronic client/patient record established (agreed record structure for file recording and sharing of information)

• Increased community awareness and knowledge of PHC services/programs provided in the Bonne Bay region

• Increased community

participation in PHC initiatives/programs

• Increased access to and

effective use of video- conferencing by PHC Team members

• Increased effectiveness of

technology in PHC Team communications

• Increased use of common

client/patient records

22

Page 3 of 7

Bonne Bay Primary Health Care Project: PLM Performance Indicators (1) Outcomes

Inputs

Activities

Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact • Number of paid OPHC person

days invested in Bonne Bay Project on an annual basis and associated wage and benefit costs

• Number of participants in the Primary Health Care Advisory Council and number of meetings on an annual basis

• Number of participants in the Evaluation Advisory Group and number of meetings on an annual basis

• Number of paid PHC Project Coordinator person days and wage and benefit costs

• Number of Health Care service providers by professional discipline participating in the Project (e.g. Primary Health Care Team, Primary Health Care Network)

• Number of other key stakeholders/interest groups participating in the Project

• Quarterly or semi-annual financial statements (comparison of planned budget vs. actual)

• Number of meetings between OPHC and Bonne Bay Project within the 1st and 2nd year of the Project

• Development of the PHC Team within the 1st year of the Project

• Number of PHC Team meetings conducted on an annual basis

• Development of an inventory of job descriptions for PHC Team members within the 1st year of the Project and updated semi-annually

• Development of practice protocols and referral processes within the 1st year of the Project

• Development of conflict resolution processes within the 1st year of the Project

• Number of PHC orientation sessions offered to new employees on an annual basis

• Number of PHC team building activities conducted on an annual basis

• Number and type of interdisciplinary training sessions offered to service providers on an annual basis (e.g. ISSP training, education and training for ambulance services)

• Number and type of consultations between PHC Team and PHC Network

• Development of partnerships with other key stakeholders/interest groups

• Development of Internet based inventory of PHC programs, services and referral protocols within the 1st year of the Project and updated on a quarterly basis

• Number of health care service providers by professional discipline participating on the PHC Team

• Number of PHC Team members participating in monthly meetings

• Number of PHC Team job descriptions listed in the Internet based inventory

• Practice protocols and referral processes formally defined by the end of year 1

• Conflict resolution processes formally defined by the end of year 1

• Number and percentage of new employees attending PHC orientation sessions on an annual basis

• Number and percentage of PHC Team members participating in team building activities on an annual basis

• Number of PHC Team members by professional discipline participating in interdisciplinary training sessions on an annual basis

• Number of health care service providers by professional discipline participating in the PHC Network

• Number of partnerships established with other stakeholders/interest groups and types of activities

• Number of PHC programs and services listed on the Internet site

• Number of PHC Team members by professional discipline accessing the Internet inventory on a quarterly basis

• Increased understanding of PHC Team goals and objectives (Score on TET questions A1,A4-A8,A10: 2004 baseline results compared to June and Dec. 2005 results)

• Increased understanding and

knowledge of the role and ability of each PHC Team member (Score on SPT questions A1-A4, B12: 2004 baseline results compared to June and Dec. 2005 results)

• Increased provider

participation in PHC planning, implementation and evaluation processes (Score on TET questions A2,A3,A9: 2004 baseline results compared to June and Dec. 2005 results)

• Increased support provided

to individual Team members (Score on TET questions C1-C11: 2004 baseline results compared to June and Dec. 2005 results)

• Improved communication

between PHC Team members (Score on TET questions B1-B13: 2004 baseline results compared to June and Dec. 2005 results)

• Enhanced satisfaction of PHC professionals (Score on TET questions A10,B13,C11,D8,E1-E3: 2004 baseline results compared to June and Dec. 2005 results ; Score on SPT questions C1-C4: 2004 baseline results compared to June and Dec. 2005 results)

• Increased efficiency of

health care system (Score on SPT questions A6,A7,B9,B12: 2004 baseline results compared to June and Dec. 2005 results

• Increased participation by

client/patient in decisions related to self, family, and community programs (Score on TET questions D2,D6: 2004 baseline results compared to June and Dec. 2005 results)

• Increased community

satisfaction with health care access and quality of health care (Score on CPST: 2004 baseline results compared to March and Sept. 2005 results)

• Improved health status for the residents of the Bonne Bay region (Selected health status indicators: Increase in the percentage of women having annual pap smears – baseline results compared to 2005 results)

• Increased self-reliance

among community members in regards to health care (Score on CPST: 2004 baseline results compared to March and Sept. 2005 results)

23

Page 4 of 7

Bonne Bay Primary Health Care Project: PLM Performance Indicators continued (2) Outcomes

Inputs

Activities

Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact • Number and type of working groups

established to address coordination of specific services and initiatives (e.g. youth services, diabetes management, cervical screening) on an annual basis

• Terms of Reference established for each working group

• Number of meetings conducted for each working group on an annual basis

• CAC established with Terms of

Reference within the 1st year of the PHC Project

• Number of CAC meetings conducted on an annual basis

• Number of CAC team building sessions conducted on an annual basis

• PHC promotional articles and

advertising developed within year 1 • Portable PHC information and display

board developed within year 1 • PHC public information sessions

developed within year 1 (note: the display board could be used in conjunction with these events)

• PHC signature logo developed for all communications within year 1

• Number of PHC Team members by professional discipline and number of community partners by sector/interest group participating in each working group

• Number of reports/recommendations prepared by working groups and submitted to governing agency and/or senior management sub-committee on an annual basis

• Number of participants in the CAC by

gender, sector/interest group and place of residence

• Number of CAC members participating in monthly CAC meetings

• Number of CAC members participating in CAC team building sessions

• Number of reports/recommendations prepared by the CAC and submitted to the PHC Team on an annual basis

• Number of articles or promotional features

appearing in local newspapers on an annual basis

• Number, type and location of events where the portable display is used on an annual basis

• Number, type and location of PHC public information sessions conducted on an annual basis

• PHC signature logo applied to all communications related to PHC project areas within the 1st year of the project

• Improved coordination of intervention services involving PHC Team members and network providers (Score on TET/SPT questions D3,D4/B12,B16: 2004 baseline results compared to June and Dec. 2005 results)

• Enhanced scope of practice for PHC Team members (Score on SPT questions A1-A7, B1-B15, C1-C4: 2004 baseline results compared to June and Dec. 2005 results; Increase in the number of ISSP meetings and ISSP profiles submitted 2004-2005)

• Increased involvement of community and intersectoral partners in planning and delivery of programs (Score on TET questions D1,D4,D5: 2004 baseline results compared to June and Dec. 2005 results)

• Increased community awareness and knowledge of PHC services/programs provided in the Bonne Bay region (Score on TET questions D7: 2004 baseline results compared to June and Dec. 2005 results; Increased percentage of the population registered with the Bonne Bay PHC Team 2004-2005)

• Improved coordination of intervention services involving PHC Team members and network providers (Number of working group recommendations submitted compared to the number acted on as reported in the APR: year 1-2 baseline compared to year 3, 4, 5)

• Increased involvement of

community and intersectoral partners in planning and delivery of programs (Number of CAC recommendations submitted compared to the number acted on as reported in the APR: year 1-2 baseline results compared to year 3, 4, 5)

24

Page 5 of 7 Bonne Bay Primary Health Care Project: PLM Performance Indicators continued (3)

Outcomes Inputs

Activities

Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact

• Development of Terms of Reference for Local Wellness Coordinator within year 1

• Development of one-day workshop to establish the Local Wellness Coalition within year 1

• Local Wellness Coalition established within year 1

• Number of Wellness Coalition meetings conducted on an annual basis

• Number of wellness strategies identified on an annual basis

• Number and type of funding sources identified to support wellness strategies within year 1 and 2

• Development of volunteer base within year 1 and 2

• Introduction of Broadband

communication capacity in PHC sites in Bonne Bay region by March 2006 (pending approval of a Broadband initiative)

• Installation of video conferencing equipment in PHC sites by March 2006

• Wellness coordinator hired within the first year of the PHC Project

• Number of PHC Team members by professional discipline and number of community partners by age, gender, sector/interest group and place of residence attending the one-day workshop to establish the Local Wellness Coalition

• Number of Wellness Coalition members participating in meetings

• Number of wellness strategies implemented on an annual basis in comparison to the number of strategies identified

• Number of citizens by age, gender, education, income and place of residence participating in Wellness initiatives

• Amount and type of funding (cash and in-kind) raised to support initiatives on an annual basis

• Number of volunteers recruited in year 1 by age and gender

• Number of new volunteers recruited in year 2 by age and gender and number of year 1 volunteers retained in year 2

• Number and location of PHC sites with or

without Broadband access by end of year 1, 2, 3 etc.

• Number and location of PHC sites with or without videoconferencing equipment by end of year 1 and 2

• Increased community participation in PHC initiatives/programs (e.g. health and wellness programs) as reported in the APR: year 1 baseline results compared to year 2 results; Increase in program leaders and volunteers as reported in the APR: year 1 baseline results compared to year 2 results)

• Increased use of video

conferencing equipment (Increase in hours of use and reduction of travel costs as reported in APR: year 1 baseline results compared to year 2 results)

• Increased effectiveness of technology in PHC Team communications (Score on TET question B6: 2004 baseline results compared to June and Dec. 2005 results)

25

Page 6 of 7 Bonne Bay Primary Health Care Project: PLM Performance Indicators continued (4)

Outcomes Inputs

Activities

Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact

• Development of the PHC client/patient roster within year 1 and 2 of the project

• All relevant service providers have access to Meditech and CRMS

• Development of electronic client/patient record

• Number and percentage of population registered with the Bonne Bay PHC Team by age, gender and place of residence by the end of year 1 and 2

• Number and percentage of relevant service providers using Meditech and CRMS by the end of year 1 and 2

• Number and percentage of PHC Team members using the common electronic patient record by the end of 2006 (this timeline is contingent on the development and activation of a common client/patient record system)

• Increased use of common client/patient records (Score on SPT question B8: 2005 baseline results compared to 2006 results – this timeline is contingent on the development and activation of a common patient record system)

26

Page 7 of 7

Bonne Bay Primary Health Care Project: PLM Assumptions and Risks (1) Outcomes

Inputs

Activities

Outputs Short term (1-2 yrs) Medium (2-5 yrs) Long term Impact Assumptions:

• The majority of health care service providers in the Bonne Bay region are committed to pursuing PHC approach to service delivery. • Improvements/adjustments will be made to current funding methods to provide fair remuneration to Fee for Service Physicians and other private service providers (i.e. for time and involvement in activities such as consultations,

case conferences, PHC Team meetings, interdisciplinary clinics, and investment involved in the transition to an electronic network). • All health service providers will work collaboratively as custodians of client/patent records to ensure the confidentiality of files. • Improvements/adjustments will be made to current funding methods for salaried employees to address PHC Team collaboration and communication time needs including flexible time arrangements and overtime where

applicable. • Salaried employees work responsibilities will be redefined to include scheduled interactions between various providers (activities such as consultations, case conferences, PHC Team meetings, interdisciplinary clinics, etc.) • Community members will take an interest in providing input throughout the process.

Risks:

• Loss of interest by local leaders could result in lost momentum. • Integration of boards could impact the delivery of coordinated services. • The past merger of institutional boards and of community boards with programs from Department of Human Resources and Employment are still evolving processes. • Participation by service providers may change if an adequate funding method is not developed - could result in loss of providers. • PHC staff turnover may disrupt and slow pace of adoption. • The system introduced may not be sustainable beyond the funding horizon. • Funding is not secured to introduce Broadband communications capacity. • Sustainability. • Geography – the Bonne Bay region has a small population for a large geographic area.

27

8. What is an Evaluation Matrix? Evaluation matrices are a tool for systematically identifying evaluation questions, indicators of success, and appropriate data sources and data collection methods. Whereas the PLM gives a picture of the whole program, the matrix indicates the aspects of the program that will be the focus of the evaluation. Completing an evaluation matrix will ensure that all necessary issues will be covered. Table 3 is an example evaluation matrix that provides explanations for each column and continues with the example of a poverty reduction program. Table 4 is an actual evaluation matrix from a Framework for Evaluating the Quality Assurance Programs of the Colleges of Health Professions in Ontario written by HCA. Evaluation Issues and Questions Some of the more common issues addressed in evaluations are rationale, efficiency, effectiveness, access/reach, and impact. Each issue forms one separate row in the matrix. Rationale refers to the extent to which a program contributes to the overall goal or strategic direction of the organization. The exploration of rationale considers the question “Is this the most appropriate way to achieve the goal?” Efficiency refers to the extent to which program inputs were supplied and managed and activities organized in the most appropriate manner at the least cost to produce the outputs. It is the link between outputs and outcomes in the PLM. Efficiency questions usually refer to the timeliness, quality and quantity of the delivery of inputs in relation to the program plans and needs. Effectiveness refers to the extent to which the program outputs produced outcomes (or results) and thereby contributed to the program goal. It records changes in the beneficiary group(s) that have happened as a result of the project. It is the link between outputs and outcomes in the PLM. Effectiveness questions examine whether the program outputs are being used in a manner consistent with the program goals. Access and reach are concerned with determining whether the project and its benefits are accessible and of benefit to all members of the population and whether the community was given the opportunity to participate in the development of the project. It applies to all columns of the PLM. Impact refers to the long term and sustainable changes experienced as a result of the project. Indicators Indicators seek to measure results and to provide evidence that progress is being made toward the achievement of the goal. Please refer to the section on indicators in the PLM section for more details. Data Required and Sources of Data The Data Required column contains the information that is required to determine the indicator. The Sources of Data column contains the individuals,

28

organizations, documents, or reports from which the data will be obtained. A data source should be specified for each indicator. Data sources should remain consistent as switching sources may jeopardize the reliability of the data. Method of Collection and Analysis The Method of Collection column lists the methods and techniques that will be used to gather the data that will measure results. Some examples include, interviews, testimonials, participant observation, or document analysis. When deciding on methods it is good to consider whether sampling techniques need to be random or purposive, how the data collection instruments will be designed, and what level of disaggregation of the data is required (e.g. gender, income level, age). The Analysis column explains how the data will be analyzed. It is important to consider whether it is necessary to compare disaggregated data, what baseline data already exists, how several sources can be used to increase validity, and whether it will be cost effective. Responsibility and Timing These columns are meant to ensure that those involved in the evaluation agree on whom will do what and the time frame in which they are expected to do it.

29

Table 3: Evaluation Matrix with Definitions and Hypothetical Example Issue Evaluation

questions Indicator Data required Source of data Method of

collection Analysis Responsibility Timing

List the key evaluation issues that need to be addressed E.g. effectiveness

List the key questions that need to be asked in order to determine how the program is doing with regard to the issue being researched E.g. Have the program interventions improved the ability of low-income households to have access to employment?

The units used to measure quantity, quality, and timeliness of inputs, activities, outputs, and outcomes. E.g. % change in employment for participating households

What data will be needed to determine the indicators E.g. employment statistics and household registration

Individuals or organizations or documents from which data is obtained E.g. organization’s household registry, households

How will the data be collected? E.g. survey of household participants

How will the data be analyzed? E.g. data will be disaggregated into two parent households and one parent households and compared to pre-program baseline data

Who will collect and analyze the data? E.g. John will collect and Jane will analyze

When will the collection and analysis of data be carried out or completed? E.g. Collection done by 1 September 03 – Analysis done by 1 December 03

HCA’s recommended model: adapted from various sources.

30

Table 4: Evaluation Matrix-HCA Example Evaluation Issue

Key Questions Indicators Data Sources and Collection Methods

Rationale 1) Does the QAP follow the principles of the RHPA and the Ministry Guidelines? 2) Is the selected mix of QAP components likely to achieve the intended goals? 3) Is the program acceptable to College members? 4) Is the program flexible enough to meet the different learning style of members? 5) Is the time investment for college members manageable? 6) Does participation in the QAP interfere with members’ practice responsibilities? 7) Does the QA approach reflect current principles of CQI? Is the QA approach still relevant?

1) QAPs contain measures to: a) identify deficient members, b) remediate members, c) increase competence of members, and d) address behaviour and remarks of a sexual nature 2a) Literature/consultation research supports Colleges’ selection of QA methods; 2b) independent experts support the methods selected 3a) % members complying with the QAP 3b # of member concerns and complaints lodged with the College 3c) member satisfaction with QAP 4) existence of options for members’ participation 5) amount of time members spend annually in QAP 6) members’ perceptions

1) content analysis of College QAPs

2a) review of QAP background documents and literature search; 2b) peer review panel of independent experts 3a) college tracking system connected with annual registration 3b) college tracking of member complaints 3c) members’ survey or focus groups 4) content analysis of College QAPs; members survey or focus groups 5) members’ survey; college tracking system connected with annual registration 6) members perceptions

Efficiency 1) Have the QAP regulations been passed? 2) Is the program being implemented as scheduled in the implementation plan? 3) Have adequate resource been allocated for timely and efficient implementation of the QAP? 4) Is there sufficient capacity in QAP management in the college? 5) Have adequate data collection systems been established? 6) Is the program affordable and sustainable in the long-run?

1) Regulations passed 2a) Implementation is on schedule # 2b) of QAP components implement3ed and degree of implementation 3a) implementation is on schedule 3b) actual expenditures correspond with planned expenditures 3c) materials/ products produced as planned 4a) staff secured with experience and skills in QAP management 4b) staff and management’s perceptions of Colleges’ management capacity 5) software and information tracking systems/data base have been established 6a) cost of QAP / member 6b) % of QA P budget to overall college budget 6c) increases in members fees

1) Regulations 2a) And 3a) comparison of progress to implementation plan 2b) And 3c) and management consultation / program records review 3b) Financial records/statements 4a) And 4b) management and staff consultation 5) Administrative records; management consultation 6a) , 6b) and 6c) administrative records

31

Evaluation Matrix -HCA Example Cont’d Evaluation Issue

Key Questions Indicators Data Sources and Collection Methods

Effectiveness 1) Have college members been adequately informed and educated about the QAP 2) Are members complying with and participation gin the program? What are the barriers to participation/compliance? 3) What members participated in the QAP and in which components 4) Have the clinical knowledge, skills, and judgement of practitioners increased? To what extent ware members applying this knowledge to their practice? 5) Is the QAP identifying deficient members? 6) Is remediation improving the skills of deficient members? 7) Are quality and/or deficiencies issues across the profession being identified? 8) Have monitoring systems been put in place? Has QAP data been collected and analyzed? 9) Are members being informed about the findings, results and learning of the QAP? 10) Have adequate tools been developed to facilitate member participation in the QAP 11) Have practice standards/guidelines been established? 12) Has members’ knowledge of standards and guidelines increased?

1a) # and % of members calling for further information 1b) % of member not properly participating 1c)variety and # of promotion methods used 2a) % members complying with QP 2b) barriers identified 3a) # of members participating in different QAP components 4a) test scores from courses 4b) % of practitioners indicating increased knowledge due to QAP 4c) % of members applying new knowledge to their practice 5) # of deficient members identified 6a) reassessment undertaken and 6b) # of members found deficient, and non-deficient following remediation 7) systems established to monitor aggregate issues and the nature of the profession as a whole 8a) systems established 8b) data collected and analyzed 9) information dissemination mechanisms have been put in place 10) members use of tools; members satisfaction with the tools 11) standards and guidelines developed 12) % members indicating increase in knowledge and application of standards

1a) 1b) and 1c) college tracking and records system 2) college tracking system 2b) member survey; focus group 3a) college tracking system 4a) test administration 4b) and 4c) member survey 5) assessment records 6) re-assessment records 7) administrative / program records 8a) and 8b) administrative / program records 9) administrative / program records 10) member survey 11) review of guidelines/standards 12) member survey

32

Evaluation Matrix-HCA Example Cont’d Evaluation Issue

Key Questions Indicators Data Sources and Collection Methods

Access and Equity

1) Does the program involve clients and the public? 2) Has consultation to develop and mange the QAP involved members from diverse geographic areas, practice settings and scopes of practice? Will improved patient care be available to all people in the province? 3) Do practitioners in different geographic areas, different practice settings, and different scopes of practice have equal access to all program components and to the benefits of the program? 4) Have provisions been put in place to protect members on leave from practice 5) Are members’ rights to fair and judicious assessment processes and assessors protected?

1a) Mechanisms are in place, and 1b) have been used for public feedback and involvement 2a) Diverse repres4entation on QAC; 2b) diverse participation in consultation processes 3a) Existence of provisions for different forms of QAP participation, 3b) members perception of their access to the QAP 4) Leave provisions are in place and utilized objective assessment criteria have been established and are utilized 5a) Members have the right to ask for alternate assessors 5b) Members are provided with results of their assessments in a timely manner 5d) Member’s rights are clearly outlined in the QAP regulations

1a) And 1b) review of program records 2a) QAC and management consultation 3a) Review of program records 3b) Member survey or focus group 4) Program records 5a) , 5b), 5c) and 5d) review of assessment records and QAP regulations

Impact 1) Has the QAP resulted in improved delivery of care? 2) Have there been any unplanned effects

1a) Level of client satisfaction 1b) Decrease in client complaints 1c) Decrease in complaints from other practitioner 2) Perceptions of members and other professionals

1a) Client focus group; questionnaires 1b) And 1c) college tracking records 2a) Members survey, focus groups 2b) College tracking system

33

9. Evaluation Matrix Worksheet Use either case study to fill out this worksheet Issue Evaluation

questions Indicator Data required Source of data Method of

collection Analysis Responsibility Timing

HCA’s recommended model: Adapted from various sources.

34

10. BIBLIOGRAPHY AND REFERENCES Canadian Evaluation Society. (1989). The Value in Evaluation. A Statement for

Managers. Canadian Evaluation Society. Canadian International Development Agency. (1997). The Logical Framework:

Making it Results Oriented – Guidelines. Performance Review Branch. Canadian International Development Agency. Available online at: http://www.acdi-cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/c36ebd571b6fa02985256c620066cd6f?OpenDocument

Canadian International Development Agency. (1999). Results-Based

Management in CIDA: An Introductory Guide to the Concepts and Principles. Performance Review Branch, Canadian International Development Agency. Available online at: http://www.acdi-cida.gc.ca/cida_ind.nsf/0/b83025bcf2da296785256c6b001a1b36?OpenDocument

Canadian International Development Agency. (2000). CIDA Evaluation Guide.

Performance Review Branch, Canadian International Development Agency. Available inline at: http://www.acdi-cida.gc.ca/cida_ind.nsf/0/061a4e025fc6fe2785256c6b00155e2d?OpenDocument

Canadian International Development Agency. (2000). RBM Handbook on

Developing Results Chains. Results-Based Management Division. Canadian International Development Agency. Available online at: http://www.acdi-cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/d321e7d06cb2a65885256c62001b1b6c?OpenDocument

Canadian International Development Agency. (2001). How to Perform

Evaluations – A Ten Part Series Performance Review Branch, Canadian International Development Agency. Available online at http://www.acdi-cida.gc.ca/cida_ind.nsf/0/d4207e5842f66aa385256c5000589701?OpenDocument

Canadian International Development Agency. (2001). A Results Approach to the

Implementation Plan. Performance and Knowledge Management Division, Canadian International Development Agency Available inline at: http://www.acdi-cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/05dde51fd3e6e77f85256c620019b585?OpenDocument

35

Canadian International Development Agency. (2000). Training workshop - Introduction to Results-Based Management. Performance and Knowledge Management Division, Canadian International Development Agency. Available online at: http://www.acdi-cida.gc.ca/cida_ind.nsf/49d9f10330ed2bb48525677e00735812/daa38a8fb771568985256c5600191ddc?OpenDocument

Cummings, F. H. (1992). Monitoring and Evaluation in Community Economic

Development. Community Futures Training Project. Employment and Immigration Canada.

Cummings, F. H. (1997). Logic Models, Logical Frameworks and Results-Based

Management: Contrasts and Comparisons. Canadian Journal of Development Studies, 28, 587-596.

Cummings, H. (1998). Evaluation Models. Guelph: Harry Cummings and

Associates. Cummings, H. (1998). The Logical Framework. Guelph: Harry Cummings and

Associates. Cummings, H. and Bhanjee, T. (2002). Evaluation Plan of Practice Review for

the Quality Assurance Program – Ontario College of Pharmacists. Guelph: Harry Cummings and Associates.

Cummings, H. and Bhanjee, T. (2003). RBM and Evaluation Workshop. Guelph:

Harry Cummings and Associates. Cummings, H. and Kora, G. (1999). An Evaluation Manual for Rural Community

Based Organizations. Rural Planning and Development. University of Guelph.

Cummings, H., Lipski, U. and Wasteneys, C. (1997). A Framework for

Evaluating the Quality Assurance Programs of the Colleges of Health Professions in Ontario. Guelph: Harry Cummings and Associates. Available online at: http://www.hprac.org/english/pageDisplay.asp?webDocID=2195

Cummings, H. and Morris, K. (1999). Monitoring and Evaluating the Success of

Your Community Project. New Tecumseth Economic Development Corporation. Guelph: Harry Cummings and Associates.

Den Heyer, Molly. (2001). Literature Review from: The Development of a

Temporal Logic Model. Available online at: http://web.idrc.ca/fr/ev-27872-201-1-DO_TOPIC.html

36

Framst, G. (1995). Application of Program Logic Model to Agricultural Technology Transfer Programs. Canadian Journal of Program Evaluation 10 (2),123-132

International Development Research Centre. (1994). Worksheets for Evaluation

Planning. Evaluation Unit. Ottawa: International Development Research Centre.

International Development Resource Centre. (2002). Available online at:

http://www.worldbank.org/poverty/impact/index.htm Kellogg Foundation. (1997). Evaluation Handbook. Michigan: Kellogg

Foundation. [Online]. Available: http://www.wkkf.org/Knowledgebase/Pubs/

Kellogg Foundation. (2000). Logic Model Development Guide. Michigan:

Kellogg Foundation. [Online]. Available: http://www.wkkf.org/Knowledgebase/Pubs/

Sawadogo, J. and Dunlop, K. (1997). Managing for Results with a Dynamic

Logical Framework Approach: From Project Design to Impact Measurement. Canadian Journal of Development Studies, 28, 597-612.

Taylor, A. R. and Botschner, J. V. (1998). Evaluation Handbook. Ontario

Community Support Association. Centre for Research and Education in Human Services.

Thomas, L. and C. Wasteneys. (1998). OP2000 Evaluation Handbook for

Community Based Organizations. School of Rural Planning and Development, University of Guelph

Treasury Board of Canada Secretariat. (1981). Guide on the Program

Evaluation Function. Program Evaluation Branch. Treasury Board of Canada Secretariat.

Treasury Board of Canada Secretariat. (1991). Program Evaluation Methods -

Measurement and Attribution of Program Results. Program Evaluation Branch. Treasury Board of Canada Secretariat.

Treasury Board of Canada Secretariat. (2002). Results for Canadians: A

Management Framework for the Government of Canada. Treasury Board of Canada Secretariat. Available online at: http://www.tbs-sct.gc.ca/res_can/rc_e.asp

37

Treasury Board of Canada Secretariat. (2001). Guide for the Development of Results-based Management and Accountability Frameworks. Available online at: http://www.tbs-sct.gc.ca/eval/home_accueil_e.asp

UNDP, (1997). Results-Oriented Monitoring and Evaluation. A Handbook for

Programme Managers. New York: ESP Handbook Series. Weir, Claire. (2002). Summative Evaluation: Huron Manufacturing Strategic

Initiatives Evaluation. Program Evaluation Course, 2002. University of Guelph.

World Bank - PovertyNet. (2002). Impact Evaluation website at:

http://www.worldbank.org/poverty/impact/index.htm