Process Evaluation

Intermediate Injury Prevention CourseAugust 23-26, 2011

Billings, MT

Session Goal

To provide Participants with the information to design a process evaluation for an injury prevention project.

Session Objectives• Define Process Evaluation• Describe why and when to use process

evaluation• Recognize the type of data collected

during Process Evaluation• Describe Process Evaluation methods• Design a Process Evaluation for an

injury prevention project

Formative Evaluation

Process Evaluation

Impact Evaluation

Outcome Evaluation

Short-term:

Long-term:

Timely Feedback

Baseline Data Collection Project Implementation

The 4 Stages of EvaluationReview

Pre-Testing MonitoringHealth Outcomes

Knowledge, Attitudes & Practice

Changes in the Target Audience

Process EvaluationDefined

The type of evaluation used to determine if your project is:

• Being implemented as planned, and

• Reaching its target audience.

Process EvaluationDefined

“Did I do what I set out to do,

and did it make a difference?”

Process Evaluation

Impact/Outcome Evaluation

Process Evaluation asks:– What was actually done?

– Where and when was it done?

– How often was it done?

– Who did it, and who did they do it for?

Process EvaluationDefined

• Process evaluation planning takes place during project planning and before formative evaluation starts.

• Begins immediately after your project is implemented.

• Continues throughout the life of your project.

Process EvaluationWhen its Used

• Allows you to make adjustments in a timely manner.

• Needs of target population might change, and project may need to adapt.

• Identifies any problems that occur in reaching the target population.

Process EvaluationWhy it’s Used

• Can be used to show funding agencies the project’s level of activity.

• Tells you how well your project is being implemented.

• Tells other interested programs, the “how” and “why” your program works.

Process EvaluationWhy it’s Used

• Is necessary but not sufficient in evaluating a program’s effectiveness.

• Depends upon accurate record keeping and effective communication.

Process EvaluationLimitations

Process EvaluationMeasures

Implementation Target Audience

Process Evaluation

Measures if your project is reaching its target audience

Measures & evaluates project implementation

Process EvaluationImplementation

Implementation activities:• Developing project’s goals and objectives• Creation of an implementation protocol• Monitoring daily operations • Data collection • Coalition building • Ensuring staffing is at proper level to meet

program needs• Ensuring that staff are sufficiently trained

Process EvaluationImplementation Questions

• What was actually done?• When and where was it done?• Who did it?• Who did they do it for?• Were any materials distributed?• What barriers or challenges were

discovered?• What was the cost?

Process EvaluationTarget Audience

Reaching the Target Audience activities:• Ensuring the target pop. is being reached• Measuring program participation by target pop.• Making materials and resources available and

understood by target pop.• Determining if program is relevant to target pop. • Tracking the distribution of materials

Process EvaluationTarget Audience Questions

• Is the target population being reached?

• What was the nature of this contact?

• How often and for how long was the target population involved?

• Are the project’s messages and materials appropriate for the target population?

Process EvaluationData Collection Techniques

A. Project Exposure

B. Progress Review

C. Internal Audit

D. Target Population Survey

E. Project Site Survey

Process EvaluationData Collection Techniques

A. Project Exposure• Monitors all project contacts (telephone,

email, classes, etc.) and materials distributed (brochures, products, etc.)

• Monitors who utilizes project information (e.g., sign-in sheets)

Process EvaluationData Collection Techniques

B. Progress Review• Reviews program activities to determine if

goals and objectives are being met (e.g., meetings with coalition, tribal members, courses provided)

• Conducts interviews with staff and coalition members

• Conducted by project staff and interested parties

Process EvaluationData Collection Techniques

C. Internal Audit• Compares implementation plan to actual

activities • Documents staff efforts, resources,

amount of time devoted to each task, and date of task completion

• E.g., Sleep Safe Coordinator’s quarterly reports

Process EvaluationData Collection Techniques

D. Target Population Survey• Measures whether program is reaching

target audience • Describes target population’s awareness

of project, level of interest in the project, number who utilize project

Process EvaluationData Collection Techniques

E. Project Site Survey• Determines if materials are being

distributed effectively• Monitors the number of materials used

over a certain period of time • E.g., Sleep Safe smoke alarm

follow-up data collection form

Process EvaluationInterpreting the Data

The 5 process evaluation methods are now producing results…

Results of Internal Audit are used to inform project staffNo problems discovered: status-quoProblems discovered: timely revisions to projectMay require Formative evaluation to determine the problem’s cause

Formative Evaluation

Process Evaluation

Impact Evaluation

Outcome Evaluation

Short-term:

Long-term:

Timely Feedback

Baseline Data Collection Project Implementation

Process EvaluationInterpreting the Data

Pre-Testing MonitoringHealth Outcomes

Knowledge, Attitudes & Practice

Changes in the Target Audience

• Process evaluation is a management tool that is used to make sure that your project is implemented as planned and on schedule.

• Information collected during process evaluation can be used to make adjustments to your project.

• Tells you how well your project is being implemented.

Process EvaluationConclusion

Process EvaluationConclusion

“The only difference between stumbling blocks and stepping stones is how you use them.”

(And your ability to recognize them.)

• Using the provided goal & objectives and the Process Evaluation worksheet, your group should:– Select an objective.– Describe one way you will measure how

well your project is implemented and how it will reach the target audience.

– Describe how you will measure the objective using one of the process evaluation data collection techniques.

Process EvaluationExercise

Time allowed: 15 minutes

Child Passenger Safety• Goal: ..to increase the use of child

passenger safety seats & correct use of seats

• Objectives:– Coalition to increase number of CPST and

Instructors by end of this year.– Media advertisement of clinics,

checkpoints, and importance of car seats by March 2007.

– Conduct a clinic & checkpoint per community yearly, starting next year.

Formative Evaluation

Process Evaluation

Impact Evaluation

Outcome Evaluation

Short-term:

Long-term:

Timely Feedback

Baseline Data Collection Project Implementation

The 4 Stages of Evaluation

Pre-Testing MonitoringHealth Outcomes

Knowledge, Attitudes & Practice

Changes in the Target Audience

Impact Evaluation

• Collects baseline information of people’s knowledge, attitudes, beliefs or behaviors.

• Long-term data collection (3 to 5 years) to measure and reduce morbidity mortality.

Impact Evaluation

• Changes in elders use (observed, self-reported) of walkers.

• Changes in community members behavior in using occupant restraints (seat belt wearing for adults or car seats for children.

Impact Evaluation

• Children bicycle helmet use (observed, self-reported) before/after a bicycle campaign.

• Changes in elders use (observed, self-reported) of walkers.

• Changes in number of operable smoke detectors installed/maintained in homes after a comprehensive fire prevention campaign.

Impact Evaluation Example: Alaska PFD Promotion Project

• Drowning rate in Alaska’s YK Delta more than 3 times greater than the State average

• PFD promotion project began in YK Delta

• Observational surveys of PFD use were conducted after baseline data was collected

• Timely changes made to increase sales and PFD use

Impact Evaluation Example: Alaska PFD Promotion Project

Usage Rates %

# Sales

4930 29

77

125

241200

300

59 58 58

898891

0

50

100

150

200

250

300

350

90 91 92 93 94 95 96 97 98

0102030405060708090100

Formative Evaluation

Process Evaluation

Impact Evaluation

Outcome Evaluation

Short-term:

Long-term:

Timely Feedback

Baseline Data Collection Project Implementation

The 4 Stages of Evaluation

Pre-Testing MonitoringHealth Outcomes

Knowledge, Attitudes & Practice

Changes in the Target Audience

Outcome Evaluation

• Usually requires significant resources, long periods of time, and ongoing data monitoring.

• Used less frequently than impact evaluation in NA injury prevention programs.

• Focuses on the program’s long-term effect on its target audience.

• Conducted after a program has been completed

Outcome Evaluation Example: Drowning Rates YK Delta vs. All-Alaska

0

10

20

30

40

50

60

70

80

90

88 89 90 91 92 93 94 95 96

YK Starts PFD Program

YK Delta

All Alaskan

43% Reduction

Rate Per 100,000

Evaluation Summary:Identify the evaluation type

• Number of PFDs distributed • Decrease in the number of drinking and

driving violations • Review of IP materials describing storage of

poisons • Number of meetings with Tribal council to

discuss possible speed limit ordinance• Number of completed suicides

Additional Resources

www.nhtsa.dot.gov

www.cdc.gov/eval/resources.htm