FORMATIVE EVALUATION

Intermediate Injury Prevention Course

August 23-26, 2011, Billings, MT

Session Goal

To provide participants with a basic knowledge of formative evaluation and

its use when developing an injury prevention program.

Describe the purpose of formative evaluation

Describe several formative evaluation methods

Provide resources for each formative evaluation method

Session Objectives

The 4 Stages of EvaluationReview

Formative Will this work?

Process Are we doing what we set out to do?

Impact Can we see any changes?

Outcome Have we reduce injuries?

The 4 Stages of EvaluationReview

Process Evaluation

Impact Evaluation

Outcome Evaluation

Long-term:

Timely Feedback

Baseline Data Collection Project Implementation

Pre-Testing MonitoringHealth Outcomes

Looks at Changes in Target Audience

Formative Evaluation

Short-term:Knowledge, Attitudes & Practice

Formative EvaluationDefined

The process of testing and /or obtaining expert review of program plans,

messages, materials, strategies, or modifications, for weaknesses and

strengths before they are put into effect.

Formative EvaluationPurpose

Get target audience perception of the possible interventions

Gain community support Ensure program materials & activities are

acceptable or culturally appropriate Identify problems with program concepts

or design Used in conjunction with process

evaluation

Formative EvaluationWhen to Conduct

New program New materials Existing program is:

modified, having problems, or adapted for a new setting, population,

problem, or behavior

Ongoing at all stages of program

Formative EvaluationWho to Involve - Stakeholders

The people with a vested interest in a program and the future use of

its evaluation

Formative EvaluationCategories of Stakeholders

Implementers

Partners

Participants

Decision makers

Formative EvaluationStakeholders

Example: Car seat distribution program

Who should we include?

Implementers

Partners

Participants

Decision makers

Formative EvaluationData Collection Methods

1.Focus Groups2.Key Informant Interviews3.Town Hall Meetings4.Expert Reviews5.Surveys

Formative EvaluationFocus Groups - Purpose

To gather information on a particular topic by conducting an informal discussion between 6 to 10 individuals similar to target population

Formative EvaluationFocus Groups

8 Important Steps

Secret Ingredients:-Focus group similar in makeup to target population-Requires an experienced facilitator

Formative Evaluation Key Informant Interviews - Purpose

To obtain detailed information by asking broad open-ended questions

and by probing interviewee

Formative EvaluationKey Informant Interviews

7 Important Steps

Secret Ingredients:-Take care to select informants with various points of view-Remember to first establish rapport

KEY INFORMANT EXERCISE

Scenario:Local IPS is seeking to determine attitudes and beliefs about ATV related injuries and possible interventions. They will interview two key community members.

Formative EvaluationTown Hall Meetings - Purpose

To gather a diverse group of community members together to think and talk about

an issue

Formative EvaluationTown Hall Meetings

8 important steps

Special Ingredients Plan early Make sure there is plenty of time for interaction

between audience and presenters

Formative EvaluationExpert Reviews - Purpose

To identify mistakes, concerns, and general areas for improvement.

Formative EvaluationExpert Reviews

7 important steps

Special Ingredients The committee should be continuously

involved in the process. Include detailed instruction in the job

description.

Formative EvaluationSurveys - Purpose

To obtain SPECIFIC information from a targeted group of people using a set of defined questions.

Formative EvaluationSurveys

7 important steps

Special Ingredients Avoid overloading the survey Design questionnaire such that its clear

and simple.

Formative EvaluationExamples in IHS

Sleep Safe and Ride Safe Curriculums – Focus Groups, Key Informant Interviews, Expert Review

Animal Control Project - Surveys Child Passenger Safety Training - Expert

Review IHS Area Injury Prevention Program

Evaluations - Expert Review

Formative EvaluationConclusion

Formative evaluation Tests program plans, messages,

materials, strategies, or modifications for weaknesses and strengths before they are put into effect.

Allows the evaluator to probe the feelings, beliefs, and impressions of participants.

Generates community support for the proposed project or program.