Program Planning & EvaluationBegin with the End in Mind

Dr. Dallas L. HolmesDr. Dallas L. HolmesSpecialist Institutional Specialist Institutional ResearchResearchUtah State University Utah State University ExtensionExtension

Program Planning Process

Avoid Frustration: Ask these Questions Before Beginning Your Program Not After It Has Begun

•What will you evaluate?

•What will you evaluate immediately?•Less intense skill and knowledge development

•Pre-Post test or Retrospective Tests

•What will you evaluate via follow-up assessment?•More intense long-term effects, behavior changes, skills and knowledge

•Surveys or telephone interviews

Key Factors to Planning

•Keep the “main thing” (Objectives and Goals) the ‘main thing.’

•Is it worthwhile to evaluate a one-shot brief program?

““Two hour rule.Two hour rule.””

•Evaluate larger efforts, i.e. program series over a year or five years, multi-dimensional efforts to teach, change behavior which lead to measurable IMPACTS.

•Will follow-up “Outputs” be necessary to achieve the desired outcomes? Have you planned for these?

Identification of the Problem or Issue

1. Clear vision of needs and assets of your target Create a statement page of the problem or issue.

2. Is there clientele consensus leader and agency “buy in”

3. Facilitate public awareness of problem or issue Foundation of support

4. Put together statements to create public awareness, seek program and financial support

Contact stakeholders at beginning of program for their support not at the end of the program.

Needs Assessment

•Who should be assessed or asked about needs?•BUGS Formula

•How will assessment be completed?•focus groups

•interviews

•written or telephone interviews

•self-report surveys

•Review available social and scientific indicators surrounding the issue or problem.

•Demographic changes, percentages, increases and decreases in…

Writing Goals

•Long term goals - one to five years to accomplish

•WHO will DO WHAT for WHAT PURPOSE?

•Verbs which usually accompany goals include

•change, promote, offer, increase, decrease, train or enable.•Reported Utah bankruptcies in Utah will decrease by 15% in 2000 as a result of the Financial Management, Budgeting Awareness program conducted by Utah State University Extension

Micro Level Macro Level

Community/neighborhood,region/state - levels

Individual, family, ranch, farm, club - levels

Goals

Written on

Writing Objectives - The Basics

•Short Term Objectives - One year or less to accomplish

•They Target:

•knowledge, attitudes, skills, aspirations, tax dollar support, behaviors and other

program characteristics which will change as a result of the intervention.

SMARTSMART

S - Specific

M - Measurable

A - Attainable

R- Relevant

T- Time-based

The FormulaThe Formula

By (DATE) (WHO) will (DO WHAT IMPLEMENTING/ACTION VERB) for

(WHAT OUTCOME)?

Evaluation Plan

Overall purpose for the evaluation, and specific questions the evaluation needs to answer. Short summary of what the evaluation is to accomplish

Methods to be used evaluation strategy

mail survey, telephone survey, etc.

target audience from which data is collected

time frames persons responsible

How Results will be used

Budgetary Needs

Model for Evaluation Planning

Outcomes Specific Information Needed- Indicators

Data Collection method(s)

Date(s) for data collection

Data source(s)

Program Goals: Integrated Weed Management Program

Ecologically basedweed management

Reduced herbicide use

Utilization of IPM

Utilization of other weed control methods cultural,biological, GMO’c

Data searches USDA statistical baseline

Dealer Coop records

Commodity group data records

Nov-Feb statistical collection from groups

Measures for pre-herbicidal June

On-going IPM measures

USDA commodity groups

Special interest groups

Commercial applicators

Scientists involved in specialized research

Plan EvaluationBegin by asking the BIG questionBegin by asking the BIG question

What outcomes do I hope to show from this program?•Social•Environmental•Economic

Begin with the end in mind!Begin with the end in mind!

Outcomes and Success MarkersOutcomes and Success Markers

•We expect to see

•We would like to see

•We would love to seeSuccess Markers = Identified actions/behaviors which indicate successful accomplishment of the outcome

??1.Who:

2.Clause which describes the successful attainment of identified change

3.Complete statement with a behavioral intention that represents a significant attainment for the person targeted

Outcome #1

SM1

SM2

SM3

Assess Outcomes Measuring Impacts

Techniques to consider include:

•Interviews

•Focus Groups

•Post-test only, post-test then pre-test, pre-test-post-tests, pre-test-post-test follow-up evaluations, experimental-control groups, quasi-experimental or comparison groups

Using Measures that determine changes in (KASA)

•Knowledge

•Attitude

•Skills

•Aspirations

Measuring for change in Practices•Interventions -That which has changed peoples lives

•Social

•Economic

•Environmental

SEESEE

Reporting Results to Stakeholders

Remember who the stakeholders are you previously identified in the initial program planning. REPORTS MAY BE DIFFERENT FOR VARIOUS STAKEHOLDERS. Address the stakeholders’ needs!

- One or two basic demographics about the problem

- Your basic objectives/ intention of your program

- What you did

- What results/outcomes you found. What happened as a result of the program?

Generally no longer than one page, concise, organized and clearly presented. Consider an executive summary approach.

A Basic Outline to Consider:

Redesign the Program Based on Evaluative Results

The most important use of your evaluation results is to use the results to redesign your program so that it will be more effective.

ASK:ASK:

•What did you learn?

•What worked?

•What did not work?

•What can you learn about your approach or teaching style?

•What did you learn about your audience?

•What additional services should/could be added?

Loop back to the beginning phase of program planning and apply the learnings to each step in the planning process.

Program Planning Process

Redesign Program

12

3

456

7