WASHINGTON STATE UNIVERSITY EXTENSION

Evaluation Evaluation Based Based

Program Program DevelopmentDevelopment

This material was developed in partnership with Rutgers's University and

Lydia B. Blalock, Youth Development Specialist, New Brunswick, New Jersey

Why Evaluate?

• I dunno• I think I knew why–

once• They make me do it

– Department chair– Funding

organization– My mom

• I don’t

• Demonstrate success• Understand failure• Provide feedback for

program improvement– Process– Outcome

• Nip problems in the bud

Process Objectives

If I:

• Develop and present seminar on EBPD

• Develop and distribute appropriate handouts

• Engage participants in group activities

• Provide participants an opportunity to practice

• Encourage questions

THEN

Outcome Objectives

Participants will: • Program Development

– Understand how to clearly define issue program will address.

– Use program theory to plan activities.

– Write clear outcome and process objectives.

– Use EBPD Model to plan programs.• Program Evaluation

– Develop skills to evaluate programs.

– Select appropriate evaluation methods and designs.

– Select criteria to evaluate objectives.

– Use EBPD Model to plan evaluation.

BUT FIRST: Is it a Program or a program

PROGRAM

program/s event/sactivity/ ies

•Stakeholder leadership

•Faculty leadership

•Multi-disciplinary

•Sustainable

•Academic department

•Student component

•Research, teaching, outreach

PROGRAM

program/s event/sactivity/ ies

The Steps of Evaluation-Based Program Development

5. Implementation6. Dissemination & Scholarship 41. Define the Problem

4. Plan Program Evaluation

1. Define the Problem

4. Plan Program Evaluation

5. Implementation

6. Dissemination & Scholarship 4

3. Plan ProgramÑ

2. Develop Outcome Objectives Ï

3. Plan ProgramÑ2. Develop Outcome Objectives Ï

Step 1: Define the Problem

• Describe Current Situation – How does the world look today?

• Why a Problem– Clearly articulate why a problem – Ask why until get to root of problem– Symptom vs. Cause

• Identify Stakeholders– Population: Who is affected? How

many?– Stakeholder Leadership: interest in

solving problem?– Potential Collaborators: Already

working on problem?• How: Evidence? Learn more?

References & Facts– Learn all you can about the

issue/problem –

• Set Priorities– Factors that influence priorities

• Difference between actual and target states

• Degree of difficulty in addressing needs

• Cost of implementing solutions• Political and other factors

– Derive priorities from several data sources

– Priorities without resources• Select Solution Strategies to Meet

Needs– Consider alternative solutions for

each need– Evaluate alternatives – Select solutions to implement– Scholarship Opportunities?

Table 1. Define the Problem Identify who, what, where, when, why.

1. Describe Current Situation How does the world look today?

2. Why a Problem?

3. Identify Stakeholders

Clearly articulate why current situation is a problem. Ask why until you get to root of problem.

Population. Who is affected by problem? How many people are affected?

Stakeholder Leadership. Who’s interested in solving problem?

Potential Collaborators. Who’s interested in solving problem and/or already working on problem?

What is the evidence? How can I learn more? Provide references and facts.

TALK IT OVER

Fill Out Table 1 with your group.

Step 2: Develop Outcome Objectives

• Describe Ideal Situation– How would world look w/o

problem (perfect world)?• Define Target Audience/s

– Who/what can we change (youth, families, policies, etc.)?

– What do we know about culture/s of audience?

• Identify Outcome Objectives– What changes can I influence in

my target audience/s?• Identify Resources

– What do we need?– What do we have?

Step 2. Develop Outcome Objectives

2. Develop Outcome ObjectivesÏWhat can we change in audience?

Define Target Audience

Who can we change to solve problem?

Identify ResourcesWhat resources are required, available,

needed?

Short Term LEARNING& Awareness, Skills, Attitudes, Opinions,

Knowledge Aspirations, Motivations

Medium Term ACTIONS·Behavior, Practice, Decision-making,

Policies, Social Action

Long Term CONDITIONS þSocial, Economic,

Civil, EnvironmentalIMPACT?

Writing Measurable Objectives

• Specific– Will increase knowledge vs. Will

know 3 sources of Vitamin C

• Time delineated– At the end of the program, in 6

weeks, etc.• Measurable

– 75%, all, etc.• Example

– Participants will increase knowledge of nutrition.

– 75% of program participants will know 3 sources of Vitamin C 6 weeks after the program.

Table 2: Clarifying Your Projected Impact

Table 2. I dentify Outcome Objectives

1. Describe I deal Situation How would the world look if the problem did not exist (the perfect world)?

3. I dentify Outcome Objectives What changes can I influence in my target audience/s?

2. Define Target Audience/ s Who or what can we realistically change (youth, families, farmers, policies, etc.)? What do we know about the culture/s of the audience? Do we need to learn more?

Learning Objectives

Short term changes in awareness, skills,

knowledge, attitudes,

aspirations, motivation. List for

each target.

Action Objectives

Medium term changes in

behavior, policies, practice, decision-

making, social actions. List for

each target.

Condition Objectives

Long term changes in social, civic, economic, environmental

conditions. Impact?

Target 1:

Target 2:

4. Identify Resources

Resources Required Revenue Enhancement Development Opportunities

Talk it Over

Fill out Table 2 with your group.

#4: Brainstorm Resources

Step 3. Plan Program

Plan ProgramÑ(Activities)

Use “Program Theory” to develop process objectives! IF:THEN

Develop Process ObjectivesÑWhat will you do to facilitate changes in

target audience/s?

Program Theory: The Key to Unlocking Change

IF we

do this . . .

THEN they

will change . . .IF:THEN StatementsIF we ABC: THEN XYZ

where ABC = the events, programs, activities you will conduct, deliver, develop, train, provide, access, facilitate, etc.

and whereXYZ = participant outcome objectives previously developed

ABC will become your process objectives.

Process Objectives

The actions you will take to produce the desired

outcomes (write curriculum, conduct

training)

Outcome Objectives

Changes in target population as result of

your actions(teachers adopt

curriculum, kids are respectful)

PROCESS:OUTCOME Statements

Delivery Modes

Projects and ActivitiesResultsPRODUCE

Output the Process of an Outcome …wha…?

Outcome Objectives: the changes you will influence in your targetaudience

(Learning, Action, Condition) Process Objectives: the events, programs,

activities you will conduct, deliver, develop, train,provide, access, facilitate to reach your outcome objectives

Program Outputs: your deliverables

Outcome & Process Objective Activity

Talk it Over

Fill out table 3 with your group

SO WHAT?

Friend to Groucho Marx: “Life is difficult!” Marx to Friend: “Compared to what?”

Why We Don’t Evaluate

• Too hard• Not qualified• Costs too much• Takes too much time• Don’t think about it “in

time”• No one ever reads

evaluation reports• Everyone “knows” it’s a

good program

Step 4: Program Evaluation

4. Plan Program Evaluation

Specify Process Indicators ÑHow will you know if you

successfully implemented program?

Specify Outcome Indicators ÏHow will you know if the audience achieved the specified outcomes?

Select Evaluation Design Select Data Collection Method

Evaluation Design 101

• One-Shot– Data collected following

program,• Retrospective Pretest

– Data collected following program; participants recall behavior prior to and following program.

• One Group Pretest-Posttest - Data collected prior to and following program• Time Series

– Data collected prior to, during, after program

• Pretest-Posttest Control-Group– Data collected from 2

different groups prior to and following program.

– Treatment group: program– Control group: no program

• Post-Only Control-Group– Data from 2 different groups

collected following program only

Evaluation Design 101

Evaluation Design PretestPosttes

t# Times

AssessedFollow-up

Control Group

One-Shot Design X 1

Retrospective Pretest X X 1

One Group Pretest-Posttest X X 2

Time Series X X 3 (min) X

Pretest-Posttest Control-Group

X X2x per group

X

Post-Only Control-Group X1x per group

X

Data Collection Methods

• Face to Face Surveys– Respondent completes survey in presence of evaluator– Evaluator may record responses and is able to answer

questions• Self Administered Questionnaires

– Respondent given survey to complete on his/her own.– Respondent’s responsibility to return survey.

• Personal Interviews– Respondent asked pre-determined questions. – Evaluator records responses, can ask respondent to

clarify answers.• Focus Groups

– Several respondents meet with evaluator. – Evaluator facilitates discussion around pre-determined

questions. – Discussion recorded.

Learned Anything? Let’s Check!

Learning, Action & Condition Objectives

Learning Objectives Short term changes in awareness, knowledge, skills, attitudes, aspirations, motivation

1. Develop Success Indicators NOTE: May have multiple indicators for each objective.

2. Select Evaluation Design*

3. Select Data Collection Method*

We will know (insert Learning Objectives)

I f (insert criteria for success)

Type Pro/Cons Method Pros/Cons

Think it Over

Fill out Table 4 with your group.

Learning IndicatorsAction IndicatorsCondition Indicators

Cultural Aspects of EBPD

• What is culture?– “Behaviors and beliefs characteristic of

a specific social, ethnic, or age group.” – We are a product of our culture– Culture influences (unconsciously) what

we think of people from other cultures. – Recognize cultural biases towards

participants and strive to eliminate them.

• Be aware of how differences between cultures may impact EBPD

• Neutralize cultural biases and prevent EBPD design flaws:– Know your audience (Step 2 – Develop

Outcome Objects)– Develop cross-cultural efficacy!

Culturally Sensitive Design

Cultural Competence vs. Cross-cultural Efficacy

Cultural competence

I can deal with them.

My culture is superior to their culture. (It’s all about me!)

Cross-cultural efficacy

I can effectively interact with people from other cultures.

I can benefit from our interactions. Neither culture is superior to the other. (It’s all about us!)

Reporting & Scholarship

Table 6. Dissemination and Scholarship How will you share your work?

Evaluation-Based Program Development

1. Identify Scholarship Activities

2. Identify Products & Deliverables

Table 1. Define Problem

Table 2. Identify Outcome Objectives and Resources

Table 3. Plan Program

Table 4. Plan Program Evaluation

Table 5. Implement Program, Conduct Evaluation and Document Impact

Table 6. Dissemination and Scholarship

Report Results

Demonstrate Impact on Profession

Attain Recognition

3. Plan ProgramÑ

2. Develop Outcome Objectives Ï2b.

ResourcesÏ2a. Target

Audience Ï

4. Plan Program Evaluation 5. Implementation 6. Dissemination & Scholarship 4

1. Define the Problem

Process Objectives

If I, as facilitator:• Develop and present

seminar on EBPD• Develop and distribute

appropriate handouts• Engage participants in

group activities• Provide participants an

opportunity to practice • Encourage questions

THEN

Outcome Objectives

Participants will: • Program Development

– Understand how to clearly define issue program will address.

– Use program theory to plan activities.

– Write clear outcome and process objectives.

– Use EBPD Model to plan programs.• Program Evaluation

– Develop skills to evaluate programs.

– Select appropriate evaluation methods and designs.

– Select criteria to evaluate objectives.

– Use EBPD Model to plan evaluation.

Condition Changes

We’ll report stunning impacts from well planned programs

And the folks in Pullman will love us the most

And people will dump bucket loads of money into our programs

And the world will be a better place!