The Evaluation Process

Nancy F. Barrett EdDFebruary 12, 2014

ASSESS PLAN

DOEVALUATE

Definition

Evaluation is the determination of the value

or worth of a product, process or program.

Type Purposes

Needs Assessment

Identify new programsReach new audiencesIncrease revenue

Make adjustmentsIncrease efficiency

Document goal attainmentReport to funding agenciesOutcomes

Improvement

Improvement OutcomesNeedsAssessment

Present

What COULD BE

What IS

What WAS

PastFuture

Question

TimeFrame

Another View

Groteleuschen A (1980). “Program evaluation”. in Knox, A. (Ed). Developing,

administering and evaluating adult education. San Francisco: Jossey-Bass.

Other Important Considerations

• Evaluation Question• Internal/External• Data Sources• Evaluation Plan• Data Analysis• Report

Evaluation Question

• Key to process• Focused• Concise

• Measureable• Negotiated

Outcome Attainment Needs Assessment

Evaluation expertise Objectivity

Lack of access to data and resourcesUnfamiliar with program

Improvement Needs Assessment

Access to internal data and resources Familiar with program

May lack evaluation expertise May lack objectivity

External Evaluation

Internal Evaluation

Purpose

Greatest Strength

Greatest Weakness

Hybrid

Qualitative or Quantitative??Qualitative Quantitative

Use

Weakness

StrengthPurpose Why What

Process Measuring Outcomes

Objectivity Context

Improvement Outcome Attainment

Use Depends on• Evaluation Question

• Availability

(Usually some combination)

Evaluation Plan“Road map”

Connects questions to data sourcesAddresses validity and reliability

Validity: Do findings address your question?Reliability: Are the findings stable over time?

Multiple data sources are important

Data Collection Methods

Surveys o Forced choiceo Open ended o Scaled response

Focus groupsInterviewsPre/post testing

Observationso Obtrusiveo Unobtrusive

Document reviewo Websiteso Newsletterso Publicationso Financial reports

Data Analysis Considerations

• Review data once it is collected.• Summarize each individual data source first.• Have someone else review work.• Make sure every data source addresses at

least one evaluation question.• Look for themes.• Use tests of significance where appropriate.

Evaluation SummaryUrbana Youth Summit

May 12, 2012

Program Origins

• Low income, high crime area of Urbana• Need for youth programming

Evaluation Design

Evaluation Question: “How effective was the conference in identifying resources and programs for Urbana youth?”Outcomes/Needs AssessmentInternal (ish) evaluationIntended data sources:

o Survey o Qualitative data during conference

1. Observations2. Content review of focus groups

Implementation ProblemLack of Role Clarity

Unfamiliarity with the structure

KeyStakeholders

left out

Survey not implemented

Any Outcomes?

Yes!Enthusiastic participants

Knowledgeable of the needsIdentified problems and solutions

Findings: Problems

• Schools• Lack of community cohesion • Lack of resources

o Housing, jobso Recreation

• Unsupervised time• Lack of “sense of self”

Findings: Strengths

• Academic and tutoring programs• Anti-bullying efforts in schools• Governmental support• Public Health initiatives• Neighborhood itself

Findings: Suggestions

• Youth programming• Community building activities• More family friendly businesses • Opportunities for parental involvement• Additional heath/nutrition programs• Job opportunities for youth• Better relationship with police• Volunteer/service opportunities

Impact (Immediate)

Youth Activities

1. City-funded summer program for youth

2. Community based summer arts program

Impact (Longer Term)

Summer arts program continuing for 3rd yearIncludes parent mentor component

Healthy Kids initiative

Community building efforts:Grant to train mentors for young menAdvocacy with city around business developmentCommunity gardening efforts

Discussing community safety with the police

Continued city support

ASSESS PLAN

DOEVALUATE

References

Barrett Evaluations. http://www.barrett-evaluations.com/forms.html.Last accessed 2/7/2014.

Barrett, N. (2013). Program Evaluation: A Step-by-Step Guide. Urbana IL: Sunnycrest Press.

Barrett N. (2012) Urbana Youth Summit Evaluation. http://urbanaillinois.us/sites/default/files/attachments/youth-summit-summary_0.pdf

Fitzpatrick J, Sanders J, and Worthen B. (2004). Program evaluation: alternative approaches and practical guidelines (3rd Edition). Boston: Allyn and Bacon.

Groteleuschen A (1980). “Program evaluation” in Knox, A. (Ed). Developing, administering and evaluating adult education. San Francisco: Jossey-Bass.