Measuring and Measuring and Monitoring Program Monitoring Program OutcomesOutcomes

Evaluation: A Systematic Evaluation: A Systematic ApproachApproachRossi, Rossi, et al.et al. Chapter 7Chapter 7

Ian MalcolmIan Malcolm

Definition of OutcomeDefinition of Outcome

““The state of the target population The state of the target population or the social conditions that a or the social conditions that a program is expected to have program is expected to have changed.” (Rossi changed.” (Rossi et al.et al., 2004, , 2004, p.204)p.204)

By this definition, program targets By this definition, program targets need not necessarily have changed need not necessarily have changed or been impacted by the program.or been impacted by the program.

Key TermsKey Terms

Outcome Level – the status of an Outcome Level – the status of an outcome at some point in time.outcome at some point in time.

Outcome Change – the difference Outcome Change – the difference between outcome levels at different between outcome levels at different points in time.points in time.

Program Effect – that portion of an Program Effect – that portion of an outcome change that can be outcome change that can be attributed uniquely to a program as attributed uniquely to a program as opposed to the influence of some opposed to the influence of some other factor.other factor.

Identifying Relevant Identifying Relevant OutcomesOutcomes The determination of relevant The determination of relevant

outcomes can be affected by the outcomes can be affected by the following:following:– Stakeholder PerspectivesStakeholder Perspectives– Program Impact TheoryProgram Impact Theory– Prior ResearchPrior Research– Unintended OutcomesUnintended Outcomes

Stakeholder Stakeholder PerspectivesPerspectives Stakeholders’ values are often Stakeholders’ values are often

articulated in the stated program articulated in the stated program objectives, goals, and mission.objectives, goals, and mission.

Often, the evaluator must Often, the evaluator must translate input from stakeholders translate input from stakeholders into practical form.into practical form.

Program Impact Program Impact TheoryTheory Expressed through logic modeling.Expressed through logic modeling. Outcomes are identified at two levels:Outcomes are identified at two levels:

– Proximal outcomes: services are expected to Proximal outcomes: services are expected to affect these most directly and immediately. The affect these most directly and immediately. The program has the greatest capability to affect program has the greatest capability to affect proximal outcomes, and it is easiest to attribute proximal outcomes, and it is easiest to attribute changes in these outcomes to the program.changes in these outcomes to the program.

– Distal outcomes are those that often occur Distal outcomes are those that often occur indirectly and over longer periods of time, and indirectly and over longer periods of time, and attribution of change is much more difficult, partly attribution of change is much more difficult, partly due to the fact that factors other than the program due to the fact that factors other than the program impact them.impact them.

Metal finishers attend Environmental Workshops

Increased Compliance with Enviro. Regs.

Decreased Toxic Waste Discharge

Better water quality

Program Proximal Effects Distal Effects

Prior ResearchPrior Research

Referring to prior research can Referring to prior research can call attention to relevant call attention to relevant outcomes that might have been outcomes that might have been overlooked.overlooked.

Research may turn up standard Research may turn up standard definitions and measures that definitions and measures that have policy significance. have policy significance.

Unintended OutcomesUnintended Outcomes

Outcomes not identified in the program’s Outcomes not identified in the program’s impact theory.impact theory.

Prior research may shed light on potential Prior research may shed light on potential unintended outcomes.unintended outcomes.

Program personnel may provide insight into Program personnel may provide insight into some that may arise during formative some that may arise during formative evaluation, so it is important for the evaluation, so it is important for the evaluator to be in close contact with the evaluator to be in close contact with the program during implementation (program during implementation (e.g., e.g., the the case of the Navajo schools in which certain case of the Navajo schools in which certain key information revealed an important key information revealed an important unintended outcome). unintended outcome).

Measuring OutcomesMeasuring Outcomes

Selection of important outcomes Selection of important outcomes for measurement must be done for measurement must be done carefullycarefully– Some are not essentialSome are not essential– Some (esp. distal) may not be Some (esp. distal) may not be

feasible to address because of feasible to address because of difficulty and/or costdifficulty and/or cost

Measurable outcomes must be Measurable outcomes must be observables that vary systematically.observables that vary systematically.– Some are one-dimensional and simple to Some are one-dimensional and simple to

assess (assess (e.g., e.g., whether students from XYZ whether students from XYZ CC automotive program are able to CC automotive program are able to diagnose electronic problems)diagnose electronic problems)

– Some are multidimensional (Some are multidimensional (e.g.e.g., the , the example of chargeable juvenile offenses example of chargeable juvenile offenses (p.210)), and require the assessment of (p.210)), and require the assessment of multiple facets to evaluate program multiple facets to evaluate program effectiveness.effectiveness.

Examples of Examples of Multidimensional Multidimensional OutcomesOutcomes Juvenile delinquencyJuvenile delinquency

– Number of chargeable Number of chargeable offenses in a given offenses in a given periodperiod

– Severity of offensesSeverity of offenses– Type of offense: Type of offense:

violent, property crime, violent, property crime, drug offenses, otherdrug offenses, other

– Time to first offense Time to first offense from an index datefrom an index date

– Official response to Official response to offense: police contact offense: police contact or arrest; court or arrest; court adjudication, adjudication, conviction, or conviction, or dispositiondisposition

Toxic Waste DischargeToxic Waste Discharge– Type of waste: Type of waste:

chemical, biological; chemical, biological; presence of specific presence of specific toxinstoxins

– Toxicity, harmfulness of Toxicity, harmfulness of waste substanceswaste substances

– Amount of waste Amount of waste discharged during a discharged during a given periodgiven period

– Frequency of dischargeFrequency of discharge– Proximity of discharge Proximity of discharge

to populated areasto populated areas– Rate of dispersion of Rate of dispersion of

toxins through aquifers, toxins through aquifers, atmosphere, food atmosphere, food chains, etc.chains, etc.

Since many outcomes are Since many outcomes are multidimensional, it will often be multidimensional, it will often be necessary to utilize multiple measures to necessary to utilize multiple measures to ensure that the overall impact of the ensure that the overall impact of the program can be accurately assessed, and program can be accurately assessed, and that outcomes will not be that outcomes will not be underrepresented as a result of underrepresented as a result of considering a poorly performing measure.considering a poorly performing measure.

It may be possible to statistically combine It may be possible to statistically combine related measures into a composite related measures into a composite measure with greater validity (see p.217). measure with greater validity (see p.217).

Measurement Measurement ProceduresProcedures Must be operationalized and Must be operationalized and

systematizedsystematized Often, procedures are relatively Often, procedures are relatively

standard for a given area of studystandard for a given area of study Ready-made procedures may be Ready-made procedures may be

availableavailable

Key Properties of Key Properties of Measurement Measurement ProceduresProcedures ReliabilityReliability

– The extent to which consistent results are The extent to which consistent results are obtained when measuring the same thingobtained when measuring the same thing

ValidityValidity– The extent to which the procedure measures The extent to which the procedure measures

what it is intended to measure (may be what it is intended to measure (may be assessed partly through comparison with assessed partly through comparison with alternative measures)alternative measures)

SensitivitySensitivity– The extent to which the values on the measure The extent to which the values on the measure

change when there is a change or difference in change when there is a change or difference in the thing being measuredthe thing being measured

Outcome MonitoringOutcome Monitoring

Continual measurement and reporting Continual measurement and reporting of indicators of the status of the social of indicators of the status of the social conditions the program is accountable conditions the program is accountable for improvingfor improving

Key outcome indicators must be Key outcome indicators must be selected and monitored.selected and monitored.– Should be highly responsive to program Should be highly responsive to program

effects.effects.– Should be things that only the program is Should be things that only the program is

likely to affect appreciably likely to affect appreciably

Pitfalls in MonitoringPitfalls in Monitoring

Indicators may be inappropriate or Indicators may be inappropriate or fail to address important outcomesfail to address important outcomes

““corruptibility of indicators” – corruptibility of indicators” – book-cookingbook-cooking

Interpretation problems – must be Interpretation problems – must be sure that indicators are interpreted sure that indicators are interpreted in context and other potentially in context and other potentially impactful factors are considered impactful factors are considered

Outcome Data Outcome Data InterpretationInterpretation Changes in key variables during program Changes in key variables during program

implementation must be taken into account (this implementation must be taken into account (this highlights the need for monitoring)highlights the need for monitoring)

Process and service utilization information is Process and service utilization information is important, esp. when comparing sitesimportant, esp. when comparing sites

Need standards for judging quality of outcomes within Need standards for judging quality of outcomes within the data limitations (pre-post comparisons may be the data limitations (pre-post comparisons may be helpful, though confounding effects may exist)helpful, though confounding effects may exist)

Generally, outcomes are judged by administrators, Generally, outcomes are judged by administrators, stakeholders and experts in relation to expectations of stakeholders and experts in relation to expectations of performance qualityperformance quality– These judgments are easy at the extremes, but harder These judgments are easy at the extremes, but harder

when data fall in the mid-range.when data fall in the mid-range.

SourceSource

Rossi, P. H., Lipsey, M. W., & Freeman, H. Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). E. (2004). Evaluation: A SystematicEvaluation: A Systematic ApproachApproach (7th ed.). Thousand Oaks: (7th ed.). Thousand Oaks: Sage Publications.Sage Publications.