Data-driven Approaches to Monitoring and Evaluating Environmental Strategies

Training Workshop for Vermont Community Prevention Coalitions

March 20, 2012

PIRE

TODAY’S TRAINERS AND PRESENTERS…

VDH: Lori Uerz Suzanne Kelley

PIRE: Bob Flewelling Amy Livingston

Vermont Center for Rural Studies: Erin Roche

AND WORK GROUP #1 FACILITATORS….

Cindy HayfordBeth CraneMaryann Morris

And now…. Introductions

Just What Just What IsIs Environmental Prevention, Environmental Prevention, Anyway?Anyway?

Environmental strategies in a community seek: Environmental strategies in a community seek:

1.1. To bring about system-level change (including To bring about system-level change (including physical space, local community policies, availability physical space, local community policies, availability of drugs and alcohol, etc.) of drugs and alcohol, etc.)

in order in order

2.2. To reduce substance abuse problems (increase To reduce substance abuse problems (increase health and safety) at the population level. That is health and safety) at the population level. That is Public Health.Public Health.

Both conditions must be met.Both conditions must be met.

One of many examples…..

What about Strategies Designed to Change Community Norms?

• Yes, norms are an attribute of the community environment.

• Communications strategies (e.g., media campaigns) can contribute to normative change

• But are most effective when combined with other environmental change strategies; for example:– Alcohol free events– Bans on outdoor advertising– Stricter enforcement of alcohol laws

Environmental Strategies:Implications for Evaluation

Long-term outcome measures should be at the population-level;

The short-term outcome measures (intervening variables) will typically be environmental conditions or perceptions of such

Implementation often is less well prescribed, creating challenges for planning, implementation, and process evaluation

Evaluating Community-Based Prevention Strategies: For Who?

First and foremost, the evaluation serves the community organization that’s doing the prevention work: By documenting activities By monitoring implementation and making

adjustments as needed By indicating if short- and long-term objectives are

being achieved (or going in right direction) By providing empirical data on activities and progress

to be shared with stakeholders

Evaluation Components

Evaluation involves collecting and interpreting data on: Process

(information regarding implementation of strategies) Short-term outcomes

(also referred to as intervening variables) Longer-term outcomes

(indicators of the target problem being prevented or reduced)

Assessing Outcomes

Short and long term outcomes need to be measured at multiple points in time (or at least two points)

Focus is on whether and how measures change over time

Comparisons to the state (or nation) may enhance interpretation

Challenges and Limitations of Evaluating Community-Level Strategies Focused on Population-Level Change

• Community is an N of 1• There are many external factors that

influence outcomes• Population-level outcome data requires

different approaches to collect (no “captive” audience)

• Standards for implementation (especially intensity) less well known

Implications for Interpreting Findings from Community-Level Evaluations

• Cannot definitively attribute positive changes to your interventions

• But positive changes can suggest your interventions deserve some of the credit*

• Lack of positive changes does not mean your interventions were ineffective

* Especially if evidence-based strategies were well-implemented, intervening variables also changed, and you have some reasonable comparison data.

Whether your evidenced-based strategies were appropriate and well implemented (hence the importance of process evaluation)

Whether short and long term outcome indicators are moving in:

the “right direction” or the “wrong direction”

(This is also referred to as “program monitoring” or “performance monitoring)

Essential Foci of Any Community-Level Program Evaluation

Logic Models

Simply a way of showing the connections between: Strategies Desired Outcomes The mechanisms through which the strategies are

designed to work (i.e., intervening variables) May be expanded to also show the measures

for each intervening variable and outcome Provide a blueprint for outcome evaluation

Substance-related

consequences and substance

use

Intermediatevariables

Programs/ policies/practices

Implementing the Strategic Prevention Framework

Planning, Monitoring, Evaluation, and Replanning

Outcomes-Based Prevention

Important Attributes of Logic Models

Can be depicted with a variety of formats Show the connections between each strategy

and each IV the strategy is expected to change Shows the measure(s) to be used for each IV

and outcome Identifying the “contributing factors” for each

IV will help define appropriate measures Usually best to focus each model on a single

ultimate outcome

One Example of a Logic Model Template

One Example of a Logic Model Template(Filled In)

Measuring Intervening Variables

Each intervening variable may have multiple possible measures (i.e., “contributing factors”)

Typically there is more than one way to measure an IV

Desirable attributes of measures: Valid Periodic collection (with consistency) Available (free or for minimal cost) Reflective of the target population

Data Sources for Intervening Variables

Established surveys (e.g., YRBS) Other surveys (e.g., a local parent survey) Archival data (e.g., compliance check results from

Dept. of Liquor Control) Public records (e.g., liquor outlet locations,

enactment of policies, enforcement activities) Observation (e.g., community scans) Active surveillance (e.g., purchase surveys) Focus groups Interviews with key informants

Measuring Outcomes

Same criteria and sources as for IVs In addition, keep in mind:

Outcomes based on small numbers of events are not very sensitive (e.g., fatalities)

Measures that may reflect the intensity of intervention efforts are not good measures (e.g., arrest rates, treatment admissions)

The further a measure is from the strategy in the logic model, the more external influences there are

Monitoring Implementation

What did you do? How much did you do?

And how well did you do it?

Why is it important to track and monitor implementation activities?

Document what was done so: You can determine how well it was done You and/or others can repeat what was done

The information can be used to adjust and improve implementation going forward (formative evaluation)

The information can be used to help understand outcome findings

Monitoring Helps Us Understand Why Strategies Succeed or Fail

25

It’s not a miracle!

If you are using appropriate evidence-based strategies and implementing them well (following available implementation guidelines closely, completing all core activities with minimal deviations, enough dosage and intensity)

You can expect that they will lead to the desired outcome!

Steps for effective program monitoring

1. Identify the core components/activities for your strategy

2. Develop an implementation plan that lists out the core activities and timeline for completion.

Strategy Implementation Plan Organization Name:

PROBLEM(S) ADDRESSED

TARGET POPULATION SHORT-TERM MEASURES OF SUCCESS

1. 2. 3.

STRATEGY KEY ACTIVITIES: Description of key activities that will be completed to fully implement the strategy

TIMEFRAME: Specify start date for each key activity

KEY PARTNERS: Partners you will work with to carry out each activity

PROGRESS: Describe your progress including, but not limited to, process measures (dates of meetings, number of participants/media spots/meetings) and completion date for each activity

1.

Steps for effective program monitoring contd.

3. As you implement the activities, document progress including anything that can be measured (number of meetings, number and types of media spots, how many people reached, etc.)

Steps for effective program monitoring contd.

4. Assess progress toward short-term outcomes and make adjustments as needed. Is this moving us toward our end goal?

RECAP OF KEY POINTS• Environmental strategies focus on:

• Changing the context, not the individual• Population-level outcomes

• ES require different (and creative) approaches for measuring process

and outcomes

• Although outcome monitoring is important, conducting conclusive

outcome evaluations is not the goal for community-level efforts

• Implementing evidence-based strategies with the highest possible

fidelity should be a goal for every community

• Dramatic change usually takes time and perseverance (e.g., smoking

rates in the U.S.)