Perspectives on Impact Evaluation

Cairo, Egypt March 29 – April 2, 2009

Presented by:

Wayne M. Harding. Ed.M., Ph.D. , Director of Projects, Social Science Research & Evaluation, Inc., Burlington, MA USA wharding@ssre.org

Cheryl Vince-Whitman, M.B.A., Ed.M., Senior Vice President, Education Development Center, Newton, MA USA cvincewhitman@edc.org

A Model for Using TechnicalA Model for Using TechnicalAssistance to Improve the Evaluation Assistance to Improve the Evaluation

Capacity of Local ProgramsCapacity of Local Programs

2

Agenda

• Brief Overview of Service to Science (STS).

• Evaluation Findings About Service to Science

• Key Characteristics of The Technical Assistance Provided By Service To Science

Overview of Overview of Service to Service to ScienceScienceInitiativeInitiative

4

The Need for Service-to-Science

• Federal agencies and others in the USA have promoted the use of evidence-based programs to prevent substance abuse.

• The supply of “proven” programs is too limited to meet many local conditions.

• There are many other programs that may work, but lack evidence of their effectiveness

5

Service-to-Science Goals

• To help innovative prevention interventions that address substance abuse (or related issues) develop, improve, and document evidence of their effectiveness.

• To increase the pool of effective and appropriate interventions.

6

Organization of Service-to-Science

• Funded by Center for Substance Abuse Prevention (CSAP) in 2004.

• Implemented through 5 regional prevention centers (Centers for the Application of Prevention Technologies – CAPTs).

• In 2007, CSAP funded a related initiative for programs designed for Native Americans. Implemented by the Native American Center for Excellence (NACE).

7

Service-to-Science Activities

• CAPTs solicit prevention program nominations from State prevention officials.

• CAPTs assess the readiness of candidate programs.

• CAPTs conduct regional Service to Science Academies.

• CAPTs provide follow-up technical assistance.

8

Service-to-Science Activities (Continued)

• Service to Science Academy participants – eligible to compete for “subcontract awards” to further enhance evaluation capacity.

• CAPTs provide limited technical assistance post subcontract award.

9

Number of Programs Served Through Service to Science

• As of July 2009, the CAPTs had provided 294 prevention programs with technical assistance.

• Since 2006, 110 programs have received the subcontracts of up to $30,000.

Selected Selected Evaluation Evaluation Findings Findings

11

Methods• Pilot study (2006) interviews and/or online survey

with program Directors. Target sample of 79 programs; 58 (73%) responded.

• Six case studies of 2005 STS participants and extended follow-up case studies on 5 that received mini-subcontracts. Six case studies of 2006 participants.

• 2007 online survey targeted 142 programs that had been participants for at least 15 months. 93 responded (a 77% rate after removing 21 programs for which respondents were no longer available).

Participant Participant ProgramProgram

Characteristics Characteristics

13

Primary Behaviors Targeted for Change

Percent (a)

Alcohol Use/Abuse 80%

Illicit Drug Use/Abuse 60%

Tobacco Use 58%

Violence (including bullying) 37%

HIV/AIDS 8%

(a) Multiple responses allowed. N=93.

14

Ages of Population(s) Served

Percent (a)

Adolescents (ages 13-17) 71%

Childhood (ages 6-12) 51%

Young Adults (ages 18-25) 19%

Early Childhood (ages 0-5) 10%

Adults (ages 26-55) 10%

Older Adults (ages >55) 7%

No Distinction by Age 3%

(a) Multiple responses allowed. N=93.

15

Race/Ethnicity of Population(s) Served

Percent (a)

No Distinction 85%

Hispanic or Latino 9%

American Indian or Native Alaskan 5%

Black or African American 4%

White 4%

Asian American 3% Native Hawaiian or Other Pacific Islander 3%

(a) Multiple responses allowed. N=93.

16

Strategies Used

Percent (a) Percent

Improve Knowledge / Awareness

90% Mobilize Community 27%

Improve Skills 85% Improve Identification and Referral

25%

Healthy Alternatives 79% Improve Policies 17%

Improve Norms 59% Improve Laws 12%

Improve Mental / Physical Health

54%Improve Access to and Quality of Care

11%

Build Collaboration 36% Improve Enforcement 8%

(a) Multiple responses allowed. N=93.

Service To Service To ScienceScience

Outcomes Outcomes

18

Value of STS ServicesPercent (a)

Satisfied with CAPT TA(Very Satisfied or Somewhat Satisfied vs. Somewhat Dissatisfied or Very Dissatisfied)

72%

CAPT TA Useful (A Little, Somewhat, or Very Useful vs. Not at All Useful)

91%

Would Recommend STS to Similar Programs (Definitely Yes or Probably Yes vs. Probably No or Definitely No)

91%

(a) N=93.

19

Factors That Facilitated Progress

When asked to discuss factors that facilitated progress on their evaluation the largest percentage of clients (79%) identified TA from the CAPTs.

(a) From the 2006 pilot study. N=58.

20

TA Contributed to Program Capacity(a) in 5 Most Common Issues

Percent (b)

Improving understanding of program rating systems such as NREPP & NASADAD

95%

Improving understanding of evaluation 95%

Developing and/or improving a logic model 92%

Developing and/or implementing a more rigorous evaluation design

95%

Improving organizational commitment to evaluation

84%

(a) Contributed “A Little,” “Somewhat,” or “A Great Deal” vs. “Did Not Contribute at All.” (b) N=93.

21

Seeking Recognition and/or Funding

Percent (a)

Plan to seek or are seeking recognition from NREPP or NASADAD

57%

Sought recognition from local, regional or other national organizations

13%

Prepared a journal article 13%

Presented evaluation findings at professional conference or meeting

13%

Used evaluation findings to help secure funding

23%

(a) N=93.

Defining Defining Features Of Features Of Service To Service To Science TAScience TA

23

Defining Features of Service to Science TA

• Client-centered

• Customized

• Long-term

• Relationship-based

• Flexible