evaluation 101 for non-researcher practitioners...charlene zil, npc research nadcp conference july...

Evaluation 101 for Non-Researcher Practitioners

Charlene Zil, NPC Research

NADCP Conference

July 2017

EVALUATION 101 FOR NON-RESEARCHERSWhat to Collect and How to Use it for Self-Review

2

OVERVIEW

• Why do evaluation?

• What is evaluation (Process, Outcome, Cost)

• What data should you collect?▪ Data collection tips!

• How do you perform a self-assessment?▪ Analysis tips!

• How do you use the results?

3

WHY DO EVALUATION?

4

WHY DO EVALUATION?

• Better program outcomes:▪ Reduced recidivism

▪ Increased cost savings

• Program sustainability

• Replicate program success with new courts▪ Know what works and what doesn’t

5

WHY DO EVALUATION?

20% 20%

41%37%

Review Data(n=38), p<.05

Used Evaluation(n=34), p<.10

Possible Reductions in Recidivism

YesYesNo No

105% 85%

Carey, Finigan, & Pukstas (2008);Carey, Mackin, & Finigan (2012)

6

WHAT IS EVALUATION?

7

WHAT IS EVALUATION?

General definition: Systematic efforts to collect and use program information for multiple purposes, including:

▪ Program improvement

▪ Program accountability

▪ Program management

▪ Program development

8

WHAT IS USEFUL EVALUATION?

Evaluation should:▪ Inform and improve programs

▪ Be continuous

▪ Not only focus on what “worked,” but also what “didn’t work”

▪ Be collaborative (process & interpretation of results)

9

WHAT IS EVALUATION?

Three main areas of evaluation:▪ Process (program improvement)

▪ Outcome (impact)

▪ Cost (cost-benefit)

Process

OutcomeCost

10

PROCESS EVALUATION

Purpose: Examine program policies & procedures to:

▪ Determine how the program was implemented and if it was implemented as intended

▪ Learn whether and how well the program is following the intended model (best practices & standards)

11

WHY DO PROCESS EVALUATION?

Benefits:▪ Useful information about program functioning

▪ Allows an assessment of the reasons for successful or unsuccessful performance

▪ Provides information for replicating the program in another site

▪ Contribute to program improvement

▪ Increase effectiveness for participants

▪ Better Outcomes, Better Cost-Benefits

12

PROCESS METHODS: PROGRAM SELF REVIEW

• Study research on Best Practices/Standards▪ Team members take turns bringing articles▪ Watch short videos/webinars at policy meetings

(see NDCRC.org, DrugCourtOnline.org)

• Self review program practices:▪ Feedback from team members & participants

(what’s working for them, what’s challenging, suggestions for improvement) – interviews, surveys, focus groups

▪ Review policy manual – does manual adhere to best practices, and are you following them?

13

PROCESS METHODS: PEER REVIEW & EXTERNAL EVALUATION

▪ Program survey

▪ Interviews with program staff

▪ Document review

▪ Site visit/observations

▪ Focus groups with participants

▪ Review of program participant info

▪ Analyze results (summarize data)

▪ Interpret results –best practices

14

OUTCOME AND IMPACT EVALUATION

Purpose: Determine whether the program has improved participant outcomes during and after participation:

▪ Determine if services are delivered as intended

▪ Determine if participants are completing (graduating) the program as intended

▪ Determine the impact (rearrests, reincarceration) of the program

15

OUTCOMES VS. IMPACTS

Characteristics Outcome Evaluation Impact Evaluation

Time Period Within Program Outside/After Program

Performance Indicators

Services ReceivedGrad Rate

Time to CompletionFactors Leading to Grad

RearrestsReincarcerationSubsequent Tx

Social Services & Health Care Utilization

Comparison Group?

Not Needed Needed!

MethodsInternal Monitoring or

External EvaluationExternal Evaluation

16

OUTCOME METHODS: INTERNAL ACTIVITIES

• Track and enter data into database ▪ Paper files are useless for this

▪ A database that can run reports is best!

• Examine participant info, compare subgroups▪ Examples: grad rates, phases non-grads drop out

▪ Compare by gender, race, age ranges, drugs of choice, etc.

17

OUTCOME METHODS: EXTERNAL ACTIVITIES

• Obtain access to data (IRB, MOU)

• Collect data

• Prepare datasets (examine/understand, put in format for analyses, merge with other data)

• Identify and select comparison group

• Analyze data

• Interpret results

18

“CLEANING” DATA EXAMPLE: CHARGES

19

COST EVALUATION

Purpose: Determine how much the program costs to operate, and how the input costs relate to program outcomes

▪ How will taxpayers be impacted by decisions to add, expand, eliminate, or shrink these programs?

▪ What does it cost different partnering agencies to contribute resources?

▪ What are the benefits to all agencies?

20

TYPES OF COST EVALUATIONS

• Cost-effectiveness: the benefits are not measured in $$, but in terms of outcomes▪ Example: for every $1 million spent, there is a

10% reduction in recidivism

• Cost-benefit: the benefits are measured in $$▪ Example: for every $1 spent on treatment courts,

$2 is saved in outcomes

▪ Easier to compare across variety of programs

21

COMPARISON GROUPS

Purpose: to answer the question, “Is this program effective?”

▪ To learn whether the program is effective, there has to be a comparison, i.e., “effective compared to what?”

▪ A comparison group is the baseline

▪ It tells us what would have happened if there had been no program

22

COMPARISON GROUPS

Without a comparison group, you have to make assumptions:

“Participant would have been in prison, so the program saved time/cost of that sentence”

Not everyone gets the full sentence, or at all, and almost never serve the full sentence

23

COMPARISON GROUPS

Without a comparison group, you have to make assumptions:

“Participant’s children would have been in foster care or a shelter, so the program saved those foster care resources”

Sometimes other family can care for children, and also difficult to predict how long child will stay in foster care

24

COMPARISON GROUPS

Comparing program graduates to unsuccessful participants

▪ Can be okay if you want to know what is different between those who do and don’t graduate

▪ Example: if more men than women graduate, might indicate a need for gender-specific services

▪ Not appropriate for impact evaluation

25

GOOD COMPARISON GROUPS

• Should represent what would have happened if no program

• Ideally, individuals who are exactly like the individuals who participated, but did not receive the services▪ Example: individuals turned away due to capacity

▪ Example: individuals who fell through the cracks

▪ Example: historical comparison groups

26

WHAT DATA SHOULD YOU COLLECT?And Data Collection Tips!

27

WHAT DATA TO COLLECT?

Depends on your purpose:

Case Management

MonitoringEvaluation

28

CASE MANAGEMENT DATA

Data useful to every day work in the program:▪ Whose color came up for a drug test today?

▪ Who tested positive for drug use yesterday?

▪ Who missed a treatment session?

▪ Who complied with all program requirements for the past two weeks?

Examples: drug test dates, results, dates of treatment sessions

29

MONITORING DATA

Think about your key questions:▪ What is different about those who graduate and

those who don’t?

▪ When in our process do participants struggle (and drop out)?

▪ Are participants receiving the services indicated by their assessment?

Examples: program status, and dates of phase advancement, demographics, background, etc.

30

EVALUATION DATA

What are your program goals?▪ Reduce recidivism?

▪ Reduce substance use?

▪ Increase employment & education?

▪ Family reunification?

Track the progress (dates) of these goals for each participant

31

DATA COLLECTION TIPS

• A relational database is best (e.g., Access)▪ Example, Buffalo Drug Court Database

• Data collection is time-consuming, so focus on the areas that are most useful▪ Participant characteristics

▪ Program-level inputs/services

▪ Participant-level outputs

32

DATA COLLECTION AND PROGRAM FLOW

33

Treatment Court Outcomes

Treatment Supervision

Demos

Risk Factors

Needs

Short Term

Long Term

Participant Characteristics




Demos

Risk Factors

Needs

Short Term

Long Term



Demos & Background:Identifiers (names, IDs)Age (DOB), Gender, Race/ethnicityMilitary serviceNumber of kids, living/custody situationEducation levelEmployment statusDrugs of choice (primary/secondary)Date of eligible arrest/event

Green = Essential34




Demos

Risk Factors

Needs

Short Term

Long Term



Risk Factors:Risk assessment scores/levels & dateAge first arrested, first substance use# of prior arrests, typePrior violencePrior treatment, and completion statusFamily/friends Hx of crime/substance usePsychopathyAntisocial personality disorders

Green = Essential35




Demos

Risk Factors

Needs

Short Term

Long Term



Needs:Needs assessment results & datesSubstance use disorder diagnosesMental health diagnosesEducation/employment skills neededExecutive/frontal cognitive dysfunctionTraumatic brain injury (TBI)Hx of traumaOther: housing, ID documents, legal

Green = Essential36


Treatment Court Inputs/Services

Treatment:Substance use Tx status, by modalityDates of Tx sessionsAttendance at Tx sessionsGeneral Tx info/progressDates of other services received, typeDates of drug & alcohol tests, resultsRestorative justice servicesAftercare services (dates, types)

Green = Essential37


Supervision Court Inputs/Services

Supervision:Court supervision (start dates, status)Dates of phase advancementProbation supervision (status, level)Dates of probation field visits, case mgmtAttendance at court sessionsRewards and sanctions, esp jail staysDates of non-compliant behavior, type

Green = Essential38


Participant Level Outcomes

Short term (in-program):Housing, education, employment changesProgram retention, graduationReason for discharge (if applicable)Abstinence or reduction in substance useDrug-free babiesNew arrests, violations (and dates)School related data, out-of-home placements, & reunifications (juv & family)

Green = Essential39


Participant Level Outcomes

Long term (post-program):New arrests, convictions (and dates)New incarceration (dates)New supervision episodes (dates)Subsequent Tx episodesSocial and health services infoSchool related data, out-of-home placements, & reunifications (juv & family)

Green = Essential40


• Pay special attention to accuracy of identifiers

• Use drop-down lists or lookup tables to make data entry faster and consistent▪ Don’t spend time typing out the risk level each

time, select from a list!

• Use data validation (restrictions) ▪ Restricting the format on a date field can help

eliminate data entry errors

41


If you must use Excel:▪ Limit one data element per column

▪ Put participant characteristics and outcomes on one sheet

▪ Put treatment, services received, drug tests, incentives & sanctions, and other items that come in multiples on another sheet

▪ Use drop-downs (under data validation) to link the list of names on one sheet to the others

42

EXCEL DROP-DOWN EXAMPLE

One sheet with participants:

Another with drug tests:

43

HOW DO YOU PERFORM A SELF

ASSESSMENT?

44

PERFORMING A SELF ASSESSMENT

Set time aside at least yearly to talk about program progress (e.g., policy meetings):

• Come prepared to talk openly about successes and challenges over the year

• Review program materials▪ Contract, handbook, eligibility requirements

• Review recent guidelines, literature, webinars in treatment court best practices

45


• Review your program data:▪ What are your program demographics?

• Do they match the general supervision population of your jurisdiction?

▪ What is your graduation rate?*• By year, subgroups (gender, race, other background)

▪ Does the population match your stated eligibility criteria (arrests, risks, needs)? If not, why?

*Note, don’t include the current year in your calculations—unsuccessfully discharged will be overrepresented!!

46


• Review your program data (cont’d):▪ How long (on average) does it take graduates to

complete the program?

▪ In what phase (or typical length of service) do unsuccessfully discharged participants drop-out?

▪ Are there any other differences between grads and non-grads (employment, housing, education status)?

▪ How long (on average) does it take from arrest/event to program entry?

47

ANALYSIS TIPS!

• If you don’t have a database that can automatically run these types of reports, ask!

• If you’re using Excel:▪ Use the formula bar to calculate ages, referral

time, and length in program

48

Calculate days: =[End Date]-[Start Date]

ANALYSIS TIPS!

• If you don’t have a database that can automatically run these types of reports, ask!

• If you’re using Excel:▪ Use the formula bar to calculate ages, referral

time, and length in program

49

Calculate years: =INT(([End Date]-[Start Date])/365)

OTHER USEFUL EXCEL FEATURES

Filters:

50

OTHER USEFUL EXCEL FEATURES

Pivot Tables:

51

HOW DO YOU USE THE RESULTS?

52

USING THE RESULTS

• Internal planning & improvement▪ Make a list of areas for improvement

▪ Prioritize your list based on importance, resources needed, and feasibility

▪ Create a staggered action plan (focus on 1-2 things at a time)

▪ Assign someone to each area to monitor progress

▪ Report back to the team regularly

53

USING THE RESULTS

• External dissemination of results▪ Fact sheets (1-2 pages) highlighting positive

outcomes (reductions in rearrests, cost savings)

▪ Include simple graphics or tables

▪ Add the “human touch” by including anecdotes from or photos of graduates

▪ Invite community stakeholders, agencies, and representatives to attend graduation ceremonies, meet participants, or serve on your advisory board

54

USING THE RESULTS

Interested external stakeholders:▪ Media: disseminate results to your community

▪ Legislature (key committees): Senate Health & Welfare, House & Senate Judiciary Committees, Senate & House Appropriations, House Institutions & Correction

▪ State Policy Executives: Secretary of the Agency of Human Services, Deputy Commissioner of Health, Commissioner of Mental Health, Commissioner of Corrections

55

EXAMPLES: GRAPHS & FACT SHEETS

56

EXAMPLES: COSTS PER AGENCY

57

EXAMPLES: SHOWING THE HUMAN SIDE

58

Before Drug Court After Drug Court

RESOURCES

Buffalo Drug Court database:

https://jpo.wrlc.org/bitstream/handle/11204/1080/Buffalo%20MIS%20Announcement%20(New%20York).pdf?sequence=4

Using drop-down lists in Excel:

https://support.office.com/en-us/article/Create-a-drop-down-list-7693307a-59ef-400a-b769-c5402dce407b

Using pivot tables in Excel:

https://support.office.com/en-us/article/Create-a-PivotTable-to-analyze-worksheet-data-a9a84538-bfe9-40a9-a8e9-f99134456576

59

CONTACT INFORMATION

Charlene Zil, MPA

Researcher @ NPC Research

[email protected]

60

evaluation 101 for non-researcher practitioners...charlene zil, npc research nadcp conference july...

Documents