![Page 1: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/1.jpg)
Evaluation 101 for Non-Researcher Practitioners
Charlene Zil, NPC Research
NADCP Conference
July 2017
![Page 2: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/2.jpg)
EVALUATION 101 FOR NON-RESEARCHERSWhat to Collect and How to Use it for Self-Review
2
![Page 3: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/3.jpg)
OVERVIEW
• Why do evaluation?
• What is evaluation (Process, Outcome, Cost)
• What data should you collect?▪ Data collection tips!
• How do you perform a self-assessment?▪ Analysis tips!
• How do you use the results?
3
![Page 4: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/4.jpg)
WHY DO EVALUATION?
4
![Page 5: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/5.jpg)
WHY DO EVALUATION?
• Better program outcomes:▪ Reduced recidivism
▪ Increased cost savings
• Program sustainability
• Replicate program success with new courts▪ Know what works and what doesn’t
5
![Page 6: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/6.jpg)
WHY DO EVALUATION?
20% 20%
41%37%
Review Data(n=38), p<.05
Used Evaluation(n=34), p<.10
Possible Reductions in Recidivism
YesYesNo No
105% 85%
Carey, Finigan, & Pukstas (2008);Carey, Mackin, & Finigan (2012)
6
![Page 7: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/7.jpg)
WHAT IS EVALUATION?
7
![Page 8: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/8.jpg)
WHAT IS EVALUATION?
General definition: Systematic efforts to collect and use program information for multiple purposes, including:
▪ Program improvement
▪ Program accountability
▪ Program management
▪ Program development
8
![Page 9: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/9.jpg)
WHAT IS USEFUL EVALUATION?
Evaluation should:▪ Inform and improve programs
▪ Be continuous
▪ Not only focus on what “worked,” but also what “didn’t work”
▪ Be collaborative (process & interpretation of results)
9
![Page 10: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/10.jpg)
WHAT IS EVALUATION?
Three main areas of evaluation:▪ Process (program improvement)
▪ Outcome (impact)
▪ Cost (cost-benefit)
Process
OutcomeCost
10
![Page 11: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/11.jpg)
PROCESS EVALUATION
Purpose: Examine program policies & procedures to:
▪ Determine how the program was implemented and if it was implemented as intended
▪ Learn whether and how well the program is following the intended model (best practices & standards)
11
![Page 12: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/12.jpg)
WHY DO PROCESS EVALUATION?
Benefits:▪ Useful information about program functioning
▪ Allows an assessment of the reasons for successful or unsuccessful performance
▪ Provides information for replicating the program in another site
▪ Contribute to program improvement
▪ Increase effectiveness for participants
▪ Better Outcomes, Better Cost-Benefits
12
![Page 13: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/13.jpg)
PROCESS METHODS: PROGRAM SELF REVIEW
• Study research on Best Practices/Standards▪ Team members take turns bringing articles▪ Watch short videos/webinars at policy meetings
(see NDCRC.org, DrugCourtOnline.org)
• Self review program practices:▪ Feedback from team members & participants
(what’s working for them, what’s challenging, suggestions for improvement) – interviews, surveys, focus groups
▪ Review policy manual – does manual adhere to best practices, and are you following them?
13
![Page 14: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/14.jpg)
PROCESS METHODS: PEER REVIEW & EXTERNAL EVALUATION
▪ Program survey
▪ Interviews with program staff
▪ Document review
▪ Site visit/observations
▪ Focus groups with participants
▪ Review of program participant info
▪ Analyze results (summarize data)
▪ Interpret results –best practices
14
![Page 15: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/15.jpg)
OUTCOME AND IMPACT EVALUATION
Purpose: Determine whether the program has improved participant outcomes during and after participation:
▪ Determine if services are delivered as intended
▪ Determine if participants are completing (graduating) the program as intended
▪ Determine the impact (rearrests, reincarceration) of the program
15
![Page 16: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/16.jpg)
OUTCOMES VS. IMPACTS
Characteristics Outcome Evaluation Impact Evaluation
Time Period Within Program Outside/After Program
Performance Indicators
Services ReceivedGrad Rate
Time to CompletionFactors Leading to Grad
RearrestsReincarcerationSubsequent Tx
Social Services & Health Care Utilization
Comparison Group?
Not Needed Needed!
MethodsInternal Monitoring or
External EvaluationExternal Evaluation
16
![Page 17: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/17.jpg)
OUTCOME METHODS: INTERNAL ACTIVITIES
• Track and enter data into database ▪ Paper files are useless for this
▪ A database that can run reports is best!
• Examine participant info, compare subgroups▪ Examples: grad rates, phases non-grads drop out
▪ Compare by gender, race, age ranges, drugs of choice, etc.
17
![Page 18: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/18.jpg)
OUTCOME METHODS: EXTERNAL ACTIVITIES
• Obtain access to data (IRB, MOU)
• Collect data
• Prepare datasets (examine/understand, put in format for analyses, merge with other data)
• Identify and select comparison group
• Analyze data
• Interpret results
18
![Page 19: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/19.jpg)
“CLEANING” DATA EXAMPLE: CHARGES
19
![Page 20: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/20.jpg)
COST EVALUATION
Purpose: Determine how much the program costs to operate, and how the input costs relate to program outcomes
▪ How will taxpayers be impacted by decisions to add, expand, eliminate, or shrink these programs?
▪ What does it cost different partnering agencies to contribute resources?
▪ What are the benefits to all agencies?
20
![Page 21: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/21.jpg)
TYPES OF COST EVALUATIONS
• Cost-effectiveness: the benefits are not measured in $$, but in terms of outcomes▪ Example: for every $1 million spent, there is a
10% reduction in recidivism
• Cost-benefit: the benefits are measured in $$▪ Example: for every $1 spent on treatment courts,
$2 is saved in outcomes
▪ Easier to compare across variety of programs
21
![Page 22: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/22.jpg)
COMPARISON GROUPS
Purpose: to answer the question, “Is this program effective?”
▪ To learn whether the program is effective, there has to be a comparison, i.e., “effective compared to what?”
▪ A comparison group is the baseline
▪ It tells us what would have happened if there had been no program
22
![Page 23: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/23.jpg)
COMPARISON GROUPS
Without a comparison group, you have to make assumptions:
“Participant would have been in prison, so the program saved time/cost of that sentence”
Not everyone gets the full sentence, or at all, and almost never serve the full sentence
23
![Page 24: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/24.jpg)
COMPARISON GROUPS
Without a comparison group, you have to make assumptions:
“Participant’s children would have been in foster care or a shelter, so the program saved those foster care resources”
Sometimes other family can care for children, and also difficult to predict how long child will stay in foster care
24
![Page 25: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/25.jpg)
COMPARISON GROUPS
Comparing program graduates to unsuccessful participants
▪ Can be okay if you want to know what is different between those who do and don’t graduate
▪ Example: if more men than women graduate, might indicate a need for gender-specific services
▪ Not appropriate for impact evaluation
25
![Page 26: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/26.jpg)
GOOD COMPARISON GROUPS
• Should represent what would have happened if no program
• Ideally, individuals who are exactly like the individuals who participated, but did not receive the services▪ Example: individuals turned away due to capacity
▪ Example: individuals who fell through the cracks
▪ Example: historical comparison groups
26
![Page 27: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/27.jpg)
WHAT DATA SHOULD YOU COLLECT?And Data Collection Tips!
27
![Page 28: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/28.jpg)
WHAT DATA TO COLLECT?
Depends on your purpose:
Case Management
MonitoringEvaluation
28
![Page 29: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/29.jpg)
CASE MANAGEMENT DATA
Data useful to every day work in the program:▪ Whose color came up for a drug test today?
▪ Who tested positive for drug use yesterday?
▪ Who missed a treatment session?
▪ Who complied with all program requirements for the past two weeks?
Examples: drug test dates, results, dates of treatment sessions
29
![Page 30: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/30.jpg)
MONITORING DATA
Think about your key questions:▪ What is different about those who graduate and
those who don’t?
▪ When in our process do participants struggle (and drop out)?
▪ Are participants receiving the services indicated by their assessment?
Examples: program status, and dates of phase advancement, demographics, background, etc.
30
![Page 31: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/31.jpg)
EVALUATION DATA
What are your program goals?▪ Reduce recidivism?
▪ Reduce substance use?
▪ Increase employment & education?
▪ Family reunification?
Track the progress (dates) of these goals for each participant
31
![Page 32: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/32.jpg)
DATA COLLECTION TIPS
• A relational database is best (e.g., Access)▪ Example, Buffalo Drug Court Database
• Data collection is time-consuming, so focus on the areas that are most useful▪ Participant characteristics
▪ Program-level inputs/services
▪ Participant-level outputs
32
![Page 33: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/33.jpg)
DATA COLLECTION AND PROGRAM FLOW
33
Treatment Court Outcomes
Treatment Supervision
Demos
Risk Factors
Needs
Short Term
Long Term
Participant Characteristics
![Page 34: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/34.jpg)
DATA COLLECTION AND PROGRAM FLOW
Treatment Court Outcomes
Treatment Supervision
Demos
Risk Factors
Needs
Short Term
Long Term
Participant Characteristics
Participant Characteristics
Demos & Background:Identifiers (names, IDs)Age (DOB), Gender, Race/ethnicityMilitary serviceNumber of kids, living/custody situationEducation levelEmployment statusDrugs of choice (primary/secondary)Date of eligible arrest/event
Green = Essential34
![Page 35: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/35.jpg)
DATA COLLECTION AND PROGRAM FLOW
Treatment Court Outcomes
Treatment Supervision
Demos
Risk Factors
Needs
Short Term
Long Term
Participant Characteristics
Participant Characteristics
Risk Factors:Risk assessment scores/levels & dateAge first arrested, first substance use# of prior arrests, typePrior violencePrior treatment, and completion statusFamily/friends Hx of crime/substance usePsychopathyAntisocial personality disorders
Green = Essential35
![Page 36: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/36.jpg)
DATA COLLECTION AND PROGRAM FLOW
Treatment Court Outcomes
Treatment Supervision
Demos
Risk Factors
Needs
Short Term
Long Term
Participant Characteristics
Participant Characteristics
Needs:Needs assessment results & datesSubstance use disorder diagnosesMental health diagnosesEducation/employment skills neededExecutive/frontal cognitive dysfunctionTraumatic brain injury (TBI)Hx of traumaOther: housing, ID documents, legal
Green = Essential36
![Page 37: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/37.jpg)
DATA COLLECTION AND PROGRAM FLOW
Treatment Court Inputs/Services
Treatment:Substance use Tx status, by modalityDates of Tx sessionsAttendance at Tx sessionsGeneral Tx info/progressDates of other services received, typeDates of drug & alcohol tests, resultsRestorative justice servicesAftercare services (dates, types)
Green = Essential37
![Page 38: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/38.jpg)
DATA COLLECTION AND PROGRAM FLOW
Supervision Court Inputs/Services
Supervision:Court supervision (start dates, status)Dates of phase advancementProbation supervision (status, level)Dates of probation field visits, case mgmtAttendance at court sessionsRewards and sanctions, esp jail staysDates of non-compliant behavior, type
Green = Essential38
![Page 39: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/39.jpg)
DATA COLLECTION AND PROGRAM FLOW
Participant Level Outcomes
Short term (in-program):Housing, education, employment changesProgram retention, graduationReason for discharge (if applicable)Abstinence or reduction in substance useDrug-free babiesNew arrests, violations (and dates)School related data, out-of-home placements, & reunifications (juv & family)
Green = Essential39
![Page 40: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/40.jpg)
DATA COLLECTION AND PROGRAM FLOW
Participant Level Outcomes
Long term (post-program):New arrests, convictions (and dates)New incarceration (dates)New supervision episodes (dates)Subsequent Tx episodesSocial and health services infoSchool related data, out-of-home placements, & reunifications (juv & family)
Green = Essential40
![Page 41: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/41.jpg)
DATA COLLECTION TIPS
• Pay special attention to accuracy of identifiers
• Use drop-down lists or lookup tables to make data entry faster and consistent▪ Don’t spend time typing out the risk level each
time, select from a list!
• Use data validation (restrictions) ▪ Restricting the format on a date field can help
eliminate data entry errors
41
![Page 42: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/42.jpg)
DATA COLLECTION TIPS
If you must use Excel:▪ Limit one data element per column
▪ Put participant characteristics and outcomes on one sheet
▪ Put treatment, services received, drug tests, incentives & sanctions, and other items that come in multiples on another sheet
▪ Use drop-downs (under data validation) to link the list of names on one sheet to the others
42
![Page 43: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/43.jpg)
EXCEL DROP-DOWN EXAMPLE
One sheet with participants:
Another with drug tests:
43
![Page 44: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/44.jpg)
HOW DO YOU PERFORM A SELF
ASSESSMENT?
44
![Page 45: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/45.jpg)
PERFORMING A SELF ASSESSMENT
Set time aside at least yearly to talk about program progress (e.g., policy meetings):
• Come prepared to talk openly about successes and challenges over the year
• Review program materials▪ Contract, handbook, eligibility requirements
• Review recent guidelines, literature, webinars in treatment court best practices
45
![Page 46: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/46.jpg)
PERFORMING A SELF ASSESSMENT
• Review your program data:▪ What are your program demographics?
• Do they match the general supervision population of your jurisdiction?
▪ What is your graduation rate?*• By year, subgroups (gender, race, other background)
▪ Does the population match your stated eligibility criteria (arrests, risks, needs)? If not, why?
*Note, don’t include the current year in your calculations—unsuccessfully discharged will be overrepresented!!
46
![Page 47: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/47.jpg)
PERFORMING A SELF ASSESSMENT
• Review your program data (cont’d):▪ How long (on average) does it take graduates to
complete the program?
▪ In what phase (or typical length of service) do unsuccessfully discharged participants drop-out?
▪ Are there any other differences between grads and non-grads (employment, housing, education status)?
▪ How long (on average) does it take from arrest/event to program entry?
47
![Page 48: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/48.jpg)
ANALYSIS TIPS!
• If you don’t have a database that can automatically run these types of reports, ask!
• If you’re using Excel:▪ Use the formula bar to calculate ages, referral
time, and length in program
48
Calculate days: =[End Date]-[Start Date]
![Page 49: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/49.jpg)
ANALYSIS TIPS!
• If you don’t have a database that can automatically run these types of reports, ask!
• If you’re using Excel:▪ Use the formula bar to calculate ages, referral
time, and length in program
49
Calculate years: =INT(([End Date]-[Start Date])/365)
![Page 50: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/50.jpg)
OTHER USEFUL EXCEL FEATURES
Filters:
50
![Page 51: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/51.jpg)
OTHER USEFUL EXCEL FEATURES
Pivot Tables:
51
![Page 52: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/52.jpg)
HOW DO YOU USE THE RESULTS?
52
![Page 53: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/53.jpg)
USING THE RESULTS
• Internal planning & improvement▪ Make a list of areas for improvement
▪ Prioritize your list based on importance, resources needed, and feasibility
▪ Create a staggered action plan (focus on 1-2 things at a time)
▪ Assign someone to each area to monitor progress
▪ Report back to the team regularly
53
![Page 54: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/54.jpg)
USING THE RESULTS
• External dissemination of results▪ Fact sheets (1-2 pages) highlighting positive
outcomes (reductions in rearrests, cost savings)
▪ Include simple graphics or tables
▪ Add the “human touch” by including anecdotes from or photos of graduates
▪ Invite community stakeholders, agencies, and representatives to attend graduation ceremonies, meet participants, or serve on your advisory board
54
![Page 55: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/55.jpg)
USING THE RESULTS
Interested external stakeholders:▪ Media: disseminate results to your community
▪ Legislature (key committees): Senate Health & Welfare, House & Senate Judiciary Committees, Senate & House Appropriations, House Institutions & Correction
▪ State Policy Executives: Secretary of the Agency of Human Services, Deputy Commissioner of Health, Commissioner of Mental Health, Commissioner of Corrections
55
![Page 56: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/56.jpg)
EXAMPLES: GRAPHS & FACT SHEETS
56
![Page 57: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/57.jpg)
EXAMPLES: COSTS PER AGENCY
57
![Page 58: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/58.jpg)
EXAMPLES: SHOWING THE HUMAN SIDE
58
Before Drug Court After Drug Court
![Page 59: Evaluation 101 for Non-Researcher Practitioners...Charlene Zil, NPC Research NADCP Conference July 2017 EVALUATION 101 FOR NON-RESEARCHERS What to Collect and How to Use it for Self-Review](https://reader033.vdocuments.us/reader033/viewer/2022053119/60a0e7b92b2d7551a4352b74/html5/thumbnails/59.jpg)
RESOURCES
Buffalo Drug Court database:
https://jpo.wrlc.org/bitstream/handle/11204/1080/Buffalo%20MIS%20Announcement%20(New%20York).pdf?sequence=4
Using drop-down lists in Excel:
https://support.office.com/en-us/article/Create-a-drop-down-list-7693307a-59ef-400a-b769-c5402dce407b
Using pivot tables in Excel:
https://support.office.com/en-us/article/Create-a-PivotTable-to-analyze-worksheet-data-a9a84538-bfe9-40a9-a8e9-f99134456576
59