overview of odrc’s oras quality assurance program
TRANSCRIPT
Accurate Assessments
Overview of ODRC’s ORAS Quality Assurance Program
What is Quality Assurance?
ORAS File Review
Development of the QA Program
Results of the QA Program
Recommendations
Overview
Proactive Improve Confidence
Credibility
A planned and systematic pattern of actions necessary to provide adequate confidence that the product optimally fulfills customer expectations.
What is Quality Assurance
Create a culture of quality Good fidelity = replication of results Mission “reduce recidivism”
Why quality assurance?
FAQ Flow charts Internal audit standards Quality Assurance Reports Case file reviews Quality Assurance Program
Supporting ORAS Quality
Keep these 12 things clean for good assessments
Dirty Dozen of ORAS QA
Duplicates Sources used Verifying collateral information Notes
Keeping the Dirty Dozen Clean
Building a QA Program Pilot site evaluations Benchmarking Literature Review University of Cincinnati
Goal = accurate assessments What contributes to the goal?
◦ Fidelity to the assessment process◦ Good interview skills
How will the QA program be delivered?◦ Create measures/tools◦ Timelines
Implement the QA program Analyze and share results Evaluate program and make adjustments.
Building Blocks of a QA Program
QA Coordinator Random selection of staff Scheduling Communication Work flow Selection of Evaluators
Program Delivery
ORAS Direct Observation Tool
Performance Measurement Guide
Direct Observation Tool and Measures
Pre-brief meeting Direct Observation of the interview Individual Scoring of the tool Debrief and Coaching Results to the QA Coordinator
Implement
Goals Program process Results End users All levels of management
Communication
Subject matter experts/Certified ORAS Trainers
IPC skills Recommendation from their immediate
supervisor. Completion of evaluator training.
Selection of Evaluators
Direct Observation Tool Performance Measurement Guide Double coding General process Common rating errors Providing good feedback that is balanced Giving and receiving criticism
Evaluator Training
Survey Evaluation ◦ Evaluators (19)◦ Assessors (116)◦ Do survey results demonstrate gaps? Perceived
value verses real value.
Share results
Evaluation of the QA Program
Value of the pre-brief meeting◦ 65% of the assessors and 59% of the evaluators rated it
successful or very successful. Concentrated areas of coaching
◦ Assessors and evaluators rated using the interview guide and using collateral information highest.
Did assessors learn anything new?◦ 61% of the assessors and 90% of the evaluators
responded yes. Ability of the evaluators to add value
◦ 82% of the assessors rated the evaluators with the ability or strong ability to add value.
How important are accurate ORAS assessments?◦ 65% of the assessors rated it as important or very
important.
Survey Results
What are the barriers to completing accurate ORAS assessments?◦ The length of time it tales to complete the
interview and input the information into the portal. Also the program runs slow.
◦ Clients not being fully truthful.◦ Not having an investigation in the file.◦ Time consuming when you have a lot of other
things to do.
Survey Comments from End Users
What was most valuable about the ORAS QA process?◦ Getting reassurance that I was doing some things
right. ◦ The tool has potential if done accurately.◦ Helped understand my struggles were being felt
by others also being evaluated. ◦ One on one discussion and feedback.◦ Don’t think we need this, to concerned about QA
and not the time consuming ORAS tool.
Survey Comments from End Users
How would you improve the ORAS QA process?◦ It was fairly upfront and open. The evaluator did
a good job with me. Maybe assess everyone at some point.
◦ I think the ORAS QA process can be very effective, if administered by someone that has actually completed an ORAS assessment of some kind.
◦ Have the assessor sit in on more than one assessment.
◦ Figure out a way across the state to bring all officers to the same page period. Good Luck!
Survey Comments from End Users
QA Results
Domains 6 and 7 Interview Skills
Trends
Survey
Interview Skills Reliability
Direct Observation Tool
Double Coding Survey
Resources Subject Matter Experts Have a QA Coordinator Explain QA process to the offender Scoring Guide Have follow up plan ready to implement Staff support – supervisors need to be more
knowledgeable. Sustainability
Lessons learned
Interview training Booster trainings focused on weak areas
and/or changes RIB rules card Supervisor staffing requirements Individual Improvement Plan Consider assessor models Continue QA efforts statewide with all user
groups
Next Steps
Goals Concerns/barriers for staff Resources Process flow Direct Observation Training Communication Implementation Evaluation Follow up – use results to impact positive
change
How to create QA Program
Good Quality Assurance
Proactive Improve Confidence
Credibility
