virginia department for aging and rehabilitative services 1 barbara j. burkett, ph.d., m.s.p.h....
TRANSCRIPT
Virginia Department for Aging and Rehabilitative Services
1
Barbara J. Burkett, Ph.D., M.S.P.H.Division of Policy and Planning 804-726-1911
Report Requested by the Commissioner and Field Services Director on the following: Utilization of vocational evaluation services Program’s effectiveness Outcomes Counselors’ satisfaction Best practices Other state’s use of vocational evaluation
2
3
Review of Internal DARS Data Number of Clients Served Timing of Vocational Evaluations Within The Rehabilitation Process Demographics of Clients Receiving Evaluations Outcomes for Clients Receiving Vocational Evaluations Compared to Those
Who Do Not Receive Vocational Evaluations Cost Metrics of Vocational Evaluations
DARS Case Audit Reviews Literature Search
Roles and Functions of Vocational Evaluators Review of Various State Vocational Rehabilitation Programs Use of Vocational
Evaluators Population Specific Instruments/Best Practices
Survey of DARS Counselors Regarding Current Program
Review of Internal DARS Data Data was extracted from AWARE, the DARS case
management system, and imported in to SPSS for statistical analysis.
Study Years: Federal Fiscal Years 2011, 2012 and 2013▪ Vocational Evaluation Cases Identified▪ 2011 case notes only▪ 2012 case notes and vendor file codes▪ 2013 case notes and vendor file codes
4
Case Notes String search for every iteration of “vocational evaluation” (Example: vocational evaluation, voc. eval., v.e., evaluation, evaluation, eval,….inclusive of v* and e* with and without
punctuation and capitalization) Verified string search results by opening each case in AWARE.
AND Vendor File
Career Services Comprehensive Evaluation Exploratory Evaluation Referral/Staffing/Consultation Report Writing
5
17 Vocational Evaluation Positions
15 Currently Filled 2 Vacancies (Eastern, Southwest)
6
Six percent of DARS clients received a vocational evaluation during the study period.
7
FFY
# Clients (Application –Post
Employment Status)
# Vocational Evaluations
% of All Clients In Application
through Post Employment Status
Receiving A Vocational Evaluation
2011 30,795 1,843 6.0%
2012 28,877 1,527 5.3%
2013 28,112 1,944 6.9%
There was no regional variation in requests for vocational evaluation.
There was great variation in the use of vocational evaluation by counselors.
The range in the number of vocational evaluations requested by DARS counselors during the three year time period was 1-127.
Average number of vocational evaluations requested was 8.8 per counselor per year (s.d of 8.7).
The mode was 6 requests per counselor per year.
8
9
Overall for the 3 year time period, transition students were more likely than non transition students to receive a vocational evaluation.
10
More males than females received vocational evaluations.
11
43% cognitive impairment 28.4% psychosocial impairment 7.9% other mental impairment
12
13
The rehabilitation rate was lower for clients who received a vocational evaluation over the 3 year period.
14
Average hourly earnings were lower for clients receiving a
vocational evaluation.
15
Clients who received a vocational evaluation were less likely to obtain a job at closure that was the same as indicated on the IPE.
Clients who received vocational evaluations were less likely to be referred for supported employment services.
16
17
The VR cost of the clients’ who received a vocational evaluation were lower than for those clients who did not receive a vocational evaluation.
Conducted by Scott Fraley and Tim OliveReport Written by Liz Smith
Selection Criteria for Cases Cases closed in FFY2013 that had a vocational
evaluation as a planned service Stratified random sample by region and caseload
type Total sample n=40
18
Five (12%) of the forty cases had a vocational evaluation scheduled, but never conducted. Two of these were caused by the consumer failing to appear for the vocational evaluation. The other three reasons were not evident from the case file.
35 Reports Twenty (57%) had no job recommendation in the vocational evaluation report.
Eight of the 20 cases with no recommendation (23%) were transition cases.
Of the fifteen cases that had a job recommendation in the vocational evaluation report: For twelve (80%) of the cases: the vocational evaluation recommendation, the
vocational goal in the IPE, and the job at case closure matched.
19
If it is the expectation that a job recommendation occur when a vocational evaluation is conducted, then that is not occurring on a consistent basis.
When a job recommendation is made in the vocational evaluation, there is a high level of consistency between that recommendation, the IPE goal, and the consumer’s job at case closure.
20
Response Rate For Survey: 85.7% n/N = 12/14
Number of Vocational Evaluations FFY13 x = 96.5 range = 76-123
Number of Vocational Evaluations April 2014 x = 10.6 Range 6-20 No show rate is a problem with 2 evaluators reporting a no show
rate of 46.4%
21
22
Referral Source DARS evaluators 1 report of parent/guardian 1 client request 1 supported employment vendor
Time From Referral to Evaluation X =41.7 days Range 20 – 65
Time to Complete Written Report X= 6.1 days Range = 2-16
Template Use 50% use now 25% would like to use 25% would not find it helpful
Job Recommendations Made 82% make job recommendation 18% did not (=2)
23
#1: Incomplete information on client at time of evaluation
#2: Client willingness to participate #3: Scheduling problems #4: Poor referrals #5: Client ability to participate
24
Discussion of vocational evaluation reports with DARS counselors 55% always discuss (n=6) 46% discuss sometimes (n=5) 1 person did not answer this question
25
Instruments Work Behavior Inventory used by 2 evaluators Threshold Monthly Work Evaluation: 100% not familiar with
instrument Vocational Cognitive Rating Scale: 100% not familiar with instrument Work Capacity Evaluation used by 2 evaluators Maryland Assessment of Social Competence: 100% not familiar Change Assessment for Vocational Counseling (URI): 100% not familiar IPS Fidelity Scale: 1 evaluator familiar with instrument Supported Employment Scale: 1 evaluator familiar with this instrument
26
Population Based Instruments
27
Use Do not use Would Like to Use
Ethnic Based 7 1 2
Minority Based
4 3 4
Gender Based 5 2 4
Hearing Impaired
5 1 5
Autism Based 8 0 3
Mental Illness 5 2 4
College Bound 9 0 2
Carolyn Graham, Ph.D. (VCU) contracted for literature review
Barbara Burkett, Ph.D. (DARS)
Best Practices: Guide to Vocational Assessment and Evaluation-4th Edition
Author: Paul Power “The State of Vocational Evaluators: A National Study” Sligar and Betters Journal of Rehabilitation Vol. 74, no. 4
28
Sligar and Betters interviewed 63 agencies
o 97% (n= 60) of states purchased vocational evaluation services from evaluators in private practice, psychologists, and community rehabilitation programs.
o Average number of vocational evaluators by agency was 15.1 with a range of 1-72.
o 65% of the respondents did not require a certificationo 81% of the programs had guidelines for the vocational evaluation and the
final report.
29
The vocational evaluator positions currently unfilled in the Central Region will be filled.
The vocational evaluator position in the Southwest region will not be filled.
A new position has been created. The position will supervise the vocational evaluators, the business development managers and the self employment enterprise staff. Duties of this position will include:▪ Updating the DARS Best Practices Manual▪ Developing improved labor market data for use by DARS▪ Developing requirements for report writing▪ Procuring new instruments identified in the literature review and
conducting training
30
31
Clearly communicate the purpose of the program evaluation to staff involved. Describe the role of the program evaluator. Provide a program evaluation plan as soon as possible to the stake holders. Include the stakeholders in the planning meetings if possible Present the findings to those involved and allow for input before final
evaluation report. Don’t take comments personally. This is the nature of the job.
32
www.decisionswithdata.com Blog
33
Null Hypothesis (Ho) and Alternative Hypothesis (Ha) Stated
Population Distribution Determined For Variables Probability plot
▪ Normal Distribution Non-normal Distribution or small sample size
Parametric or Non-Parametric Test Chosen Test of Difference Between Proportions▪ Z-test: parametric▪ Chi-Square Test: non-parametric
Test of Difference Between Means▪ T-test: parametric▪ Kolomogorov-Smirnov Test: non-parametric
See Appendices of Program Evaluation for statistical formulas
34