1

SARAH HUNTERRAND CORPORATION

LAURA STEIGHNERAMERICAN INSTITUTES FOR RESEARCH

NOVEMBER 16, 2009

National Evaluation of the Demonstration to Improve the

Direct Service Workforce

2

Briefing Outline

Overview of Demonstrations and EvaluationImplementation Analysis

Methods Findings

General Lessons Learned

3

Project Team Structure

Principal InvestigatorJohn Engberg, Ph.D.

RAND

Principal InvestigatorJohn Engberg, Ph.D.

RAND

Project AdvisorDonna Farley,

Ph.D.RAND

Project AdvisorDonna Farley,

Ph.D.RAND

Team LeadersTeam Leaders

Outcomes EvaluationJohn Engberg, Ph.D.

RANDNick Castle, Ph.D.Univ of Pittsburgh

Outcomes EvaluationJohn Engberg, Ph.D.

RANDNick Castle, Ph.D.Univ of Pittsburgh

Implementation EvaluationSarah Hunter, Ph.D.

RANDLaura Steighner, Ph.D.

AIR

Implementation EvaluationSarah Hunter, Ph.D.

RANDLaura Steighner, Ph.D.

AIR

4

DEMONSTRATIONSEVALUATION PLAN

MULTIPLE CASE STUDY APPROACH

Overview of Study

5

Demonstrations

Demonstrations were intended to improve recruitment and retention of direct service workers

5 grants awarded in 2003, 5 grants awarded in 2004

Grantees implemented various methods for accomplishing goals (details on next slide)

Web-based reporting overseen by LewinTechnical assistance provided by Lewin and PHI Each site had a local evaluator

6

National Evaluation

Purpose To provide a summative evaluation that would allow a

comparison across the sitesData Collection

Site visits Surveys

Evaluation Process (implementation) Outputs (participation, initiative cost) Outcomes (job satisfaction, consumer satisfaction,

turnover, retention, vacancies, turnover cost)

7

Multiple Case Study Approach

Understand implementation context across different grantees Describe initiatives, implementation, and their perceived

outcomes Identify patterns across grantees Explore costs and sustainability

Collected from multiple sources to increase validity Reviewed extant documents (grant proposals, Lewin & PHI

reports, quarterly reports) Developed logic models in collaboration with grantees Conducted telephone interviews and site visits and attended

NFI conference

8

CROSSWALK OF INITIATIVESINITIATIVE

PurposeImplementation

Perceived Outcomes

Implementation Analysis

9

Crosswalk of Initiatives

Initiatives AR DE IN KY ME NC NM VA VOA WA

Health Care Coverage

DSW Training

Supervisor and Consumer Training

Realistic Job Preview

Peer Mentorship

Merit-based or Longevity Recognition

Worker Registry

Marketing Campaign

Targeted Recruitment Strategy

10

Health Care

Purpose To provide coverage or improve accessibility to health care

Implementation Demonstration is a difficult vehicle to study coverage

Variation across Grantees Affordability Sustainability Skepticism over Free Good

Full-time status is a barrier to getting coverage Health care coverage is complicated

Perceived Outcomes Need for Coverage Past Unmet Need Improved Climate

11

DSW Training

Purpose To improve competence and job satisfaction among DSWs

Implementation Training Content Professional Development Training Mode Scheduling Incentives for Participation and Completion Participation by Supervisors/Managers Voluntary versus Mandatory Participation

Perceived Outcomes Training Effectiveness Community Building among DSWs Improved DSW Self-Care

12

Supervisor / Consumer Supervisor Training

Purpose To improve DSW job satisfaction by improving

communications with supervisors and/or consumers

Implementation Target Audience Needs Assessment Training Technique

Perceived Outcomes Training Effectiveness Retention and Turnover Community Building among Participants

13

Realistic Job Previews

Purpose To set realistic candidate expectations for the job and

improve retentionImplementation

Match between RJP Content and Job Context Third-Party Implementation of RJP Consumer Participation

Perceived Outcomes Person-Job Fit Turnover Quality of Care

14

Peer Mentoring Initiative

Purpose To provide support to newly hired workforce

Implementation Targeted new staff and mentors selected from veteran

staff Matching Process, Contact, Intensity, and Variability

in Documentation across Grantees

Perceived Outcomes Resistance among Supervisory Staff Mentors enjoyed program Unlikely to Improve Retention

15

Recognition

Purpose To improve DSW retention by providing either merit-

based and longevity incentivesImplementation

Initiated longevity award with early retention goals (6th months, 1 year) and later added longer retention awards to acknowledge all staff

Logging Tenure, Applying for Award Type of Recognition Public Recognition

Perceived Outcomes Staff Appreciation

16

Worker Registry

Purpose To improve both recruitment and retention by establishing

better match between job and workerImplementation

Development Time Worker Eligibility Consumer Eligibility

Perceived Outcomes Income Stabilization Balancing Supply and Demand Consumer Choice Employment and Costs

17

Marketing

Purpose To increase public awareness and recruit new DSWs

to the field

Implementation Understanding the Community Mode of Marketing

Perceived Outcomes Effectiveness of Mass Marketing Efforts Effectiveness o f Targeted Marketing Efforts

18

Targeted Recruitment Strategies

Purpose To identify non-traditional populations to recruit to

the field

Implementation Sources for Recruitment Limited Resources or Staff to Recruit Candidates Overcoming the Barriers to Working as a DSW Case Management Approach

Perceived Outcomes Participation and Employment

19

MARKETING AND PARTICIPATIONIMPLEMENTATION

EVALUATIONOUTCOMES

General Lessons Learned

20

Marketing and Participation of Initiatives

Research the population you are trying to reach

Develop and showcase program championsSupport behavioral change through case

management and modeling

21

Implementation of Initiatives

Consider who is in chargeInitiatives take timeDon’t overwhelm workforce with optionsCarefully plan incentives

22

Local Evaluation of Demonstration

Consider the heterogeneity among worker sample

Engage local evaluators early and oftenDon’t set up to assess a “moving target”

23

Demonstration Outcomes

Consider ways to improve job satisfactionInitiatives may have positive, unanticipated

benefits Improved Agency Support Worker Community Building

Consider a “package” of initiatives

24

Contact InformationContact Information Link to ReportLink to Report

Sarah Hunter RAND

(310) 393-0411 ext 7244shunter@rand.org

Laura SteighnerAIR

(202) 403-5064lsteighner@air.org

http://www.rand.org/pubs/technical_reports/TR699

Additional Information