1

The Effects of Customer Choice:The Effects of Customer Choice:First Findings from the First Findings from the

Individual Training Account Individual Training Account (ITA) Experiment(ITA) Experiment

Mathematica Policy Research, Inc.Mathematica Policy Research, Inc.Social Policy Research AssociatesSocial Policy Research Associates

Presenter: Irma Perez-JohnsonPresenter: Irma Perez-Johnson

DOL Workforce Innovations ConferenceDOL Workforce Innovations ConferencePhiladelphia, PA – July 11-13, 2005Philadelphia, PA – July 11-13, 2005

2

Policy Background:The Workforce Investment Act

Key objective was to promote informed Key objective was to promote informed customer choicecustomer choice

Required use of ITAs to give customers choice Required use of ITAs to give customers choice of training programof training program

Customer selections to be made from Eligible Customer selections to be made from Eligible Training Provider listsTraining Provider lists

Established Consumer Report Systems to Established Consumer Report Systems to provide information on programsprovide information on programs

3

Overview of the ITA Experiment

Goal was to assess important tradeoffs in the Goal was to assess important tradeoffs in the design of ITA approachesdesign of ITA approaches

Tested 3 approaches side-by-side in 8 LWIAsTested 3 approaches side-by-side in 8 LWIAs

Customers randomly assigned to approachesCustomers randomly assigned to approaches

Approaches selected to:Approaches selected to:

– Reflect spectrum of emerging ITA approachesReflect spectrum of emerging ITA approaches

– Promote innovation in ITA program designPromote innovation in ITA program design

4

Approaches Tested

Approach 1: Approach 1: Structured Structured Customer Customer

ChoiceChoice

Approach 2:Approach 2:Guided Guided

Customer Customer ChoiceChoice

Approach 3:Approach 3:MaximumMaximumCustomerCustomer

ChoiceChoice

CounselingCounseling Mandatory;Mandatory;most intensivemost intensive

Mandatory;Mandatory;moderate intensitymoderate intensity VoluntaryVoluntary

Can Can Counselors Counselors Reject Reject Choices?Choices?

YesYes NoNo NoNo

Award Award AmountAmount

Customized, Customized, High CapHigh Cap

Fixed, Fixed, ModestModest

Fixed, Fixed, ModestModest

5

Evaluation Data Sources

Implementation StudyImplementation Study**

Study Tracking SystemStudy Tracking System**

– Analyzed data for 7,922 study participantsAnalyzed data for 7,922 study participants

Follow-Up SurveyFollow-Up Survey

Unemployment Insurance RecordsUnemployment Insurance Records

* Data sources for interim report* Data sources for interim report

6

Methodology Estimate impacts on intermediate and final Estimate impacts on intermediate and final

training outcomestraining outcomes

Intermediate outcomes:Intermediate outcomes:– Participation in counseling and trainingParticipation in counseling and training– Training choices Training choices

Final outcomes:Final outcomes:– Completion of trainingCompletion of training– Customer satisfactionCustomer satisfaction– Employment and earningsEmployment and earnings– Receipt of UI, public assistanceReceipt of UI, public assistance– Cost of counseling, trainingCost of counseling, training

7

Responses to the ITA Approaches

Most customers came in with strong ideas about Most customers came in with strong ideas about trainingtraining

When counseling was voluntary, few customers used itWhen counseling was voluntary, few customers used it

Counselors tended to defer to customer preferencesCounselors tended to defer to customer preferences

Local staff preferred Approach 2 (Guided Choice)Local staff preferred Approach 2 (Guided Choice)

Private vendors repackaged their programs in response Private vendors repackaged their programs in response to capsto caps

Little response from community colleges, universitiesLittle response from community colleges, universities

8

Mandatory Counseling Discouraged Use of ITAs

59 5865***

0

20

40

60

80

100

Secured ITA Program Approval

Per

cen

t

A1: Structured Choice A2: Guided Choice A3: Maximum Choice

*** Difference relative to A2 is statistically significant at 0.01 confidence level

9

Training Costs Highest Under Approach 1, Lowest Under Approach 2

A1: A1: Structured Structured

ChoiceChoice

A2: A2: Guided Guided ChoiceChoice

A3: A3: Maximum Maximum

ChoiceChoice

Average price of Average price of selected programsselected programs $4,764***$4,764*** $3,430$3,430 $3,133$3,133

Average ITA Average ITA expenditures per traineeexpenditures per trainee $4,731***$4,731*** $2,849$2,849 $2,847$2,847

Training rate Training rate (percentage)(percentage) 5959 5858 65***65***

Average training cost Average training cost per ITA customerper ITA customer $2,791***$2,791*** $1,641$1,641 $1,853***$1,853***

NOTE: Cost estimates do not include WIA administrative costs or cost of ITA counseling.

*** Difference relative to A2 is statistically significant at 0.01 confidence level

10

Limited Differences in Training Choices by Approach

ITA customers chose similar occupationsITA customers chose similar occupations

Private schools were the dominant provider Private schools were the dominant provider regardless of approachregardless of approach

Approach 3 customers slightly more likely to Approach 3 customers slightly more likely to choose community collegeschoose community colleges

Approach 1 customers chose slightly longer Approach 1 customers chose slightly longer programsprograms

11

Remaining Questions

Do more counseling and/or higher caps lead to Do more counseling and/or higher caps lead to better customer outcomes?better customer outcomes?

Does lack of counseling lead to poorer Does lack of counseling lead to poorer customer outcomes?customer outcomes?

What is the most cost-effective approach to What is the most cost-effective approach to structuring ITAs?structuring ITAs?

12

Remainder of ITA Experiment

Follow-up surveyFollow-up survey

Administrative recordsAdministrative records

Final report (summer 2006)Final report (summer 2006)