MEANS-TESTED EMPLOYMENT AND TRAINING PROGRAMS
Burt S. BarnowGeorge Washington University
Jeffrey SmithUniversity of Michigan
For Presentation At The Organization For Economic Cooperation And Development, Paris, France
February 2, 2016
1
2
Topics Types of programs covered Rationales for government support for E&T programs History of US E&T programs from the Great Depression to
present Funding patterns over time and current programs Evaluation issues
Use of RCTs Major alternatives to RCTs Other issues: data, general equilibrium, follow-up
Findings from evaluations of WIA, Job Corps, other programs Program operation issues Conclusions
3
Types of Activities Covered Skill development increase vocational skills through classroom
or on-the-job training; Job development programs consist of public employment
programs where jobs are specifically created for the participants; Employability development programs improve personal
attitudes and attributes needed for employment (soft skills) Work experience programs provide employment experiences
intended to help workers gain the same attitudes and attributes as employability development programs.
Labor exchange programs that help match job seekers with job openings;
Counseling and assessment and labor market information (LMI) help workers learn more about their abilities and aptitudes, and provide information about the current and future labor market.
Note: Although it is sometimes assumed that E&T means training, Barnow and Trutko (2007) found < 50% of WIA exiters received trainingSource: Butler and Hobbie (1976) augmented
4
Focus on “Means Tested” Programs
Many US programs not means tested Unemployment insurance Vocational education Employment service (labor exchange services) Registered apprenticeship programs WIA/WIOA not means tested for dislocated
workers and only for adults if insufficient funds in local areas
Also exclude place-based programs, in-school youth programs, state/local funded programs
5
Why Have Government Support?
E&T programs not public goods or natural monopoly, so could rely on private sector
Musgrave “merit goods” argument Imperfect access to capital for poor Compensation for government actions or
unforeseen events (displaced workers) Imperfect information Some rationales call for means testing,
but others do not
6
History of E&T Programs:Programs Established in the Great Depression
Under Hoover, Reconstruction Finance Administration spent $300M on work relief, employing up to 2M people
Many work relief programs under FDR—largest was Civil Works Administration with 4.3M workers
In 1933 Wagner-Peyser Act established the Employment Service Not means tested, so not covered in depth here ES often used to enforce ALMP provisions, e.g., UI
work test Real budget has been reduced for years for ES,
leading to emphasis today on self-service and staff-assisted labor exchange
7
Programs in 1960s New Deal programs (except ES) stopped in
1943—just ES lived on Training programs emerged in 1960s
Area Redevelopment Act (1961) was 1st program Manpower Development and Training Act (1962)
Originally passed to deal with “automation” (technical change)
No automation, so focus on the poor with classroom training and OJT (2/3 CT and 1/3 OJT)
Served 1.9M participants 1963-1972 Note: Ashenfelter did pioneering evaluations of training
for MDTA
8
Programs from 1973-1998 Comprehensive Employment and Training Act (CETA)
Ran from 1973-1983 and established system of local agencies running programs
Included public service employment, which grew to be the largest component of CETA
Concern about “fiscal substitution” in PSE programs led to restrictions on people and work, making program ultimately unpopular with all parties (although estimated to have large impact on earnings)
Concern about “creaming” was large, leading to special programs for Native Americans & farmworkers
Non-experimental evaluations of CETA, all using the same data, had huge range of estimates, setting the stage for RCT evaluations (see Barnow 1987)
9
Programs from 1973-1998 Large youth initiative in 1977 proved poor
youth do want to work, but little else Two changes to CETA have endured:
Private Sector Initiative Program discovered employers
Government economists in DOL developed first performance measures and adjustment models
CETA replaced by Job Training Partnership Act in 1982
10
Job Training Partnership Act 1982-1998 Retained Basic CETA Structure
Programs for economically disadvantaged youth and adults continued to be locally administered;
States assumed a much greater role in monitoring performance of local programs;
Private sector was given the opportunity to play a major role in guiding and/or operating the local programs;
System was to be performance driven, with local programs held accountable and rewarded or sanctioned based on their performance;
Program added for dislocated workers Amendments in 1992 restricted who could be served and how
served—65% of participants had to be “hard to serve” First major DOL program evaluated with RCT showed modest
impacts for men and women, but no impacts for out-of-school youth No evaluation of programs for dislocated workers or in-school youth
11
Workforce Investment Act of 1998 Continued devolution of authority to states Called for services through One-Stop Career Centers (now
called American Job Centers) One-Stops were to have universal access—idea was to
avoid stigma Over a dozen mandatory partners in One-Stops who were
required to pay for infrastructure (in theory) To avoid rushing people into training who did not need it
(and to save $), participants (customers now) were to go through sequence of services: core, intensive, and then training
Training was reserved for the poor if not enough funds available for all
12
Workforce Investment Act of 1998 Customers were to have choice of training programs
through individual training accounts, which were like vouchers
To help assure high-quality training vendors, states were to establish an eligible training provider list by program, with standards for getting on list and staying on list
Performance measurement structure similar to JTPA, but DOL dropped statistical adjustments in favor of “negotiated” standards
Summer youth employment program abolished and year-round youth employment program established
Youth programs were now required to spend at least 30% of funds on out-of-school youth
13
Changes in Workforce Innovation and Opportunity Act (WIOA) of 2014 Data on training providers’ outcomes must be
made available Allows states to transfer unlimited amounts of
their grant between the adult and dislocated worker programs
Adds “basic skills deficient” as a priority category, along with low income, for Adult services
Requires that 75 percent of Youth funds be used for out-of-school youth, a large increase over the 30 percent required under WIA
Combines core and intensive service categories into “career services” and abolishes requirement that customers pass through core and intensive services before receiving training
14
Changes in Workforce Innovation and Opportunity Act (WIOA) of 2014 Permits direct contracts with higher education
institutions (class-size contracts rather than just ITAs)
Strengthens the requirements for partners in American Job Centers: ES required to be in AJCs, and TANF a mandatory partner
Reduces required employer contributions for customized and sectoral training programs
Includes specific performance measures for WIOA and other E&T programs, with employer satisfaction and longer follow-up than WIA
15
Employment and Training Expenditure Patterns Over Time
Funding has generally declined in real terms since the 1980s The share of GDP devoted to E&T programs (except ES) has
shrunk from .094% in 1985 to .048% in 2012 The Recovery Act greatly increased activity temporarily during
Great Recession, but funding ended while unemployment still high
Funding affected in part by evaluations Youth funding greatly reduced after National JTPA Study
showed youth programs ineffective Job Corps funding increased when initial results showed
program effective (but not reduced when results not sustained)
Dislocated worker funding has increased over time despite lack of evidence on program effectiveness
16
Funding for DOL Employment & Training Programs, 1965-2012
1965
1967
1969
1971
1973
1975
1977
1979
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
,0
10000,000
20000,000
30000,000
40000,000
50000,000
60000,000
Year
2012
Dol
lars
(tho
usan
ds)
17
Funding as Percentage of GDP DOL Employment & Training Programs, 1965-2012
1965
1967
1969
1971
1973
1975
1977
1979
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
0.00%
0.10%
0.20%
0.30%
0.40%
0.50%
0.60%
0.70%
0.80%
0.90%
Year
Perc
enta
ge o
f G
DP
18
Current DOL Employmentand Training Programs
Currently 14 DOL programs with at least $30M annual funding
Two largest programs are Job Corps, residential program for poor youth, and WIA Dislocated Worker program
Programs mostly targeted by economic status, age, reason for lack of employment
Wagner-Peyser Employment Service is major exception—open to all
19
Current Funding for Major DOL Employment and Training Programs
Job CorpsDOL / Employment Training Administration $1,684
WIA Dislocated WorkersDOL / Employment Training Administration $1,219b
WIA Youth ActivitiesDOL / Employment Training Administration $818
WIA Adult ProgramDOL / Employment Training Administration $764
Wagner-Peyser Funded Employment Service
DOL / Employment Training Administration $664*
Senior Community Service Employment Program
DOL / Employment Training Administration $433
Trade Adjustment Assistance (TAA)
DOL / Employment Training Administration $306c
Disabled Veterans Outreach Program (DVOP) and Local Veterans’ Employment Representative Program (LVER)
DOL / Veterans' Employment and Training Service $175
H-1B Job Training GrantsDOL / Employment Training Administration $166**
20
Major Programs Outside DOL
Pell Grants support higher education for low-income students Total support for Pell Grants $33.7B Support for E&T from Pell estimated to be
$8.7B, more than funding from all 3 WIA funds + ES
Other non-DOL programs also large TANF welfare program spends $1.5B on E&T Adult education spends $564M SNAP (Food Stamps) E&T has budget of
$416M
21
Current Funding for Employment and Training Programs Outside DOL
Pell GrantsEd / Office of Vocational and Adult Education $8,181
Temporary Assistance for Needy Families (TANF) Grants
HHS / Administration for Children & Families $1,517d
Adult Education - Grants to States
Ed / Office of Vocational and Adult Education $564e
SNAP Employment & Training
USDA / Food and Nutrition Service $416f
22
Are There Too Many Employment and Training Programs?
In 1994 GAO claimed 154 E&T programs, but many were not programs (e.g., incentive payments)
In 2011, GAO counted 47 and we count 20 with at least $30M
Many programs are pilots or have special target groups
Biggest issues are ES/WIOA and TANF/WIOA Duplication has some advantages, but
overall hard to argue there are not too many
23
Evaluation Issues for E&T Programs
Basic equation is where:Yi is the outcome of interestDi is treatment statusY01 is the outcome without treatment and Y11 is the outcome with treatment Problem is we do not observe outcome
with and without the treatment for same person
1 0(1 )i i i i iY DY D Y
24
What Do We Want to Estimate? Most commonly we want average
treatment effect on the treated: Sometimes we want average treatment
effect for entire population, which could differ if impact varies by selection:
Sometimes interested in quantile treatment effects, e.g., impact at median or other point
1 0( | 1)E Y Y D
1 0( )E Y Y
25
What Do We Assume on Treatment Effect?
Key issue is whether to assume common treatment effect Older literature assumed impact identical for all, but
theory and evidence suggest otherwise Many programs involve selection decision, which might depend
on impact Treatment received not identical, so impacts likely to vary Issue of whether impacts vary by economic conditions or
characteristics ultimately an empirical one—why assume it away?
At minimum, most studies look at impacts by sex, often by race/ethnicity
Paper reviews Heckman Robb (1985) and Heckman, LaLonde, and Smith (1999) models to show importance of selection in estimating impact, e.g., ATET>ATE>ATNT
26
Random Assignment not a Cure-All
Heckman and Smith (1995) discuss randomization bias as potential issue
Randomization while keeping enrollment constant requires increasing number of applicants to program
If selection depends on expected impact and impact varies by number selected, random assignment will give impact for wrong program size
Ideally, sites should also be randomly selected—sometimes this works (Job Corps, WIA), but not always (JTPA)
27
Random Assignment not a Cure-All Another problem is that randomization can lead to
substitution bias, where control group receives similar treatments—in JTPA evaluation large share of control group received training
Not all in experiment comply with assignment No-show rate can be high: estimate effect on treatment on
treated, but is this what we want? Can use IV (Bloom1984) HST (1998) if impact on no shows
= 0 Crossovers from C to T bigger problem but can be dealt
with (Orr 1998) See Greenberg and Barnow (2014) and Barnow (2011)
for examples of things that can go wrong
28
Non-Experimental Approaches
Include rich mix of covariates and assume that selection is based on observable variables Widely used but often with no proof valid assumptions
Propensity score matching Widely used, with mixed results when tested with RCT
data Regression discontinuity designs
Rare in E&T context Difference-in difference models
Often combined with other approaches, good control on non-varying unobservables
29
Data and Measurement Issues Data on service receipt often not
measured well, especially for control group
Administrative data and survey data have different strengths and weaknesses, and they can lead to contrasting impact findings (Barnow and Greenberg 2015)
Length of follow-up is very important for CBA; see Job Corps analysis (Schochet et al. 2006) and JTPA long-term follow-up (GAO 1996)
30
Some Issues for Cost-Benefit Analysis
When performed, usually compare average costs and benefits—more work on marginal BCA needed
Assumptions after observed follow-up are key—Job Corps is good example
Limited analyses of outcomes other than earnings, e.g., crime, fertility, health
Valuing “leisure” time of participants (to themselves and society) difficult and rarely done
31
General Equilibrium Effects Displacement, where participants take jobs that would
have gone to control group members, can make social gains less
If T group enters different labor market, wages to C group could increase, making social gains more
Large programs could change relative prices Scale effects can be captured if scale varied across
labor markets; see Crepon et al. (2013) Possible to estimate general equilibrium models, like
Davidson & Woodbury (1993), Lise et al. (2004), and Heckman et al. (2004), all of which found large GE effects
32
Findings from Major WIA Evaluations
We present findings from 3 major studies, all using exact matching and PSM Studies differ in states, time period,
variables controlled for, and method to some extent
Heinrich et al. (2013) (training v. no training) For adult women, ~$800/quarter Q4-Q16 For adult men, ~$500/quarter in later
quarters For dislocated workers, no patterns of gains
33
Findings from Major WIA Evaluations Andersson et al. (2013)
Adults M/F pooled gain $300-$400 quarter in later quarters
Dislocated workers lose ~125/quarter in one state and gain ~$300/quarter in other state
Hollenbeck (2009) Indiana pooled M/F Adults gain $549 in 3rd quarter after exit and
$463 in 7th quarter after exit Dislocated workers gain $410 in 3rd quarter
after exit and $310 in 7th quarter
34
Summary of WIA Findings Researchers generally find modest, positive
earnings gains for adults from training, that appear to persist for several years
Findings for dislocated workers much less consistent, often zero or negative, perhaps because populations differ or perhaps because hard to distinguish temporary from permanent shocks
Results from WIA RCT due later this year will help sort this out, particularly inconsistencies on dislocated workers
35
Evaluation of Job Corps Job Corps is long-term residential
program for poor youth and has larger budget than any other DOL E&T program
Exemplary RCT evaluation strategy includes Most Job Corps sites included good for
external validity Small control group at each site to reduce
bias Use of administrative and survey data Analysis of outcomes including crime
36
Evaluation of Job Corps:Major Findings Job Corps increases education and training for
T group by about 1 academic year Job Corps increased literacy skills For first 2 years after random assignment,
participants earned more, ~12% more in years 3 and 4 after random assignment
In years 5-10, no difference in earnings for T and C groups
Job Corps reduced crime ~5 percentage points Overall, B<C except for 20-24 year olds
37
Other E&T Programs of Interest Trade Adjustment Assistance evaluated by quasi-
experimental methods had no impact Many studies of welfare to work programs, often
showing modest impacts Evaluation of three sectoral programs by Public-
Private Ventures using RCTs found large impacts for 2 years after random assignment
Evaluations of dislocated worker programs mixed, but little evidence training valuable
Little credible evidence that youth programs effective
38
Program Operation Issues Research on vouchers mixed, generally indicating
vouchers popular but do not improve impacts much, if at all
Performance measurement studies show typical measures not correlated with impacts and often have perverse incentives
More studies of participation would be useful for understanding programs and evaluations
There is current interest in career pathway programs, training to obtain industry-sponsored credentials, and sectoral training programs, with evaluations underway
39
Summary and Suggestions Programs for poor adults pass CBA test, but do not
make participants self-sufficient: can we do better? Results for dislocated workers sparse and mixed:
will WIA evaluation change things? Youth programs disappointing: can we build on Job
Corps findings to do better? Getting good cost data very difficult and makes
good CBA challenging Although RCTs have key role in evaluations, use of
non-experimental designs important for looking at marginal program changes