ellig mc tigue richardson outcome based scrutiny 2000
TRANSCRIPT
Outcome-Based Scrutiny:Outcome-Based Scrutiny:Putting a Price on PerformancePutting a Price on Performance
Government Accountability Project
Mercatus Center
George Mason University
Arlington, VA
http://www.mercatus.org
What is Outcome-Based What is Outcome-Based Scrutiny?Scrutiny?
Assessment of federal programs to determine:
1. What specific benefits were produced (or harms averted) for citizens?
2. How could resources be reallocated to make government more effective?
The 4The 4thth Step in the GPRA Step in the GPRA ProcessProcess
Planning (Strategic Plans)
Implementation
Disclosure (Performance Reports)
Scrutiny
Why Outcome-Based Scrutiny?Why Outcome-Based Scrutiny?
• Improve quality of agency reporting
• Improve congressional resource allocation decisions
• Drive fundamental management change within federal agencies
A 7-Step ProcessA 7-Step Process
Mercatus demonstration study: Mercatus demonstration study: Vocational Training ProgramsVocational Training Programs
Note: This is a demonstration study only, not a full analysis that provides sufficient information for congressional decisions.
1. Outcome: “What is the issue 1. Outcome: “What is the issue and the intended outcome?”and the intended outcome?”
Focus on programs whose primary purpose is to enhance employability
Outcome measure is the number of people who moved into work or became more employable as a result of the program
2. Agencies: “Who is 2. Agencies: “Who is responsible?”responsible?”
• Labor• Education• Health & Human Services• Interior• Housing & Urban Development• Justice
[Narrowed down from 12 agencies]
3. Programs: “How do they do 3. Programs: “How do they do it?”it?”
Department # of Programs
Labor 17
Education 11
Health & Human Svcs. 7
Interior 4
Housing & Urban Dev. 2
Justice 3
Narrowing the list…Narrowing the list…
105 programs mention vocational training
44 (grouped into 25 areas in APRs) have it
as a primary goal
34 (in 14 areas) have outcome data
How comprehensive is this?How comprehensive is this?
Our analysis covers
• 77 percent of vocational programs
• 56 percent of vocational program areas defined in performance reports
• 97 percent of spending we identified
Where’d we get this stuff?Where’d we get this stuff?
• Programs: Agency Strategic Plans, Catalog of Federal Domestic Assistance (CFDA), agency web sites
• Outcomes: GPRA Annual Performance Reports, agency analysts
• Cost figures: CFDA, U.S. Budget, agency budget justifications
4. Results: “Are they effective?”4. Results: “Are they effective?”
Program Area # of PlacementsVocEd - State Grants and Tech-Prep (Education) 1,600,000Veterans in Need (Labor) 288,404School-to-Work (Education) 257,600Adult Disadvantaged Job Training (Labor) 207,000Dislocated Workers (Labor) 163,438Welfare-to-Work (Labor) 79,771Youth Transition (Labor) 71,397Job Corps (Labor) 58,010Economic Independence - Refugees (HHS) 52,298BIA Community Development (Interior) 11,158Indian and Native Americans (Labor) 7,980Migrant and Seasonal Farmworkers (Labor) 7,300Trade-Affected Workers (Labor) 4420Tribal Postsecondary Vocational Institutions (Education) 187
5: Cost/Benefit: “What is the best 5: Cost/Benefit: “What is the best value?”value?”
Program Area Appropriation Placements/$million Cost/placement
School-to-Work (Education) $149,000,000 1729 $578
Veterans in Need (Labor) $167,000,000 1727 $579
VocEd - State Grants and Tech-Prep (Education) $1,056,000,000 1515 $660BIA Community Development (Interior) $12,928,000 863 $1,159
Adult Disadvantaged Job Training (Labor) $955,000,000 217 $4,608
Indian and Native Americans (Labor) $53,700,000 149 $6,711Economic Independence - Refugees (HHS) $423,000,000 124 $8,065
Dislocated Workers (Labor) $1,347,000,000 121 $8,264
Migrant and Seasonal Farmworkers (Labor) $73,000,000 100 $10,000Youth Transition (Labor) $1,201,000,000 59 $16,949
Welfare-to-Work (Labor) $1,476,000,000 54 $18,519Job Corps (Labor) $1,188,000,000 49 $20,408Tribal Postsecondary Vocational Institutions (Education) $4,100,000 46 $21,739Trade-Affected Workers (Labor) $131,100,000 34 $29,412
6: Reallocation: “What changes 6: Reallocation: “What changes give the greatest benefits?”give the greatest benefits?”
• Total expenditure: $8.4 billion• Total benefit: 2.8 million placements• Average cost: $3000/person
• Hypothetical total benefit if most cost-effective program received all resources: 14.5 million placements
Reallocation within 4 youth Reallocation within 4 youth programs with outcome info.programs with outcome info.
• Total cost: $3.6 billion• Total placements: 2 million
• Hypothetical placements if 2 most cost-effective programs received all resources: 5.5 million
7: Forgone Opportunity: “What do 7: Forgone Opportunity: “What do we lose under the status quo?”we lose under the status quo?”
• All programs example:14.5 million hypothetical placements
- 2.8 million actual placements11.7 million people not helped!
• Youth example:5.5 million hypothetical placements
- 2 million actual placements3 million young people not helped!
Implementation TipsImplementation Tips
• Alternative values define public benefit
• Correlation is not causation
• It can be OK to mix apples and oranges
• Unit costs may not be constant
• Match results with expenditures
Alternative values define public Alternative values define public benefitbenefit
• Forgone opportunity may be 11.7 million people…
• …or 281,178 additional trade-affected workers who might have been helped if all the resources went to that program.
• If it only costs $578 per placement, do those students really need government help?
Correlation is not causationCorrelation is not causation
Observed placements
- Placements that would have occurred in the absence of the program
_____________________________________
Outcome actually attributable to the program
Apples and OrangesApples and Oranges
• Complaint: Comparisons “unfair” due to differences among target populations.
• Judgment required: Compare all vocational programs or all youth vocational programs?
• The forgone opportunities are still real.
Unit costs can changeUnit costs can change
• Easy results may already have been achieved.
• Economies of scale may not be fully realized.
• Forgone opportunity may be larger or smaller than our calculations suggest.
Marginal and average costs can Marginal and average costs can vary with outputvary with output
Match results with expendituresMatch results with expenditures
• Cost is a single year’s appropriation, but placement figures are for a single year.
• Ideal data would be time series, not snapshot.
• Match this year’s outcomes with past expenditures contributing to the outcomes?
Outcome-Based Scrutiny Outcome-Based Scrutiny improves improves KnowledgeKnowledge and and
IncentivesIncentives
• Reveals what kind of information is needed from agencies
• Reallocation improves govt. performance• Comparison promotes interagency learning• Prospect of reallocation motivates
improvement