program reporting and performance measurement 1 general session #1
TRANSCRIPT
PROGRAM REPORTING AND PERFORMANCE
MEASUREMENT
1
General Session #1
Agenda
• Introduction and Background
• Program Reporting Requirements/Revisions
• Participation, Exit and Related Terminology
• Basic Measurement Rules/Concepts
• Calculating Outcomes on Adult Common Measures
2
Introductionand
Background
3
Conference Organization and Expectations
• Lots of information to share!
• General sessions precede specific workshops
• Team approach
• Materials to be shared after all three events
4
Programs Covered
• Two agencies, same Department
– VETS and ETA are both agencies within USDOL with similar missions and visions
• Funding
– WIA, Wagner-Peyser and Trade Act monies fund positions, programs and systems whereas VETS monies primarily fund positions
• “VETS-funded programs” deliver services through Local Veterans’ Employment Representatives (LVERs) and the Disabled Veterans’ Outreach Program (DVOPs)
WIA, W-P, VETS, TradeWIA, W-P, VETS, Trade
5
Establishing Common Measures
• Effort led by Office of Management and Budget (OMB) as part of President’s Management Agenda
– Linking performance to budget
– Supporting effective programs
– Comparing similar programs across agencies
– www.whitehouse.gov/omb/budintegration
• Includes six federal Departments
– Labor; Education; Health and Human Services; Veterans’ Affairs; Interior; HUD
6
USDOL Policy
• Common measures became effective for WIA, W-P and VETS on 7/1/05, and for Trade on 10/1/05
• Final policy located in TEGL 17-05, dated 2/17/06
– Rescinds previous policy guidance
• TEGL 28-04, TEGL 7-99, TEGL 6-00 and 6-00 Change 1
– Applies to DOL-funded programs only
– The basis for this conference!
7
The Common Measures
• Adult Measures– Entered Employment– Employment Retention– Average Earnings*
• Youth Measures– Placement in Employment or Education– Attainment of a Degree or Certificate– Literacy or Numeracy Gains
*Average Earnings becomes effective PY 2006; for PY 2005, the measure is an Earnings Change
8
Common Measures: Benefits Focus on the core purposes of the workforce system;
employment for adults and skill attainment for youth
Break down barriers to integration resulting from different definitions, data and reports for each workforce program
Resolve questions raised by GAO and other oversight agencies regarding consistency and reliability of data
Reduce confusion among customers and stakeholders who want to know about results
9
Information Needs
• Common measures will be the core information for all programs but other management information will continue to be necessary
– Management and oversight of programs will continue to require information relevant to each program
– Additional information about participants, services and outcomes must supplement the story told by common measures
10
Reality Check• PY 2005PY 2005 performance accountability requirements for WIA,
Wagner-Peyser, Trade and VETS-funded programs include required performance measures, required reporting and recommended reporting
• PY 2006PY 2006 will be different
– Reporting package finalized/effective for 3 years
– Baselines generated in PY 2005 used in PY 2006
– Baseline year begins for one common measure
– Earnings Change replaced by Average Earnings
• Challenge: Although certain accountability requirements are “phased in,” system adaptations should be on a faster track
11
(1) PY 2005 Accountability
• Accountable for specific Common Measures– Entered Employment Rate (WIA, W-P, VETS, Trade)
– Employment Retention Rate (WIA, W-P, VETS, Trade)
– Earnings Change (WIA, W-P, VETS, Trade)
• Accountable for specific WIA statutory measures– Adult and DW Employment & Credential
– OY Entered Employment, Retention, Earnings, Credential
– YY Retention, Skill Attainment, Diploma Attainment
– Customer Satisfaction (Participants and Employers)
12
(2) PY 2005 Accountability
• Accountable for reporting on specific Common Measures– Youth Attainment of Degree/Certificate
– Youth Placement in Employment & Education
• Reporting recommended for Youth Literacy/Numeracy – Required beginning PY 2006
• WIAWIA performance accountability for PY 2005 still consists of the “17” measures– With some changed definitions
13
Program Reporting Requirements/Revisions
14
Overview of Basic Reporting Requirements
• Two Types of Federal Reporting
• Required Program Reports
• Reporting Schedules and Due Dates
• Rolling Four Quarter Reporting Methodology
15
Two Types of Federal Reporting Required
• ALL grantees of federal funds (whether formula or discretionary) are required to submit two types of federal reports within specified time frames
1.1. Financial (or fiscal)Financial (or fiscal)
How much are you spending, on what, and at what rate?
2.2. Program (or programmatic)Program (or programmatic)
Who are you serving, what are they getting, and with what results?
The focus of this conference is program reporting16
Required Program Reports
• Wagner-Peyser– ETA 9002 (A through E)
• VETS– VETS 200 series reports (A through C)
• WIA– WIA Standardized Record Data or WIASRD
(pronounced wizard)– Quarterly (9090) and Annual (9091) Reports
• Trade– Trade Adjustment Participant Report or TAPR
(pronounced tapper)– ETA 563
17
Reporting Schedules & Due Dates
• Wagner-Peyser, VETS, WIA and Trade require quarterly fiscal and program reports
– W-P, VETS and WIA programs operate on a program year basis (July through June); the first quarter of the program year is July—September
– Trade programs operate on a program year basis, with the program year based on the federal fiscal year; the first quarter of the program year is October—December
• Quarterly reports are due "no later than 45 days after the end of the quarter"
– If the 45th day falls on a Saturday or Sunday, reports are due the Friday before
18
Reporting Methodology: Rolling Four-Quarter
• Wagner-Peyser, VETS and WIA (not Trade) report based on a rolling four-quarter methodology
– Allows us to speak in terms of four quarters of data (or annual results) whether we’re talking about calendar years, program years, or federal or state fiscal years
– For any report quarter and for any reporting element, we have one year’s worth of data; the most recent data available is reported
• Discussed further in individual workshops19
Overview of Reporting Revisions Across Programs
• ETA’s General Policy on Program Reporting
• Unique Identifiers for Participants
• Minimum Data for Self-Service Customers
• Collection of Workforce Information
• Reporting Co-Enrollment
• Reporting on Veterans
20
Introduction• Reporting requirements were changed/modified in
order to incorporate the common measures
– Reporting changes for One-Stop employment and workforce information services (W-P/VETS)* are most profound (e.g., from registration-based performance system to exit-based)
• Information added, revised or clarified across all or most program reports to support integrationintegration efforts
• Program-specific reporting changes discussed in individual workshops
21 *new moniker to replace ‘labor exchange’
ETA’s General Policy on Program Reporting
• In general, individuals who are eligible for a program and receive any service, regardless of where or how the service is accessed, should be reported in participant counts
– This is NOT the same as saying that all individuals included in participant counts are also included in performance calculations
– Example: WIA participants who receive only self-service and informational activities are included in participant counts but not included in performance calculations
22
Emphasis on Unique/Consistent Identifiers
• The same identifier must be used for the same individual for every period of participation in the state, every local area across the state and when participants receive services funded by Wagner-Peyser, VETS, WIA and/or Trade
– Consistent with efforts to facilitate integration across partner programs
• Federal reports cannot contain SSNs – only alternate IDs or encrypted SSNs are accepted
23
Data Requirements for Self-Service Customers
• Minimum data needed from self-service participants specified
1. Social Security Number* • Upon request of SSN, customer must be offered the
opportunity to provide Equal Employment information; if no SSN provided, a pseudo-identifier must be used
2. Equal Opportunity Data (4 items)• Voluntary and self-identified (Ethnicity, Race, Disability
Status, DOB for age)
3. Employment status at participation
24*Without an SSN, the individual cannot be included
in performance calculations
Receipt of Workforce Information
• This is a new data element as of 7/1/05 that affects W-P, VETS and WIA-funded programs*
– USDOL spends >$38 million on workforce information grants but there is little data on impact
– To better understand the return on the federal investment, programs will collect/report data on provision of workforce information
• Note: Workforce Information can be accessed in many ways (e.g., through individual counseling or career guidance in a One-Stop Center, remotely via the Internet and/or in satellite agency sites)
*Trade programs don’t require this information since their primary focus is training and it is assumed participants will be
co-enrolled in other programs such as WIA where this information will be collected
25
Receipt of Workforce Information (cont’d)
• Definition:
– Includes information on state and local labor market conditions, local employment dynamics info, industries, occupations and characteristics of the workforce, area business identified skills needs, employer wage and benefit trends, short and long term industry and occupational projections, worker supply and demand, and job vacancies survey results. It also includes local employment dynamics info such as workforce availability, business turnover rates, job creation, and job identification of high growth and high demand industries.
This is the definition contained in the ET Handbook No. 406 (for W-P/VETS)
and the WIA reporting instructions26
Reporting Co-Enrollment
• Co-enrollment across WIA, Wagner-Peyser, and Trade programs must be reported
• If a WIA participant is also receiving Trade-funded services or Wagner-Peyser funded services, that must be captured in the WIA reporting system
• If a Wagner-Peyser participant is also receiving Trade- or WIA-funded services, that must be captured in the W-P reporting system
• If a Trade participant is also receiving Wagner-Peyser or WIA-funded services, that must be captured in the Trade reporting system
• This is consistent with [fill in the blank]_
27
Reporting on “Eligible Veterans”
• Definitions and reporting specifications (e.g., coding values) were aligned with requirements in the Jobs for Veterans Act (JVA) as of 7/1/05– Wagner-Peyser reports now reference “Veterans,
Eligible Persons, and TSMs” instead of “Veterans and TSMs” (TSM=Transitioning Service Member)
– WIA WIASRD and Trade TAPR coding values for veteran changed to include “eligible veteran” and “other eligible persons”
• Note: “Recently Separated Veteran” refers to 36 months for W-P and VETS, but 48 months for WIA (not collected in Trade)
28
Participation, Exit and Related Terminology
29
“Participant”• An individual determined eligible to participate in
the program who receives a service funded by the program in either a physical location (e.g., One-Stop Center) or remotely through electronic technologies
– Three components:
1. Determined eligible to participate in the program
2. Receives a funded service
3. In either a physical location or through electronic technologies
30
Clarification of ParticipantLooking at the pieces (which impact programs differently!)
1. Individual determined eligible to participate
• Depends on program/funding; doesn’t apply in the case of W-P, which is based on universal access
2. Receives a service
• Not all services trigger participation; it’s important to understand the distinction between those that do and those that don’t (see handout)
3. In a physical location or remotely
• Adding remote access to the definition is a fundamental change in policy
• Many substantial services are remotely accessed; this needs to be captured
31
Other Terms Related to Participation
• Participation Date: Date of first program-funded service– It’s possible to record an earlier date if services were
being provided by another partner at the time of participation
• Participation Quarter: Calendar quarter containing the participation date
• Participant Cohort: Group of individuals who share the same participation quarter
32
“Exiter”
• A participant who hasn’t received a program or partner-funded service for 90 consecutive days and no future services are scheduled
– Three components
1. Hasn’t received a service
2. For 90 consecutive days
3. No future services scheduled
• Often referred to as a “soft exit approach”
33
Clarification of ExiterLooking at the pieces (which impact programs differently!)
1. The participant hasn’t received a service
• Could be program- or partner-funded depending on policies/procedures (e.g., tracking)
2. For 90 consecutive calendar days
• A gap in service can stop the 90-day clock if based on specific/allowable circumstances
3. No future services scheduled
• Some services can extend the exit date
• Does not include any follow-up services or circumstances where the participant voluntarily withdraws or drops out of the program
34
Gaps in Service• Final common measures policy uses “gap in service”
as opposed to the former “planned gap”
• Three allowable circumstances, where the condition exists for at least 90 days
1. Delay before beginning of training
2. Health/medical condition of participant/family member
3. Temporary move from the area that prevents participation
• Gaps in service should be <180 days (from date of last service), although a subsequent gap could be initiated as necessary
• All gaps must be documented and are subject to audit35
Extending the Exit Date
• Services provided by partner programs can extend the point of exit
– More services potentially means higher outcomes
– Requires ability to track participants until they exit all services (program and partner)
• This actually highlights the essence of One-Stop, which is integrated services
– No one program or entity can do it all. Only by leveraging limited resources of multiple partners and players can we meet the comprehensive needs of system customers
36
Other Terms Related to Exit
• Exit Date: Last date of service (program or partner-funded)– Once a participant completes all services and 90 days
has passed, the exit date is applied retroactively to the last date of service
• Exit Quarter: Calendar quarter containing the exit date
• Exit Cohort: Group of individuals who exit during the same calendar quarter
37
Further Clarification of DATES
• Participation and Exit Dates are always dates of service– Participation Date reflects first funded service– Exit Date reflects last funded service
• Translation of “no more hard exit”– Not intended to take responsibility away from case
managers; for WIA, case managers do not have to wait 90 days, for instance, to begin providing follow-up services
– However, federal policy states that an exit cannot be officially recorded until that 90 days has elapsed
– One suggestion: It’s possible to have a “case closure” code for case managers
38
When To Exit
39
• Participants should be exited when the service plan or service strategy is complete
– The service plan is a “living document,” with additions and changes possible
– Co-enrollment in different funding streams, additional partner services and a valid gap in service can extend the exit date
• Participants should not be exited after a specific program intervention if additional services are needed
Illustration: Participation and Exit
Participation No Services
Exit DateParticipation
Date
Eligible and Receives Service
End of 90-Day Period
Last Service
40
Participation and Exit: A Systems Perspective
1st Program
3rd Program
2nd Program
Common Date of Participation
Common Exit Date
If a participant is served by multiple programs, it is possible
to record the earliest date of service as the date of
participation
In an integrated service delivery
system, a participant is
exited when all program
services are complete,
enabling all individual
programs to “share” in the final outcomes
Program Participation
Program Participation
Program Participation
41
Basic Measurement Rules/Concepts
42
Overview
• Only participants are included in performance calculations
• Calculations are based on exiters*• Specific conditions exclude participants from
calculations• Each measure is based on a specific cohort • Specific data sources must demonstrate the
outcomes
43 *The one exception is Youth Literacy/Numeracy
Only Participants Included in Performance Calculations
• From a measurement perspective, an individual doesn’t exist until they’re considered a participant
– It’s critical for state and local staff to have the same understanding regarding services that trigger participation, which may be remotely accessed
• Once a participant exits, they are subject to the measures, with exception of Literacy/Numeracy
– Translation: 5 of the 6 measures are “exit-based”
• Therefore, exit policies and procedures are of critical importance
44
Excluding Individuals From Performance Calculations
• Two basic circumstances in which participants are excluded
– Statutory Exclusion applying to WIA (adults receiving only self-service and informational activities are not included in performance calculations)
– One of six conditions exists either at exit or during the subsequent 3-quarter measurement period
• Note: The term “global exclusions” no longer part of common measures policy
45
Exclusions (cont’d)
• Allowable Exclusions
– Institutionalized
– Health/Medical or Family Care
– Deceased
– Reservists called to active duty (includes National Guard)
– Relocated to a residential or non-residential program (applies to youth only)
– Invalid or missing SSN
46
Each Measure is Based on a Specific Cohort
• Cohort = a group of similar individuals
– Example: Staff members in your unit or department; youth in the U.S. over age of 18 with HS diplomas
– In measurement, we reference many different cohorts, including participant cohorts, exit cohorts, reporting cohorts – all specify certain groups of individuals
• Translation: Not everyone is included in each measure; it’s based on the cohort
• Understanding specific cohorts is critical, particularly for front-line program staff
47
Specific Data Sources Must Demonstrate the Outcome
• Wage recordsWage records are the primary data source for any measure that includes employment (4 of 6)
– Exceptions: Youth Attainment of Degree/Certificate and Youth Literacy/Numeracy
• Some data sources have the same “stature” as wage records (i.e., considered equivalent)
– Include Wage Record Interchange System (WRIS); federal and military wage records accessed through the FEDES (Federal Employment Data Exchange System)
48
When There Aren't Wage Records
• Supplemental DataSupplemental Data can be used to demonstrate employment and retention, but not earnings*
– Implications for three common measures
• Adult Entered Employment, Adult Employment Retention, and Youth Placement in Employment or Education
– Allowable supplemental data sources include documented employer contact, employer and participant surveys, case management notes
• All supplemental data subject to audit
*Some grantees will be able to use supplemental data as an interim means of reporting on earnings (e.g., NFJP)
49
Data Sources for Measures Not Related to Employment
• Pertains to all three youth measures
Youth Placement in Employment or Education
• Administrative records utilized for placement in education (wage records for employment portion)
Attainment of Degree/Certificate
• Administrative records utilized
Literacy and Numeracy Gains
• Assessment instrument
• (Further detail in Workshop #3)50
Summary of Data Sources
Common Measure Data Source (s)
Adult Entered Employment RateWage records and supplemental data sources
Adult Employment Retention Rate
Wage records and supplemental data sources
Adult Earnings Change or Average Earnings
Wage records only (except for grantees without current wage record access)
Youth Placement in Employment or Education
Wage records and supplemental data for placement in employment or military; administrative records for placement in education or training
Youth Attainment of Degree or Certificate
Administrative records
Literacy/Numeracy Gains Assessment instrument
51
Calculating Outcomes on ADULT Common Measures
52
ENTERED EMPLOYMENT RATEENTERED EMPLOYMENT RATE
• Definition: For those not employed at participation, the percentage employed in the 1st quarter after exit
• Calculation:
Of those not employed at participation…
Number employed in the 1st quarter after exit
Number who exit during the quarter
53
ENTERED EMPLOYMENT RATEENTERED EMPLOYMENT RATE
Employed at Participation?
Employed in 1st quarter after exit?
YES
YES
YES
NO
Excluded
Numerator
Denominator
NO
54
Highlights of Adult EER
Excludes adults employed at participation
Employment at participation is based on information collected from the individual – not wage records
Those who received a notice of termination or the employer has filed a WARN or similar notice are considered not employed and are included in calculations
Transitioning Service Members (expected to retire within 24 months or separate within 12 months) are considered not employed and are included in calculations
55
EMPLOYMENT RETENTION RATEEMPLOYMENT RETENTION RATE
• Definition: For those employed in the 1st quarter after exit, the percentage employed in the 2nd and 3rd quarter after exit
• Calculation:
Of those employed in the 1st quarter after exit…
Number employed in the 2nd and 3rd quarter after exit
Number who exit during the quarter
56
EMPLOYMENT RETENTION RATEEMPLOYMENT RETENTION RATE
Employed in 1st quarter after exit?
Employed in 2nd and 3rd quarter after exit?
YES
YES
NO
YES
Excluded
Numerator
Denominator
NO
57
Highlights of Adult ERR Includes all those employed in the 1st quarter
after exit regardless of employment status at participation (this is the measurement cohort)
Positive outcome requires employment in two post-exit quarters (which does not have to be with same employer)
58
EARNINGS MeasuresEARNINGS Measures
• There are two earnings measures, one which is effective now – and a revised measure, which will be effective next program year (7/1/06)
– Beginning PY 2006, pre-program earnings will no longer be part of calculations
• PY 2005 – Earnings Change (comparison of pre- and post-program earnings)
• PY 2006 – Average Earnings (pre-program earnings not utilized)
59
(PY 2005) EARNINGS CHANGE(PY 2005) EARNINGS CHANGE
• Definition: For those employed in the 1st quarter after exit, the average change between pre- and post-program earnings using the 2nd and 3rd quarters prior to and after participation
• Calculation:Of those employed in the 1st quarter after exit…
Number who exit during the quarter
Total earnings in 2nd and 3rd
quarters after exit
Total earnings in 2nd and 3rd quarters prior to
participationminus
60
(PY 2006) AVERAGE EARNINGS(PY 2006) AVERAGE EARNINGS
• Definition: For those employed in the 1st, 2nd and 3rd quarter after exit, average six month earnings in the 2nd and 3rd quarter after exit
• Calculation:
Of those employed in the 1st, 2nd and 3rd quarters after exit…
Earnings from the 2nd and 3rd quarter after exit
Number who exit during the quarter
61
AVERAGE EARNINGSAVERAGE EARNINGS
Employed in 1st quarter after exit?
Employed in 2nd and 3rd quarter after exit?
NO
YES
NO
YES
Excluded
Wages from Q2 and Q3 post- exit included
YES
Were supplemental data sources used in either 1st,
2nd or 3rd quarter after exit?
NO
62
Highlights of Adult Average Earnings
Includes those employed in 1st, 2nd and 3rd quarter after exit, although only wages from the 2nd and 3rd quarters are utilized in the calculations
Only allowable data source is wage records; no supplemental data*
Those whose employment was determined based on supplemental data are excluded from the earnings measure
*Some DOL-funded programs permit the use of supplemental data as an interim means of reporting63
In Closing: Implementation is a Process
• The implementation of common measures and the reporting changes necessary to implement them represent such a challenge for states because they require in many cases nothing short of cultural change
– Basic operations must be reviewed, rethought and perhaps redeployed
• A quote from Yogi Berra comes to mind:
– The future ain’t what it used to be!
64
Remember this Website!www.doleta.gov/performancewww.doleta.gov/performance
… your One-Stop portal for program performance and reporting across programs
65