Measurable Skill Gains

Nathan CoppleyDirector of Performance

nathaniel.coppley@nccommerce.com

Allison StersicPerformance Assistant

allison.stersic@nccommerce.com

Agenda

• What are the 5 types of MSGs?

➢ What documentation is required for each type of MSG?

• How are MSGs keyed into NCWorks?

• How is performance for MSGs calculated?

➢ Who is included in the denominator?

➢ Who is included in the numerator?

• How is the Period of Participation unique for MSGs?

• Who is excluded from being counted in the measure?

• Where can I learn more about MSGs?

• What is a Measurable Skill Gain?

What to ExpectPolling Questions

• Anonymous

• Helpful to us to make sure you’re getting the information

you need

Question Box• Used to collect your questions at any point during the

presentation

• Questions will be answered and compiled into a

Frequently Asked Questions addendum to the Guidance

Polling Question #1

What is aMeasurableSkill Gain?

Definition of a Measurable Skill Gain

The percentage of participants who, during a program

year, are in an education or training program that

leads to a recognized postsecondary credential or

employment

who are achieving documented academic, technical,

occupational, or other forms of progress, towards

such a credential or employment.

AND

Measurable Skill Gain

The Measurable Skill Gains (MSG) indicator is one of six that make up the performance metrics for the WIOA Title I and Title III programs.

➢ Employment Rate – 2nd Quarter After Exit

➢ Employment Rate – 4th Quarter After Exit

➢ Median Earning – 2nd Quarter After Exit

More specifically, MSG applies

only to WIOA Title I programs:

➢ Credential Attainment

➢ Measurable Skill Gains

➢ Effectiveness in Serving Employers

▪ Adult

▪ Dislocated Worker

▪ Youth

Polling Question #2

What are the 5 types of Measurable Skill Gains?

Types of Measurable Skill Gains

1. Educational Functioning Level (EFL)

2. Secondary School Diploma / Recognized Equivalent

4. Training Milestone/ Progress Report

3. Transcript / Report Card

5. Skills Progression

Educational Functioning Level (EFL)Documented achievement of at least one educational functioning level of a

participant who is receiving instruction below the postsecondary education level.

1. Pre-Test and Post-Test Results

▪ Results from approved tests showing at least one EFL gain

▪ Pre- and Post-tests must be from the same version

2. Enrollment in Postsecondary Education or Training

▪ Participants who exit a program below the postsecondary level and

enroll in postsecondary education or training during the program

yearNote: A program below the postsecondary level applies

to participants enrolled in a basic education program.

EFL is measured in one of the following ways:

Secondary School Diploma or EquivalentDocumented attainment of a secondary school diploma or its recognized

equivalent by obtaining certification of achieving passing scores on all parts of a

State-recognized high school equivalency test.

➢ Copy of credential

➢ Copy of school record

➢ Follow-up survey from program participants

➢ Case notes documenting information obtained from education

or training provider

Accepted documentation includes one of the following:

Transcript / Report Card

▪ Must show grades of D or higher and/or be considered passing

▪ The semester must have occurred within the current program year

▪ The document must reflect that the participant is in good academic standing: nothing to

indicate that the participant dropped out of school or was removed from the institution on

academic/conduct grounds

Postsecondary Education

▪ Transcript demonstrates sufficient number of credit hours:

▪ Full-time – 12 hours for semester within current PY

▪ Part-time – a total of at least 12 hours over the course of

two completed semesters during a 12 month period)

▪ If semester crosses a PY, the MSG would be keyed in

the PY in which the semester ends

Secondary or postsecondary transcript or report card for a sufficient number

of credit hours that shows a participant is meeting the State unit’s academic

standards as follows:

Secondary Education

▪ Documented each

semester.

▪ Specific to youth

attending high school.

Training Milestone / Progress ReportSatisfactory or better progress report towards established milestones from an

employer or training provider who is providing training. Progress reports must

document substantive skill development that the participant has achieved.

➢ Contract and/or evaluation from employer or training provider documenting a skill gain,

or training reports on milestones completed as the individual masters the required job

skills. Increases in pay resulting from newly acquired skills or increased performance

also can be used to document progress if provided as an evaluation from employer.

Acceptable documentation includes:

➢ Documentation of a skill gained (or completed steps) through OJT or Registered

Apprenticeship. Completed steps may be a mid-point evaluation, final evaluation, or

exam results as required by Registered Apprenticeship program.

➢ Progress report from employer documenting a skill gain that the participant has

achieved.

Skills ProgressionSuccessful passage of an exam that is required for a particular occupation or,

progress in attaining technical or occupations skills as evidenced by trade-related

benchmarks, such as a knowledge-based exams.

➢ Results of knowledge-based exam or certification of completion, including a component

exam of a Registered Apprenticeship program

Acceptable documentation includes:

➢ Documentation demonstrating progress in attaining technical or occupational skills

➢ Documentation from training provider or employer such as a satisfactory attainment of an

element on an industry or occupational competency-based assessment

➢ Copy of credential that is required for a particular occupation and only is earned after the

passage of an exam. Examples include: Class A Commercial Driver’s License, Certified

Nursing Assistance License, or CompTIA A+ Certification

Polling Question #3

How are MSGs keyedinto NCWorks?

How is performance for MSGs calculated?

MSG Percent Achieved Calculation

The Measurable Skill Gains indicator is calculated by dividing the

number of participants who attain at least one type of gain during

each period of participation within a given program year (A, below)

by the total participants who are in an education or training program

that leads to a recognized postsecondary credential or employment

(B, below).

MSG

% Achieved

A

B

Denominator Inclusion

B

Participants who, during any point in the program year, are in an education or training program

that leads to a recognized postsecondary credential or employment are included in the

denominator. This number includes participants who do not exit the program and continue to

receive services beyond the end of the program year, as well as those who have exited the

program by the end of the program year.

Title I Adult and Dislocated Worker

• All participants who are in a Title I Adult- or

Dislocated Worker-funded training

program

• Training programs for a secondary school

program equivalent

• Work-based training

Title I Youth

• All ISY (in school youth) are included

• OSY (out of school youth) in the following are

included:

o Occupational skills training

o Secondary education at or above 9th grade level

o Postsecondary education

o Title II-funded adult education at or above the 9th

grade level

o YouthBuild program participants

o Job Corps participants

Title I Adult and Dislocated Worker: All participants who are in a Title I Adult or Dislocated Worker-funded training

program are included in the Measurable Skill Gains indicator (which includes

funding a training program for a secondary school program equivalent). This

includes all participants in work-based training.

Denominator Inclusion Based

on Service/Activity Codes

300 – Occupational Skills Training – on ETPL

301 – OJT Training (Not WIOA Youth)

302 – Entrepreneurial Training

304 – Customized Training

310 – Occupational Skills Training for Special Grants

314 – Registered Apprenticeship

328 – Occupational skills Training – Not on ETPL

333* – TAA – Approved Remedial Training (for those

with GED/HS Diploma)

335* – TAA – Approved Occupational Skills Training

– Approved by State

339* – TAA – Approved GED Training

222* – TAA – English as a Second Language (ESL)

*denotes TAA-only

Service codes that will include Adult and Dislocated Worker in the denominator:

Out School Youth:Only Out of School Youth (OSY) who are in one of the following are included in the

denominator:

407 – Alternative Secondary School services or dropout recovery services

416 – Youth Occupational Skills Training - on ETPL

424 – NC Registered Apprenticeship Training – Youth

429 – Enrolled in Traditional Secondary School (H.S.)

430 – Youth Occupational Skills Training - Not on ETPL

441 – Entrepreneurial Skills Training

Denominator Inclusion Based

on Service/Activity CodesIn School Youth:All In School Youth (ISY) are included in the Measurable Skill Gains indicator since

they are attending secondary or postsecondary school.

Numerator InclusionThe numerator is the number of program participants who achieved at least one type of measurable skill gain.A

1. Educational Functioning Level (EFL)

2. Secondary School Diploma / Recognized Equivalent

3. Transcript / Report Card

4. Training Milestone/ Progress Report

5. Skills Progression

Note:

▪ A participant may have achieved more than one type of gain in a reporting period, but only one gain per participant in a reporting period may be used to calculate success for the measure.

▪ However, all MSGs should still be recorded regardless to ensure success for the participant in the indicator calculation

How is the periodof participationunique for MSG?

Periods of Participation

So what does that mean? Two key factors:

1. Each unique program entry date (not exit date) triggers inclusion

in the calculation.

Unlike the other WIOA performance measures…

MSG is not an exit-based measure!

➢ However, if a participant has 2 periods of participation within 1 PY,

they could appear in the denominator twice and therefore could

receive 2 potential successful MSGs*

*A participant is included in the denominator more than one time during a PY if they exit the

program and are subsequently reenrolled in a WIOA Title I program later in the same program

year AND they participate in an education or training program during each enrollment.

Periods of Participation

2. The MSG indicator is a PY-to-PY measure

➢ One MSG outcome can be achieved in each continuing program year that

a participant is active

➢ If a participant was enrolled in an education or training program during PY

2019 and continued to receive services into PY 2020, that individual could

receive an MSG in PY 2019 AND an MSG in PY 2020.

Periods of Participation

▪ A participant can achieve a Measurable Skill Gain while still participating

in a program

▪ If you forgot to key an MSG at the time…

➢ a Successful Measurable Skill Gain may still be keyed in NCWorks

after the participant has exited the program as long as it is before

the end of the same PY in which they exited. (see FAQ slide)

▪ Key all MSGs

Reminders:

Periods of Participation – An ExampleMSG Opportunity #1

in Pink

▪ Abby’s 1st period of participation only occurs within PY18 and represents MSG Opportunity #1.

MSG Opportunity #2

in Blue

MSG Opportunity #3

in Tan

▪ Later in PY18, she reenrolls for her second period of participation (MSG Opportunity #2).

Explanation:

▪ That period of participation then crosses over into PY19 (MSG Opportunity #3).

▪ This second period of participation results in two inclusions in the denominator because it crossed over

from one Program Year to the next.

▪ Therefore, Abby will be in the MSG denominator two times for PY18 and one time for PY19.

Who is excluded fromthe MSG measure?

Exclusions

a. Institutionalized

Under what circumstances can a participant be excluded

from performance?

b. Health/Medical

c. Deceased

d. Reserve Forces called to Active Duty

e. Foster Care (for Youth participants only)

Where can I learnmore about MSGs?

Additional Resources

▪ Workforce GPS – Measurable Skill Gains E-Module https://tinyurl.com/yxvt9kpj

▪ Interactive Timing Chart by Future Works: A visual tool for understanding the

relationships between the exiting cohort, when they are being measured, and

when they are being reported for each indicator. https://tinyurl.com/y2fqgcrm

▪ TEGL 10-16 Change 1 - Performance Accountability Guidance for Workforce

Innovation and Opportunity Act (WIOA) Title I, Title II, Title III, and Title IV Core

Programs;

▪ TEGL 23-19 – Guidance for Validating Required Performance Data Submitted by

Workforce Programs;

▪ TEGL 7-18 Guidance for Validating Jointly Required Performance Data Submitted

under the Workforce Innovation and Opportunity Act (WIOA)

Attachments to MSG Guidance

1. Guide to Entering MSGs in NCWorks

Staff Online Resources

Attachments to MSG Guidance

2. MSG Desk Reference

Staff Online Resources

FAQ – Submitted so far…Q1: What happens if my participant exited and I forgot to key an MSG?

A1: The MSG can still be keyed as long as it is still in the same program year. For example, if the

participant exited March 30th, 2020 and you realize on June 10th, 2020 that they in fact achieved an MSG

on February 15th, 2020, as long as you have the proper documentation, that MSG can still be keyed to that

participant, even though they are already exited. This is only true within that program year, however. It

can NEVER extend beyond the same program year in which the participant exited.

Q2: Is there a set protocol on what case managers are to do when someone on their caseload falls into

one of the exclusions?

A2: Global exclusions should only be utilized when there are circumstances beyond the control of both the

participant and the program that are expected to last for an undetermined period of time beyond 90 days.

When this occurs, a case note should be entered detailing the circumstance and documentation to support

should be obtained, as applicable.

We urge you to follow any local policies that may be applicable and if you have questions about a

particular circumstance, to coordinate with your superuser and potentially your Regional Analyst, if

necessary.

FAQ – Submitted so far…Q3: I work with out of school youth. When someone passes 1 or 2 of the required 5 HiSET to attain their

High School Equivalency Diploma, can I enter that as an MSG?

Q4: Can we use attaining a Career Readiness Certificate for an MSG? If so, what would be selected for

that since CRC is not specifically listed as an option?

A3: No, it can only be entered as an MSG if all elements of the High School Equivalency have been

completed. In this example, all 5 HiSET component tests would need to be successfully completed to

count as an MSG.

A4: Possibly. If the CRC is a required step in the individual’s training or a normal prerequisite to

employment in a particular industry, then it can be entered as a Skills Progression MSG since the

participant would be passing a test to achieve the CRC.

FAQ – Submitted so far…Q5: If a participant is in a program for more than two program years, can I enter all the MSGs at the end

and have them count toward the first program year as well?

A5: NO!!! All MSGs must be entered within the Program Year in which they occur. We would advise you

to key them as soon as you have the necessary documentation. Waiting could result in you forgetting to

key the MSG and it not counting towards positive performance.

Q6: Can work experience be used as an MSG? Does this vary by program?

Q7: Does OJT count as an MSG and if so, does that apply to all programs?

A6: No, an internship or other work experience is not considered education or training and therefore does

not put the participant in the denominator of the MSG measure. This does not vary by program.

A7: For the Adult and Dislocated Worker programs, OJT counts as an MSG.

However, for Youth, because Youth OJT is considered a work experience and not training, Youth OJT

does not count as an MSG.

FAQ – Submitted so far…

Q8: Will this recorded webinar be available for later

viewing on the Training Center site?

A8: YES! It will be available on the Training Center

website a week or so after the webinar.

Polling Question #4

Thank You

Nathan CoppleyDirector of Performance

nathaniel.coppley@nccommerce.com

Allison StersicPerformance Assistant

allison.stersic@nccommerce.com