Expanding The Measures of Success

Nick Kremer, VERATACSharon Boivin, NCES

Michelle Massie, WDQC

Overview –The challenge – gauging successThe context—view from the field

California specific exampleFederal efforts to improve data collection National effort to facilitate data

matching/sharing

Workforce Accountability Chair the Vocational Education Research and

Technical Advisory Committee (VERATAC) Field based committee advisory to State

Chancellors OfficeMembership

CTE deansProject administratorsFacultyInstitutional researchers

Charge is to provide input into community college related accountability from a workforce perspective

California ContextOutcomes

Scorecard Degrees and certificates

General measure and a CTE sub-measure Transfer ready, Momentum, remedial

Salary SurferEffort to expand outcomes

Skill builders Take units but don’t complete a college credential

State licensureIndustry certification

Success –let a 1000 flowers bloom Current definitions limited to institutionally

granted degrees and certificatesWhat about

LicensesIndustry certification

View from the fieldWe are concerned about licensure and third

party certification because—

AccountabilityStudent successFeedback loop

The challenges -- licensure Effort to connect with state licensure boards

unsuccessful 82 boards under Department of Consumer

Affairs Concerns

Data security/privacyControl over dataSuspicion

What will colleges do with it?

Lack of perceived connection with collegesColleges prepare students with content

A successful example Apprenticeship is governed by California

Bureau of Apprenticeship standardsGrant Journey person status after training and

work requirements are fulfilledWillingly agreed to share information

Once I askedProcess took over 20 months

MOUTechnical

Data will be part of State Scorecard and Perkins Accountability

Industry certificationExamples: Cisco, NIMS, CompTia, Adobe, MCSC Colleges prepare students with content but testing

by othersColleges don’t know result unless student sharesStudents many times don’t complete college degree

or certificate and a classified as a drop out Matching with CompTIA not been very successful

Lack of common identifier

Industry Certification

The key is match dataCollege course takingTesting results

Finding common data field to facilitate matching is challengingSocial security numbers

Michelle will go into this in detail

Credential Data Pioneers: Forging New Partnerships to Measure Certifications and Licenses

October 31, 2014

www.workforcedqc.org

• Advocate for inclusive, aligned and market-relevant education and

workforce data that can help our nation’s human capital policies meet the

challenges of a changing economy.

• Promote federal and state reforms for data systems that provide useful

information for policymakers, students and workers, business

leaders and educators.

– State Blueprint with 13 key features of a high-quality data infrastructure

– Address federal legislation, funding and technical assistance

– Policy agenda developed by broad coalition of national organizations, state

leaders and technical experts across education/workforce spectrum

WDQC Mission

National Partners

WDQC is grateful for the support of our funders: • Apollo Education Group• Bill & Melinda Gates Foundation• The Joyce Foundation• Lumina Foundation

Funders

WDQC supports efforts to measure the full range of credentials — not just those associated with traditional education pathways. Data systems that capture all types of credentials can be used to:

– Show policymakers a fuller picture of the skilled workforce

– Help educators know whether their programs are appropriately preparing students to obtain credentials needed to advance in particular industries

– Assist students and workers in making education, career and credentialing choices

– Attract businesses seeking to expand or locate in areas with a supply of workers

Meeting the Non-Degree Credential Challenge

• These data linkages would enable richer and more complete analysis of education and training program alignment with industry requirements, as well as provide evidence on which licenses and certifications demonstrate value in the labor market over time.

• The report and webinar highlight states and schools that have taken steps to broker data sharing agreements with certification bodies and licensing agencies in order to better understand the attainment and value of selected non-degree credentials.

Meeting the Non-Degree Credential Challenge

Credential Data Pioneers: Forging New Partnerships to Measure Certifications and Licenses

To download the full report, visit:www.workforcedqc.org/resources-events

The Report

• Names• DOB (but not completely populated)• Gender (not completely populated)• Last 4 digits of SSN• Place of birth (but very limited)• If the licensee is an organization• Issue date• Expiration date• Status of license: active, pending, expired, suspended, inactive, and others• Date of status• Addresses (some are out of state)• Limited education (~1/3 of the licensees have school names, date enrolled, and limited

program/degree information)• Employment table (sparse) with job title, start and end date

Indiana Case Study

• Records date back to the late 1800s• Majority of those are for registered nurses• Documentation, definitions difficult to come by• Database may be consolidated, but there are dozens of boards

Indiana: Surprises

• 40% match rate for entire 1.5 million records (matched to wage records)• 80% match rate for records dated 2000 forward

Unmatched Metrics:• 17% were organizations and not people• 30% had a null DOB and many of these were often due to the very early licenses. For

more current integration, where there is no DOB we use additional identifiers for probabilistic matching.

• ~4% had a DOB before 1935 or after 1994 • Remaining unmatched, most (~60%) had a license that was expired, denied,

suspended, withdrawn, or inactive.

Indiana: Linking Metrics

Built dynamic universe of professional licenses/certificates/etc. by Year of Active Status (updated monthly)• Only ACTIVE license holders in each year based combination of status (active) and date

of status and within the <issue | expiration> dates• Persons only (no organizations)

Linked to wage/employer records:• Match based on each year, based on employment in at least one quarter with wages• Match to NAICS (2 digit, 3 digit) industry• Unemployment • New Hires

Indiana: The Build

• Active “professionals” by year by type of license• Average wage by type of license • NAICS industry of employment by license type • Unemployment and new hire patterns • Public higher education linkage confirms education on license

Indiana: The Results

• Complete “business rules” and meta data based on actual certification and license requirements and state forms

• Publish• http://iwis.in.gov

Indiana: Next Steps

• A voluntary effort to expand and improve data exchange between industry certification organizations and state longitudinal data systems by establishing a national data exchange clearinghouse

• Step 1: Raise awareness and gain consensus on need• Step 2: Develop use cases and conduct pilot projects• Step 3: Develop standards and guidelines on data sharing• Step 4. Establish national data exchange clearinghouse

Certification Data Exchange Project Case Study

• Report on characteristics and employment outcomes of students who took and passed exams

• Based on a recent Illinois and CompTIA project• Second Stage:

– Round 1 States: California, Kansas, Maryland, North Carolina, Ohio and Oklahoma

– Round 2 States: DC, Iowa, Kentucky and Washington

Pilot Participants

• Limited variables available in the CompTIA dataset – First name, last name, geographic code (zip)– CompTIA provided the individual-level data

• CompTIA records– 1/1/2005 to 12/31/2010– 3,412 records provided from candidates who “opted in”

• Illinois Community College Board records– All credit students from fiscal years 2006–2010 (A1)– 48 colleges with approximately 750,000 records per year

Illinois Data Matching Protocol

– Facilitated data exchange through signing an indemnification agreement with CompTIA

– ICCB reports provided additional variables– Demographics– Special populations– Major program areas– Employment and earnings– Statewide and by provider

Illinois Process

• 78 percent match achieved• Certified students had slightly higher employment rates• Certified students had substantially higher earnings

– About 1/3 higher in the exam quarter– Almost 1 ½ times higher by the 3rd post exam quarter

• Certified students had better outcomes to begin with and the gap widened with the passage of time

Illinois Results

• Partners formed Certification Data Exchange Project and Advisory Group • Created Multi-year Roadmap to develop a National

Third-Party Certification Data Exchange Clearinghouse• Execute data sharing agreements with one or both of the participating certification

organizations.• Develop a data matching protocol and conduct matching in cooperation with industry

certification partners in conformance with applicable privacy and confidentiality laws and regulations.

• Develop data tables on number of test takers and passage rates by student demographics including age, gender and race/ethnicity as well as 2-digit, 4-digit and 6-digit CIP. Develop additional tables by special populations, if available.

• Develop data tables on employment and earnings of targeted students comparing students with and without certifications.

• www.acteonline.org/certification_data

Next Phase

• Trade Assistance Act Community College Career Training Grant – emphasis on certifications and tracking students

• California Community College System – initiating a new comprehensive data collection system and dashboard

• NC Community College System – Large Data Initiative• Nation-wide interest and common data issues

Workforce Credentials Coalition Case Study

ContactMichelle Massie, Policy Analyst

michellem@workforcedqc.org

202-223-8355

New Data Sources to Measure Success

Sharon A. Boivin

Chair, Interagency Working Group on Expanded Measures of Enrollment and Attainment

(GEMEnA)

NCWE Conference

October 31, 2014

Background

• The need for better data on education, training, and credentials for work became apparent during the recession

• President’s 2009 State of the Nation address called for every American to have at least one year of education or training beyond high school

• Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) formed in fall 2009

Percent of 25 to 34 year olds with at least a high school diploma who report “some college”, by state (Source: 2011 ACS)

Alaska

Idah

o

Arizon

aUta

h

Miss

issipp

i

Wyo

ming

Arkan

sas

Nebra

ska

Kentu

cky

South

Car

olina

North

Car

olina

Louis

iana

Mon

tana

India

na

Was

hingt

on

Georg

ia

North

Dak

ota

Wisc

onsin

Main

e

Mar

yland

Illino

is

Minn

esot

a

South

Dak

ota

Penns

ylvan

ia

New Y

ork

Distric

t of C

olum

bia0

10

20

30

40

50

60

70

80

90

100

GEMEnAA collaborative effort of federal statistical agencies and policy offices 1) to develop new survey items to measure the prevalence and key characteristics of non-degree credentials:• Industry-recognized certifications• Occupational licenses• Educational certificates

and…

2) to improve federal data on enrollment and participation in education and training for work including initial on-the-job training programs such as apprenticeships and internships, non-credit college classes and skills oriented work-related training.

GEMEnA Member Organizations

• National Center for Education Statistics (NCES), Chair• Bureau of Labor Statistics• Census Bureau• Council of Economic Advisors• Department of Education, Office of the Under Secretary• National Center for Science and Engineering Statistics (NSF)• OMB Office of Statistical and Science Policy

GEMEnA Values

• Rigorous developmental work is time consuming and expensive

• Documentation contributes to the field of survey methodology, may undergird future work, and informs survey sponsors about the validity and reliability of items they may wish to adopt or adapt

• A collaborative approach to development and deployment ensures value and use

5-Year Development Process

• Expert groups helped define the credentials• Focus groups of potential respondents

provided insight into relevant words/terms• Cognitive interviews refined question wording

and instructions• Pilot studies tested survey items on a large

number of respondents

Synergy happens when:

• Theoretical definitions developed by experts are used consistently in legislation, regulation, and accountability systems

• Operational definitions developed through respondent testing are used consistently in surveys

Theoretical Definitions—Certifications and Licenses• Certification: A credential awarded by a certification body based on an

individual demonstrating through an examination process that he or she has acquired the designated knowledge, skills, and abilities to perform a specific job. The examination can be either written, oral, or performance-based. Certification is a time-limited credential that is renewed through a recertification process.

• License: A credential awarded by a government agency that constitutes legal authority to do a specific job. Licenses are based on some combination of degree or certificate attainment, certifications, assessments, or work experience; are time-limited; and must be renewed periodically.

Is it a Certification or a License?• Respondents do not easily distinguish

between certifications and licenses particularly when the certification is required for the license

• Based on development work, surveys ask respondent to indicate whether the credential was issued by a government or other kind of certifying body.

Operational Definition—Certification/License

Do you have a currently active professional certification or a state or industry license? Do not include business licenses, such as a liquor license or vending license.

A professional certification or license shows you are qualified to perform a specific job and includes things like Licensed Realtor, Certified Medical Assistant, Certified Teacher, or an IT certification.

 

Theoretical Definition—Educational Certificate

• Educational certificate: A credential awarded by an educational institution based on completion of all requirements for a program of study, including coursework and test or other performance evaluations. Certificates are typically awarded for life (like a degree). Certificates of attendance or participation in a short-term training (e.g., 1 day) are not in the definitional scope for educational certificates.

Operational Definition—Certificate

People sometimes earn certificates for completing an education or training program. A certificate is different from a certification or license. Do NOT include certifications/licenses here; include them in the previous section only. Have you ever earned any of the following types of certificates?

 

Let them tell us what they have

a. A certificate for completing a training program from an employer, employment agency, union, software or equipment manufacturer, or other training company, but NOT from a school.

b. A certificate for completing a vocational program at a high school.

c. A high school equivalency certificate, such as a GED®.

What we’re really after:

d. A certificate—rather than a degree—for completing courses from a community or technical college, or other school after high school. Do not include college degrees.

First Results

• Census Bureau released the first official statistics in January 2014

• Measuring Alternative Educational Credentials: 2012 http://www.census.gov/hhes/socdemo/education/data/files/p70-138.pdf

Data Currently Available

• 2008 Survey of Income and Program Participation (SIPP), Wave 13 topical module fielded in fall 2012

• 2002/2012 Education Longitudinal Survey 10 year follow up of 10th grade class of 2002 (12th grade class of 2004)

Data Available in the Near Future

• Redesigned SIPP (January 2015)• Baccalaureate and Beyond (February 2015)• Current Population Survey (February 2016)• National Survey of College Graduates (2016)• Beginning Postsecondary Students (2016)• Credentials for Work Survey (2017)

Website: nces.ed.gov/surveys/gemena– List of surveys with links to questionnaires and

data (as they become available)– Links to documentation on development process

and published reports– GEMEnA-developed working definitions of

certifications, licenses, and certificates– Link to LinkedIn group: GEMEnA Community

For More Information

Sharon A. Boivin, Ph.D.Chair, Interagency Working Group on Expanded

Measures of Enrollment and AttainmentNational Center for Education Statisticssharon.boivin@ed.gov