Key Performance Indicators for

Private Career Colleges September 29, 2015

Quality and Partnerships Unit Private Career Colleges Branch Postsecondary Education Division

Ministry of Training, Colleges and Universities

Presentation to Career Colleges Ontario

Purpose

Describe the tools and methodology used to calculate Key Performance Indicators (KPIs) for private career colleges (PCCs).

Summarize how to navigate the Forum Portal in order to upload your files (www.pcckpi.ca).

Provide key tips on the data reporting process for new and repeat

participants, including on: Populating data files; Submitting error-free files; Increasing survey response rates; and Completing your audit.

2

3

Defining KPIs

Following in-depth consultation with students, PCCs and other PCC stakeholders, the ministry developed the following five KPIs for PCCs :

These indicators are seen to be in good alignment with those collected in the public college sector and will provide all stakeholders – students, PCCs and government – useful information to aid in decision making.

Throughout the project, the ministry attempted to align the PCC KPIs, and associated tools and methodology required for their calculation, with those used in the public college sector.

PCC KPIs Public College KPIs Graduation Rate Graduation Rate

Employment Rate Employment Rate

Graduate Satisfaction with Training Graduate Satisfaction with Training

Graduate Employment Rate in the Field of Study (GEFS)

Student Satisfaction

Employer Satisfaction with Training Employer Satisfaction with Training

4

Phased Implementation Plan The first phase of KPI implementation commenced on July 12, 2013, with initial focus on PCCs

approved to accept OSAP-funded students. All five KPIs will be calculated for OSAP-approved PCCs in 2014, with implementation for all PCCs in the sector in 2015.

OSAP Default Rates will continue to be calculated for all OSAP-approved PCCs.

Phases of KPI Reporting

Rationale Phase 1 (2013)

Phase 2 (2014)

Phase 3 (2015)

PCC Type:

OSAP-approved

Mandatory: (3/5 KPIs) 1. Graduation Rate 2. Employment Rate 3. GEFS

Mandatory: All KPIs Adding Graduate Satisfaction and Employer Satisfaction

Mandatory: All KPIs

• Basic KPI-reporting infrastructure in place from before 2006 moratorium. • Provides ministry the ability to leverage learn from initial implementation issues. • Promotes accountability for government funding.

All Other PCCs

No Reporting

(preparing student files)

No Reporting (preparing student

files)

Mandatory: All KPIs

• Ability to leverage lessons learned from Phase 1. • Potential to align majority of PCCs with other ministry systems (e.g., new registration system) at point of Phase 2.

Increased Quality Assurance & Accountability

Some Key Terms

Entrants are individuals who enrolled in an approved vocational program provided by a PCC.

Graduates are individuals who completed all of the necessary requirements for an approved vocational program, including all academic and practical requirements.

An Enrolment File is a data (CVS) file used to report to Forum all entrants for one of the three defined Enrolment Terms.

A Graduate File is a data (CVS) file used to report to Forum

all graduates for one of the three defined Graduate Terms.

5

Data Uploads Data on all entrants and graduates must be collected and uploaded to Forum

three times per year corresponding to three defined enrolment and graduate terms.

Each of the three data upload windows is open for two months to provide schools with enough time to complete their upload for each term.

6

Term Upload Window Entrants and Graduates Included

Winter July 1 – August 31, 2015 January 1 – April 30, 2015

(plus historical data for first time reporters)

Summer October 1 – November 30, 2015 May 1 – August 31, 2015

Fall February 1 – March 31, 2016 September 1 – December 31, 2015

7

Graduate Outcomes Survey

Forum Research In. (Forum) uses the graduate information provided by PCCs to conduct a graduate outcomes survey starting six months after graduation. This survey collects both graduate outcomes (employment) and graduate satisfaction information.

One survey window follows each of the three data upload periods.

Forum attempts to survey as close to the six month mark as possible).The graduate survey is conducted by telephone in English or French.

The survey consists of standardized questions regarding a specific

reference week falling six months after graduation. Survey questions are aligned with those used in the public college sector. The survey reference week is the first week following the six month period after

the individual graduated.

8

Employer Satisfaction Survey

Forum derives the employer satisfaction survey population from respondents to the graduate outcomes survey who consented to their employers being contacted.

The employer survey collects information on the

educational preparedness of PCC graduates. One employer survey window follows each of the

graduate outcomes survey windows.

Auditing

PCCs’ enrolment and graduate data must be audited according to the ministry’s KPI Audit Guideline, located at www.pcckpi.ca.

Auditors are to examine an appropriate sample of the enrolment and graduate data files to provide the ministry with a level of assurance that the information included in the enrolment and graduate data files was not misstated and correctly matches the records held by the PCC.

PCCs must submit their KPI audit report by September 1st of the year after the collection cycle started (e.g., September 1st, 2016 for the 2015 collection cycle). Note: For OSAP-approved PCCs that participated in the 2014 collection cycle, any

2014 data files that have not yet been audited must still be audited by the PCC’s 2015 renewal of registration deadline, as per the former audit requirements.

In the final audit report, PCC auditors are required to provide a list of the enrolment and graduate files audited as a schedule to their final audit report, specifying the file reference numbers, original versus revised files and necessary corrections.

9

Payments/Invoices

The ministry has developed a streamlined billing process that allows for a one-time billing at the end of each KPI collection cycle, based on the number of completed graduate outcomes and employer satisfaction surveys associated with each PCC.

PCCs should expect to receive an invoice from the ministry related to KPI services in the winter of the year following the start of the collection cycle.

For example, for the 2015 collection cycle, PCCs should receive their KPI invoice in winter 2016/17.

PCCs should assume that they will pay approximately $20.00 - $30.00 for each completed graduate outcomes and employer satisfaction survey.

This will include a unit cost ($13.55 for graduate survey; $9.85 for employer survey) as well as a fixed cost allocation which is shared across all PCCs, where their graduates/employers completed the survey.

The fixed cost allocation is associated with calculating the final KPIs and producing the preliminary/final reports.

10

Steps to Uploading Data

11

Log in to Forum’s Portal

Verify your

Account Users

Prepare your Data Files

Upload your Files

Auditing your Files

1. Logging In

Your Primary User is the same person as your “Head Office Contact” as specified in RICC. This individual must log in to Forum’s portal at https://pcckpi.ca/login and create Secondary Users for each of your institution's campuses.

If you are a Primary User logging into the Forum Portal for the first time here are the next steps: Your login information – portal link, username, temporary password – will be sent

to the “Head Office Contact” email address included in the ministry’s Registration Information for Career Colleges (RICC) system. Your username will be your six-digit RICC Organization ID which starts with a “2”. If your Head Office Contact information is outdated, please update it in RICC and provide

it to Forum at pcckpi@forumresearch.com. Once your Primary User logs in with the temporary password, he/she will be

asked to change it, after which the temporary password will no longer work. You can change the password at any time after initial login. If you no longer have access to the email with your login information, please use the reset

password link: https://reporting.forumresearch.com/PCC/ResetPassword.aspx.

12

2. Verifying your Account Users

After successfully logging into the Forum portal, you must verify the name and email of your PCC’s Primary Account User: the person who will oversee the KPI reporting process, your secondary users and their campus reporting.

You must also assign Secondary Users: one individual at

each campus who is responsible for its enrolment/graduate file uploads. You can modify or delete a Secondary User at any time. Your Primary User can also be one of the secondary users if he/she will

also be responsible for uploading data files at specific campuses.

13

3. Preparing Your Data Files

Enrolment and graduate information is provided to Forum in two separate data files – one enrolment file and one graduate file – for each of the three upload windows. Therefore, each PCC provides Forum with six files per KPI collection cycle (as well as historical files for first time reporters).

Visit www.pcckpi.ca for current versions of the file templates you must use to report your entrants and graduates.

Once logged in, you can view the list of files that must be prepared and uploaded to Forum for each of the three upload windows during the 2015 Collection Cycle.

First time KPI reporters must also upload historical data – two files for each term going back 200% of your longest vocational program’s duration. For example, if your longest program is one year, you must report on all terms going back two

years. NEW: The portal will now indicate the date you must go back to. For the enrolment data file: remove any individuals who withdrew from their

program within the grace period (25% of the program to a maximum of 8 weeks). It is important that you upload your files in chronologic order as a graduate will only

be accepted into the system if he/she can be matched to a previously submitted entrant.

If you do not have any entrants or graduates for a term, you will still be required to submit a null – or “blank” – file.

14

4. Uploading Your Files

Once you have prepared your enrolment and graduate files for a specific term (e.g., Winter), you must upload them one at a time.

Each time you upload a file, you will have to provide the email addresses for your auditor and Key Executive (e.g., President, CEO, Owner, or Director).

After a file is successfully uploaded, your auditor and Key Executive will receive an email, providing them with a link to view the file.

When you submit a file that has an error, please read the error message carefully as it will explain how to fix the error in your file. You can send your error reference code to Forum if you don’t understand what it

means.

Be sure to leave adequate time for your auditor to review your uploaded files. Engage them early so they can schedule your audit it on their calendar.

15

5. Auditing – Fixing Errors

Any errors detected by the PCC auditor in the data files are to be identified by the auditor to the PCC and immediately corrected by the PCC as follows: 1. The PCC auditor is to identify to the PCC the changes required to correct errors in

the graduate and enrolment data files; 2. The PCC is then to make the changes required to correct the errors and re-submit

these files to Forum, generating new file reference numbers for the revised files; and 3. The PCC auditor is to verify that all errors have been corrected in the new file(s).

A list of the enrolment and graduate files audited, specifying “original” versus “revised” and all necessary corrections, is a requirement of the final audit report, provided by the PCC auditor to the PCC and uploaded to Forum. The PCC auditor must ensure that the revised files contain all of the required corrections.

Graduation rates are calculated based on the audited enrolment and graduate data reported by PCCs to Forum. The remaining four KPIs are calculated from the results of the Graduate Outcomes and Employer Satisfaction Surveys.

16

Now for some key tips…

17

18

Key Tips – Forum’s Portal How do I get my log in credentials?

A temporary password will be emailed to your institution's Head Office Contact, as provided in RICC. How do I log into the PCC User Management page?

Visit https://www.pcckpi.ca/ click on PCC KPI Login, which is located in the menu on the left. When logging into the PCC User Management page, you will use the Organization ID assigned in RICC as your user

name (six numbers beginning with “2”). Can my Primary User upload data files for each of my campuses?

Each campus must be attached to a unique Secondary User account (email address). It is possible to use the same account (email address) for multiple campuses, but you would need to have a different

password for each campus. What do I do after creating my Secondary User accounts?

Make sure you assign the appropriate secondary user to each of your campuses from the drop down menu provided on the portal.

Record the temporary password that is shown on the PCC management page after creating each Secondary User account. Provide this password to your Secondary Users as they will need it to log in for the first time.

How do I log in using my Secondary User account? Return to www.pcckpi.ca and instead of using the institution's Organization ID, you will use the email address that was

used to set up the Secondary User account and the temporary password provided. Where can I download my “Approved Program List”?

The list can be downloaded from the main menu on the submissions page. It’s recommended that you copy and paste program names and details from the Approved Program List when

populating the submission files in order to avoid naming errors. Can I develop my own enrolment and graduate file templates?

In order to reduce errors, we recommended that you make copies of our templates and not try to recreate your own. Templates are available under “Important 2015 Documents” at www.pcckpi.ca.

19

Key Tips – Forum’s Portal How do I name my Excel data files?

In order to upload your files, their names must not begin with a number and must not contain any spaces. “Enrolment2015w” is correct. “2015 Enrolment w” is incorrect.

Are all fields (columns) in the file required? NO, Please refer to the Operation Procedure for the required fields (pgs. 26-31). Some fields are mandatory

while others are optional. Can I enter Forum’s email or my Inspector’s email in the auditor field?

NO, the auditors e-mail should not be associated with Forum or the ministry. What if I don’t have a KPI auditor at the time of my first login?

If you have not yet acquired an auditor to conduct the KPI audit, please enter your Head Office Contact’s email address.

Can I edit the auditor email at a later date? Yes, in the “submissions page” you can edit your auditor’s email for the given data file upload. Remember to update your auditor’s email information once you have acquired an auditor or if your auditor

changes. You can do this via the PCC Management Page. What if my “Approved Program List”, as shown on the portal, is not up to date?

Please contact your inspector and make sure your programs are approved in RICC as they are being provided. Request from Forum, a copy of the Program Change Template, and after populating, email a copy to the

helpdesk pcckpi@forumresearch.com. What if there are no enrolments or graduates in a term, do we have to submit a file?

Yes, if there are no students for a given term, a NULL/BLANK file must be uploaded. IMPORTANT: If you are submitting a null file, when probed, select the box labelled “there are no records to

submit for this period”. What if I can’t interpret the error message after submitting a file?

Email the error reference code to the helpdesk at pcckpi@forumresearch.com.

20

Key Tips – Compiling your Data What if a program does not have an OSAP code?

Include “XXXX” What if my PCC does not maintain student identification numbers?

If you do not already have a unique student identification number assigned to each of your students, you are required to assign them prior to your first data upload. Each student ID number must be 15 characters or less.

This number must follow the student from enrolment to graduation. What if a student has more than one funding source?

If the student received OSAP, enter Y in G.1. If the student received non-OSAP funding, enter the primary non-OSAP funding source in G.2

What if a student transfers from one vocational program into another? Transfers between approved vocational programs are allowed at any time. Include Y in the M.1 column and

the student’s former program in M.2. Do I need to include a student’s date of birth?

YES. Date of birth is used to validate graduates on the phone. Only results from validated graduates will contribute to your KPIs.

What is Section J (PCC program information) in the enrolment data file? Section J allows a PCC to include its own terminology (e.g., program name, code, faculty, location code). If

you don’t have this information, you can leave these fields blank. What if a program is delivered via distance education (not FT/PT)?

Include “CORR” if your program is approved to be delivered via DE. Do I need to include employer information in the graduate file?

NO. Section L is an optional field. This information may, however, increase the employer survey response rate.

21

Key Tips – Successful Uploading Populate your files in advance. As individuals enrol in/graduate from your programs, update your

data files. Do not wait until the last minute to upload files. If there are problems, you will have more time

to resolve them before the deadline. Ensure your key contact information is updated in RICC so that you receive all ministry/Forum

updates. Ensure the information uploaded matches what is on RICC and update your information in RICC if

need be. Review Forum’s FAQs before you contact Forum or the ministry. Most likely your answer is

already provided. Read Appendix D and E of the KPI Operating Procedure to ensure you have the proper coding for

each data entry field. Make sure that you include an accurate telephone number and date of birth and for each of

your graduates. The more graduates Forum can contact, the higher your response rate will be. Date of birth is used to

validate graduates on the phone. Update your Hard-to-Find Graduate lists on the Forum portal (www.pcckpi.ca). If possible, inform your students/graduates and any employers with whom you have a relationship

with that they will be contacted by Forum six to 12 months after graduation to complete a graduate outcomes or employer satisfaction survey and encourage them to do so.

Screenshot your error messages if you are having problems uploading your files. If you are calling Forum and leaving a message, please remember to leave contact information.

Contact Us Questions about data files, the portal, file uploads: Forum Research Inc. Email: PCCKPI@forumresearch.com Telephone: 1-866-206-7660.

Policy-related questions: Ministry of Training, Colleges and Universities (New) Email: PCCKPI-Policy@ontario.ca

22