Examination Cycle Time Management

Personnel Testing Council of Southern CaliforniaLuncheon Topic, August 13, 2014

Rod Freudenberg, Ph.D.

The Selection Funnel

2

Interview

Test

Screen Apps

Recruitment

Ranking

Hire

Final SelectionVerifications

Start

Finish

Cy

cle

Tim

e

3

Exam Cycle Time:

Why does it matter?

The longer a vacancy exists the longer some of the work of the organization is not getting done or is being done by people taking on larger workloads.

Attrition of the strongest candidates because of loss of interest, pursuit of other employment options, negative image of agency.

Analyst exam overload – the exam requests keep coming so the longer it takes, more exams need to be juggled simultaneously (death spiral)

4

It’s a Process (right to left)

Client Organizational Effectiveness

Cycle TimeMinimization

Cycle TimePredictability

List Quality Maximization

List Predictability

Defined and managed process

Variability Reduction

Process Requirements

Feedback

Client Satisfaction

Process Maturity Stages (CMM)

5

Reducing cycle time requires controlling the process.

Ad-hoc

Defined and

Repeatable

Enabled and

Instituted

Measured and

Predictable

Enhancing

21

34 5

6

Outline

1. Defining the Process

2. Cycle-Time and Process Efficiency

3. Cycle-Time Analysis

4. Cycle Time Management Strategies

Defining the Process: Makes it “Repeatable”

7

Role 2

Role 1

Role N

Role 4

Process Step 1

Process Step 2

Process Step 4

Role 3

Decision Yes

Process Step 2.5

No Process Step 3

Exam Process Map

8

Research Phase

9

Client

Team Lead

Specialist

2. RESEARCH

Director’s Office

Analyst

Review exam history and

develop analysis plan

Review analysis plan

see Competency

Modeling detailed process

GA

TE

2Is th

e p

lan fo

r ana

lysis

so

un

d?

Client Conference

Competency Modeling

(see Competency Modeling detailed

process)

approved

Exam plan:Competencies, Methods and

Weights

ID SMEs for developing the

exam

exam designsummary from competency

analysis system

GA

TE

3Is th

e e

xam p

lan s

ou

nd

?

Design review

Update exam design in

tracking system

approved

Recruitment Phase

10

Exam Process

11

Client

Team Lead

Specialist

4. APPLICATION REVIEW

Director’s Office

Analyst

Recruiter

Paper Apps: Retrieve, clock-in, manual entry

into NEOGOV

Monitor Applications in

NEOGOV

Prior to Close of Filing, review

applications with client, extend

filing and revise recruitment if

needed

Consult and plan outreach, media,

and budget

CLOSE Filing

Check Bars in NEOGOV,

DQ where indicated

Bar Records

See bar sub-process

Qualify Applications in

NEOGOV(consult with Client where necessary)

Print and mail reject notice

Send DQ Emails

Requalifying information – allow 6 days via “backdoor”

Exam First Part

12

Client

Team Lead

Specialist

5. EXAMINATION, FIRST PART (e.g., T&E or Written) (See detailed process for examination parts.)

Director’s Office

Analyst

Assistant

Prepare TPI and Materials

for Packet

Schedule Location,

Dates, Raters

Prepare invitation

email/letter

Send letters

Finalize and Prepare Packets

Finalize Material

Administer Assessment(see Proctoring

detailed process)

GA

TE

5aA

re sched

ule/tech

nical ris

ks con

trolled

?

Review and verify

Send emails

Review Plan at Team Meeting

approved

TPI: Test Preparation Instructions

Scoring First Part

13

Client

Team Lead

Specialist

Director’s Office

Analyst

Assistant

6. SCORING AND PASS POINT SETTING (e.g. Written Test)

Scoring Formula in NEOGOV --

Parts and weights

Dismantle Packet.

Ensure return of all material

Scan/Enter in NEOGOV produce item analysis, sort candidate comment forms.

Review comment sheets, proctor

notes, item analysis.

Consult w/ SME on questionable

items

Scoring changes if needed

Complete Pass Point cover sheet" and recommend pass point

Set pass point

Qualify Apps if not done previously

Determine pass/fail results in Neogov

Send fail notices

Receive Protests (see

appeal process)

Other Parts

14

Prepare notice of results of first part and next

part invitations

Client

Team Lead

Specialist

Director’s Office

Analyst

Assistant

Prepare TPI and Materials

for Packet

Schedule Location,

Dates, Raters

Send letters

Print Applications.Finalize and

Prepare Packets

Finalize Material

Administer Assessment(see Proctoring

detailed process)

GA

TE

5aA

re sched

ule/tech

nical risks co

ntro

lled?

Review and verify

Send emails

Review Plan at Team Meeting

approved

7. EXAM PREPARATION - SUBSEQUENT PARTS (e.g., Interview)

Final Scoring and List

15

Client

Team Lead

Specialist

Director’s Office

Analyst

8. FINALIZING SCORING AND LIST PRODUCTION

Enter scores into NEOGOV and Produce Distribution

Update cut-off score

considerations form

Set final pass point

Calculate total score,

add seniority points,

calculate final scoreCollect Confidentiality

Forms, Note those who do not have essential

documents. Materials accounting.

Sign-offFinal Report

GA

TE

6Is th

e eligib

ility list read

y to

pro

mu

lgate?

Determine or confirm

withholds

Enter withholds in

NEOGOV Prepare final

eligiblity report

Check list:Minimum Qualifications verifiedPrior terminations/involuntary separationsArrest/Conviction status screenedRight-to-work status screenedEmployment Bars screenedDiploma, Degree, Transcripts, License, Certificate imagedScoring parts and weights verifiedPass points and rescaling verifiedComposite based on all rescaled or all percentage scoringSeniority points added for dual Veteran points added for entry-openEligibility dates verified (merged, 6 mo., 12 mo.)Selective certification designated and verified

Complete Eligibility Checklist

Promulgation and Review

16

Client

Team Lead

Specialist

Director’s Office

Analyst

Assistant

9. PROMULGATION AND REVIEW

Pull Exam Folders /Results

for Candidate Review Session

Candidate Review.Monitor Review and

provide appeals forms

Assist candidates in understanding

scoring

Receive protests (see

appeals process)

Assist Appeals

Investigation

Prepare final results notices

Send letters

Send emails

Schedule review location and

date

Prepare PC Agenda

Close-out

17

Client

Team Lead

Specialist

Director’s Office

Analyst

Assistant

10. CLOSE OUT PHASE

Electronic files

transfered to K:Exam Files

Job Analysis and Exam Plan

File

Teststats & item changes in

master copies

Update and Return Files

Competency ModelingExamination DesignConference notes

End notes

Complete Exam Close

Checklist

Close exam in tracking

system

GA

TE

7Is e

verythin

g p

rop

erly d

oc

um

en

ted an

d file

d?

Determine appeals disposition, enter in tracking system

Commissioners deny appeals?Accept List?

Yes

Corrective Action

No

Put on List and appeals on PC

Agenda

File Exam Materials.

Retain for 5 years

18

Cycle Time TargetsWhat’s the least time it can take?

BULLETIN CLOSE OF FILING

WRITTEN

PERFORMANCE

INTERVIEW

TE INTERVIEW

WRITTEN INTERVIEW

15

10

5

10

10

10

10 10

INTERVIEW

10

LIST25 days

(40)5

LIST35 days

(50)5

LIST15 days

(30)

LIST20 days

(35)5

5

CYCLE-TIME MANAGEMENT

19

20

Exam Cycle Time:

What is it?

• The amount of time in working days to complete of the process

• Broken down to key stages of the process.

21

Time is a Resource

Resource Requirement = Minimum cycle time to complete one thing TIMES the number of things in process

10 actual work days * 10 exams in process

= 100 total days to complete.

Process Efficiency = Resources Allocated / Resource Requirement

75 days available / 100 days to complete = .75

Actual Cycle time = Resource Requirement / PE

100/75 = 133 days

22

PE and Work in Process

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

PE

1.00

.75

.50

.25

.05 Exams

Given finite resources:

Cycle time and Process Efficiency

Reducing Total Cycle Time involves: Reducing the number of things in process (fewer exams,

putting exams in queue) Increasing process capacity (more staff/resources) Increase process efficiency

Increasing Process Efficiency involves: Standardization of materials and procedures Leveraging technology and automation Eliminating steps Eliminating rework Eliminating false starts Eliminating wait-states* Maybe even working harder or faster

23

24

Exam Cycle Time Measurement

Mean and Standard Deviation for each exam phase days in process and overall• Prep/Development time• Recruitment time• Exam administration time

Percentage of exams exceeding “control limits” for each phase

25

Regular Exams

Cycle Time Graph (2012-13 actual)

12/07' 12/08' 12/09' 12/10' 12/11' 12/12' 13/02' 13/03' 13/04' 13/05' 13/06'0

10

20

30

40

50

60

70

80

22

8 11

2518

25

39

26 2521

17

23

23 17

22

16

16

19

16 1518

14

16

1511

14

10

11

12

10 10 12

9

Average of ExamDays

Average of RecruitDays

Average of PrepDays

Exam PE = 2.08!Recruit PE = .83Prep M=22, SD=20!

Analysis of Variance

Structural: Exam TypesOpen, Promotional, ContinuousEntry, Professional, Management, ExecutiveComplexity (number of exam parts)

Situational: PriorityUrgent, Expedited, Regular, Special

Logistical:SME time, Raters, Facilities, etc.

Performance:Individual

26

Time/Process Analysis

Value-added: Time that is spend directly contributing to the quality of the product

Business Value-added: Time that is required by law or regulation (recruitment, notices, reviews, protests, approvals, documentation, etc.)

Non-Value-added: Everything else!

• What can be eliminated? Consolidated? Run concurrently? Automated?

• Where can you squeeze out the dead air?

27

28

Cycle Time Management

Set baseline for “intervals that matter” Mean and Standard Deviation Start-to-Open, Open-to-Close, Close-to-List

Set control limit (CL) for example: Fixed target (theoretic “ideal”) One SD >mean Top 15% outliers

Identify cases that exceed the CL Elevate attention/intervention for these Determine root-cause Get outliers back on track

Systemically address high-leverage root causes

29

Variability Reduction and Cycle Time Reduction

C ycle T im e

Statistical Process Control (SPC)

30

Cycle-Time Management Practices

Outlier Review Track projects weekly – monitor time

Interval times displayed in status reports in Exam Tracking

Escalate when exam exceeds an interval CL (Start-Open, Open-Close, Close-List)

Is there a problem? If so, how can it be addressed?

Follow through

Elevate if challenging or persistent (Director to Director)

31

Systemic Root Causes Need Systemic Solutions Transcend Space and Time:

• Virtual SME • On-line/electronic, auto-scoring T&Es• Video-based interviews and assessments• Un-proctored Internet Testing (UIT)

Establish partnering relationships at executive levels for rater-sharing (supply-chain management!)

Examination planning template –set reasonable expectations

Testing room, interview room, proctor supply and availability

32

Cycle-Time Management Practices

Consulting Skills

Identifying the “customer(s)” Scoping and prioritize the project Setting overall target dates for exam milestones Clarifying roles, responsibilities, and resources (SMEs

and raters) Asking permission to press for action Communicating status throughout project – keeping the

project visible and urgent Pressing for action (tough love)

33

Consulting Skills

Exam Start-up (triage)

Priority Options (from start-up)• Urgent: 25 days, • Expedited: 40 days• Regular: 60 days• Unclassified

Priority Implications • Faster exams require more concentrated effort• Faster exams require fewer parts• Faster exams require fewer assembled parts• Faster exams require no wait time• Impact of a poor hire versus impact of unfilled positions?

34

Consulting Skills

Key Levers Contact requesting manager within one working day of receipt of

request• Take lead in creating urgency

Put “in process” only after the exam is prioritized and timeline set

Open recruitment only after first exam part is ready. • “Ready” includes establishing availability of administration and scoring

resources for a specific date.• Raters are lined up.

Use “On-hold” status and “Suspended” status (take out of process)• Initiate status discussion with customer at second instance of delay or

unavailability• Cancel exam if in suspended status for more than 1 month. Restart when

time and resources permit

Reprioritize if customer requirements change• e.g., move from expedited to regular; regular to complex

Organization Climate and Culture

Process improvement and cycle-time management are functions of “employee engagement.”

Involves discretionary effort, creativity, and caring for the product, service, and customer

Involves shared ownership of the data • Must be shared, transparent, credible, and “safe”

Involves shared ownership of goals, problems, and successes

35