How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 1
How to Build the Ideal Nurse Hiring Assessment
Friday, March 23, 2012, 9:00 AM & 10:00 AMSession Number: CS-3
Presenters:
James Ostmann Sr, RN, MBA, Chief Nursing Officer, Prophecy Healthcare, High Point, NC
Dan Biddle, PhD, President CEO, Biddle Consulting Group, Inc., Folsom, CA
Prophecy Healthcare, Inc.A leading provider of healthcare assessments: over 1.5 million exams administeredStandardized competency testing focuses on screening skill levels pre- and post-hireRigorous and comprehensive validation programOver 100 nursing, allied, and home healthcare job knowledge competency exams Over 50 self-assessment clinical-skills checklists
Biddle Consulting Group, Inc.Leading U.S. EEO consulting firmNationally-recognized experts in test development and validationExperience in 200+ state and federal cases involving EEO compliance & testing
PeopleCluesInternational test publishing firm that provides modern & validated behavioral
assessments Online platform specifically designed for the commercial marketAssessments used by thousands of companies to hire, train, and promote
Prophecy Assessment Contributors
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 2
The Climate…• In June 2011, Wanted Analytics reported that employers and
staffing agencies posted more than 121,000 new job ads for Registered Nurses
• Typical annual turnover rates for RNs average 14% (KPMG survey)
• Bureau of Labor Statistics reports “the health care industry added 428,000 jobs throughout the 18-month recession from December 2007 until June 2009, and has continued to grow at a steady ratesince the end of the recession.”
• Hospitals, long-term care facilities, and other ambulatory care settings added 37,000 new jobs in March 2011, the biggest monthly increase recorded by any employment sector.
• The BLS confirmed that 283,000 jobs have been added in the healthcare sector within the last year.
The Question…• With such high growth rates expected, how do
healthcare institutions hire the most qualified “best fitting” nurses for their openings?
• What knowledges, skills, abilities, and personal characteristics (KSAPCs) should hiring authorities pay most attention to when building a robust hiring system for nurses?
• Finally, how can these “key KSAPCs” be accurately measured in the shortest testing session possible?
The Presentation• We’ll review the answers we found to these
questions while working with over 900 professionals, including: – 492 nurse staff from Saint Francis Medical
Center (470 nurses and 22 nurse supervisors)– 384 nurse staff from Frederick Memorial
Hospital (367 nurses and 17 nurse supervisors)– 13 test development professionals and
industrial-organizational psychology consultants– 30 film professionals
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 3
While Reviewing our Process…• Keep in mind… we want you to replicate,
borrow, steal, or just learn from our process• We don’t claim to have the “best” process,
but we have uncovered some of the key traits to target in your testing program
• We’ve also learned how your hiring program might benefit from using different tests in different ways for hiring nurses
Our testing program resulted in a 3-pronged assessment strategy called “Prophecy”Our goal was to create a stronger alignment between the nurse job requirements and nurse testing programs
Nurse Job Performance:
What makes nurses perform they way
they do on the job?Smarts? Drive?
Interpersonal skills?
• A typical nurse hiring process measures only limited clinical background (sometimes through a written test) and interpersonal competence (through interviews)
• Prophecy measures cognitive, interpersonal, and behavioral– Intentionally – Thoroughly– With limited overlap
Job Job PerformancePerformance
Cognitive Cognitive AbilityAbility
What Makes Up Nurse Job Performance?
Experience/Experience/BackgroundBackground
InterpersonalInterpersonalSkillsSkills
PersonalityPersonalityFactorsFactors
Clinical JobClinical JobKnowledgeKnowledge
How should they be
weighted?
But how should test skills be
tested?
Which skills are overlapping?
We tried to answer these questions while developing
Prophecy…
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 4
2008-2011 Validation Process
Prophecy Overview
Project Phase 1: Job Performance
Project Phase 2: Situational
Project Phase 3: Clinical
Project Phase 4: Behavioral
Project Phase 5: Combining the Assessments
The Prophecy Development Process
Prophecy is the aggregation of three assessment tools. The three pieces work together to form the most comprehensive employment predictor for the nursing field.
Twelve Clinical Assessments were included in the study, each having 30-70 items measuring clinical specialty job knowledge (e.g., Medical-Surgical, Emergency Care, etc.)
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 5
The Situational Assessment included 21 video-based situational judgment scenarios designed to measure the applicants interpersonal competence using a variety of complex situations that emerge in hospital settings.
A 70+ item behavioral/personality test bank was included in the study. Twenty (20) of the most predictive items were distilled into a “Nurse Effectiveness Scale” which demonstrated strong validity results.
Prophecy OverviewTypical Process:
Cognitive Ability
TypicalProcess:
InterpersonalSkills
HiringProcess
Comparison
Nurse Job PerformanceNurse Job Performance
4-6%Result = Job PerformancePredicted
16%Result = Job Performance Predicted
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 6
A job analysis process was used to distill 19 job performance dimensions that met the following criteria:
• Observable aspects of job performance that were likely to relate to traits measured by the Situational Assessment;
• Represented qualifications that were needed on the first day of the job;
• Not geographically limiting;• Not likely change over time; and • Differentiated “best performers” in the nursing profession. • Emphasis on soft skills
Each performance dimension was phrased in “observable aspects of job performance”
Project Phase 1: Job Performance
Project Phase 1: Job PerformanceJob Performance Rating Survey Dimensions
Honest & Conscientious Problem Solving
Verbal Communication Patient Care (calm & competent)
Administering Medications Report Transitioning
Assertiveness Continuous Observation
Change Adaptation Developing Patient Relationships
Conflict Resolution Patient Care Plan Management
Accountability Patient Customer Service
Critical Thinking New Technique Application
Multitasking Delegating/Managing Patient Care
Following Clinician Instructions
Project Phase 1: Job Performance
• Both Hospitals participated in the nurse rating process:– Saint Francis Medical Center:
• 470 nurses were rated by • 22 nurse supervisors
– Frederick Memorial Hospital:• 367 nurses rated by• 17 nurse supervisors
• A total of 888 nurses were rated on 19 dimensions of job performance by 39 supervisors
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 7
Project Phase 2: Situational Assessment Build
• “Situational Judgment Tests” (SJTs) pose hypothetical situations to applicants in a job-related context and then provides several plausible alternatives on the most effective and least effective way to handle the situation
• “Most effective” and “least effective” keys were determined using panel of 50 nurse supervisors
• Why SJTs?
– SJTs measure valuable skills that are typically not measured in hiring processes, or are not measured very accurately or systematically.
– Applicants perceive context-rich tests as more job relevant
– SJTs are scored consistently and reliably, compared to when the same skills are measured using interviews
– Because they are based on “consensus scoring,” they are more objective than interviews+
Project Phase 2: Situational Assessment BuildStep 1: Job analysis research was conducted at four medical/employment
centers and 19 of 169 key KSAPCs were selected
Step 2: Developed SJT Scenarios (based on 19 key competencies)
Step 3: Refined SJT Scenarios & Converted to 50-Page Script
Step 4: Staged Sets & Filming
Step 5: Multi-point Keying Process was developed based on level of expert consensus
Step 6: Validation Study was conducted by correlating test scores to 19 job performance dimensions
Saint Francis Medical Center: 27 Nurse Supervisors rated 191 nurses
Frederick Memorial Hospital: 17 Nurse Supervisors rated 235 nurses
Criterion-related Validity: Demonstrated by empirical data showing that the selection procedure is predictive of, or significantly correlated with, important elements of work behavior
21
Criterion-Related Study
010203040506070
0 20 40 60 80 100
Test Score
Perf
orm
ance
Mea
sure
Job Performance Rating Test Score
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 8
Interpreting Criterion-related Validity
U.S. Department of Labor Guidelines for Interpreting Validity Coefficients
Coefficient Value InterpretationPercentage of Job
Performance Explained(Coefficient Squared)
Above .35 Very beneficial 12.3%+
.21 - .35 Likely to be useful 4.4% - 12.3%
.11 - .20 Depends on circumstances 1.2% - 4%
Below .11 Unlikely to be useful <1.2%
Source: U.S. Department of Labor (2000), Testing and Assessment: An Employer’s Guide to Good Practices (p. 3-10).
Project Phase 2: Situational Assessment Build
How well does the Situational AssessmentPredict Job Performance?
Theoretical Expectancy Bands
Band Low Score
High Score
Cumulative % of Nurses Scoring in Band
Likelihood of Receiving “Above Average” Supervisor
Ratings
FAIL 0 69 <5% <42%E 70 82 20% 43%D 83 86 40% 54%C 87 90 60% 60%B 91 94 80% 67%A 95 109 100% 75%
Note: The “base rate” for the expectancy table was set at 60% because this portion of the distribution received ratings >=5, indicating they were rated as the median (or higher) of job performance. The theoretical (corrected) validity used for this analysis as .30 (set roughly between the two
corrected validities observed in the study).
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 9
Assuming 60% of the nurse applicants “show up qualified” (i.e.,would receive average JP ratings) and the Situational Assessment is used to select highest scoring 20%...
• 67% of this “top 20% group” will exceed performance expectations on the job (opposed to only 60% by using no assessment)
• This is a 7% improvement in overall workforce qualifications (raised from 60% to 67%)
• In terms of people, say for instance your hospital hires 100 nurses per year, 7 additional nurses would be
expected to exceed performance expectations (be successful on the job) than without using the assessment.
Situational Assessments ValidityResults Interpretation
Project Phase 3: Clinical Assessment Build
• The Clinical Assessments are designed to measure critical job knowledge domains that are necessary for clinical practice areas (e.g., Medical-Surgical, Emergency, etc.)
• They are developed and validated using a content validation methodology, which means:
– Test content is mapped to a job analysis for the relevant practice area
– Tests are balanced with respect to the questions measuring the specific aspects of each practice area
– Cutoffs are established using a “Modified Angoff” process, where nurse experts set the cutoff score by evaluating the difficulty of each item on the test
Project Phase 3: Clinical Assessment Build• The Clinical Assessments are “Mastery Based” or “Criterion-
Referenced” tests. This means:
– Designed to distinguish between who “can” vs. who “cannot” perform the job
– Test items focus on baseline competency skills and knowledge domains
– Cutoff scores are set using nurse experts with respect to the “minimum competency level necessary to perform the job on the first day”
• Without using Clinical Assessments, hospitals stand the risk of hiring unqualified and/or under-qualified nurses who may not possess the key knowledge and skill competencies and may create liability.
• Notwithstanding this focus, the Clinical Assessments still demonstrated statistically significant correlations to job performance.
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 10
Project Phase 3: Clinical Assessment Build
Giving Meds… New Techniques…Critical ThinkingConscientious
Change Adaptation
Conflict R
esolution
Amount of Job Performance Variance Predicted by the Clinical Assessment (Ranked)(1,2)
Performance DimensionPercentage
(%)Administering Medications 35%Problem Solving 34%Report Transitioning 33%Critical Thinking 33%Multitasking 32%Assertiveness 30%Following Clinician Instructions 29%Patient Care Plan Management 27%Continuous Observation 25%Patient Care (Calm & Competent) 22%New Technique Application 22%Patient Customer Service 21%Accountability 19%Delegating/Managing Patient Care 18%Developing Patient Relationships 17%Verbal Communication 16%Change Adaptation 16%Honest & Conscientious 14%Conflict Resolution 13%
Ranked list of the Job Performance Dimensions predicted by the Clinical Assessments (after controlling for the variance of the other two tests)
HARD SKILLS & “SOLO” PERFORMANCE
HARD SKILLS & “SOLO” PERFORMANCE
The test correlated to the JP dimensions just as one would expect, with the highest in the “hard skills” requiring “solo performance” and lowest with “soft skills” involving team performance.
SOFT SKILLS AND TEAM PERFORMANCE
SOFT SKILLS AND TEAM PERFORMANCE
Project Phase 4: Behavioral Assessment Build• 344+ Nurses from Saint Francis Medical Center completed
a 70-item personality/behavioral scale (scored using a 1-5 agreement scale) that measured:– Conscientiousness– Tough-Mindedness– Conventional– Extroversion– Stability– Teamwork– Good Impression
• 70-item scale was reduced to a 20-item “Nurse Effectiveness Scale” using a split-half validation methodology.
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 11
Project Phase 4: Behavioral Assessment Build
344 Nurses
176 Nurses“Calibration”
Sample•Build the test
•Maximize correlations
168 Nurses“Validation”
Sample•Re-evaluate correlations
00.050.1
0.150.2
0.250.3
0.350.4
Average Criterion…
Hon
est &…
Verbal Com
mun
ication
Adm
inistering…
Assertiv
eness
Chan
ge Adaptation
Conflict Re
solution
Accou
ntability
Critical Thinking
Multitasking
Prob
lem Solving
Patie
nt Care (calm &…
Repo
rt Transition
ing
Continuo
us…
Develop
ing Patie
nt…
Patie
nt Care Plan
…
Patie
nt Customer…
New
Techn
ique
… 0
0.05
0.1
0.15
0.2
0.25
0.3Av
erage Criterion…
Hon
est &
…
Verbal Com
mun
ication
Administerin
g…
Assertiveness
Change
Adaptation
Conflict Re
solutio
n
Accoun
tability
Critical Thinking
Multitasking
Prob
lem Solving
Patie
nt Care (calm &…
Repo
rt Transition
ing
Continuo
us…
Develop
ing Patie
nt…
Patie
nt Care Plan…
Patie
nt Customer…
New
Techn
ique
…
Project Phase 4: Behavioral Assessment Build
0
0.05
0.1
0.15
0.2
0.25
0.3
Cor
rela
tions
Behavioral Validity CoefficientsNew Techniques
Conscientiousness Accountability Delegation/Mg Patients
Project Phase 5: Combining the Assessments • The tests were weighted
using Structural Equation Modeling (SEM), allowing them to be combined in the most efficient way possible.
• After correlating the three tests with avg. job performance, the remaining contribution of cognitive ability was non-significant.
• This was because the Clinical and Situational Assessments were highly correlated with the cognitive ability test (r = .16 and .29).
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 12
Project Phase 5: Combining the Assessments Percentage (%) of Job Performance Predicted/Explained by Assessment
0%
10%
20%
30%
40%
50%
60%
70%
Averag
e Perf
orman
ce
Honest
& C
onsci
entio
us
Verbal
Commun
icatio
n
Admini
sterin
g Med
icatio
ns
Assertiv
eness
Chang
e Ada
ptatio
n
Conflic
t Reso
lution
Accoun
tabilit
y
Critica
l Thin
king
Multi-ta
sking
Follow
ing Clin
ician
Instr
uctio
n
Problem
Solving
Patien
t Care
(Calm
& C
ompe
tent)
Report
Transit
ioning
Contin
uous
Observ
ation
Develo
ping P
atien
t Rela
tions
hips
Patien
t Care
Plan M
anag
emen
t
Patien
t Cust
omer
Service
New Tech
nique
App
licati
on
Delega
ting/M
anag
ing Pati
ent C
are
Job Performance Dimension
% o
f Job
Per
form
ance
Exp
lain
ed
Clinical Situational Behavioral
Project Phase 5: Combining the Assessments
Top Six Ranked Performance Dimensions Predicted by Each Assessment(1)
Clinical Situational BehavioralAdministering Medications Patient Care (Calm & Competent) Accountability
Problem Solving Developing Patient Relationships Honest & ConscientiousReport Transitioning Verbal Communication Conflict Resolution
Critical Thinking Patient Customer Service New Technique ApplicationMultitasking Patient Care Plan Management Delegating/Managing Patient Care
Assertiveness Conflict Resolution Change Adaptation
Note (1): After controlling for the predictive variance of each of the other two tests.
• The final battery collectively measures each job performance dimension
• The tests load on different aspects of job performance in a balanced way
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 13
How Does This Translate to Actual Hiring Benefits?
Practical Effectiveness of Using Prophecy Assessments(1)
Base Rate of Qualified Nurses
Applicant-to-Hire Ratio
% Qualified Nurses Hired
Hiring Process Improvement Rate
80% 50% 89% 9%
80% 30% 92% 12%
80% 10% 95% 15%
50% 50% 63% 13%
50% 30% 69% 19%
50% 10% 77% 27%Notes: (1) Assumes an average job performance of 50 and a SD of 10.
Project Phase 5: Combining the Assessments
Practical Impact of Using Prophecy Assessments in a Nurse Hiring Process (Assuming 50% "Qualification Base Rate")
30%
40%
50%
60%
70%
80%
90%
100%
100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
Selection Ratio (% of Applicants Screened in Using Prophecy Assessments)
Lik
elih
ood
of H
irin
g an
"A
bove
A
vera
ge"
Nur
se
Likelihood of Hiring an "Above Average" Nurse Using Prophecy Assessments
Ability to Predict Hiring an "Above Average" Nurse Using No Assessments (or invalid assessments)
27% Benefit
9% Benefit
17% Benefit
Scenario Results
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0.04
0.045
15 20 25 30 35 40 45 50 55 60 65 70 75 80 85
Nurse Job Performance Levels
Nurse Workforce Job Performance Improvement Levels Using Prophecy Assessments (.45 Validity Coefficient, 20% Selection Ratio)
Using the Prophecy Assessments creates a 6.3% average performance gap between the tested and untested group
6.3%
24% non-overlap between the tested and untested groups
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 14
• Making the following assumptions…– Hospital with 720 nurses– Job Performance levels in the corporate nurse workforce is normally
distributed, with half “above average” and half “below average”– Turnover is 13.9% (per American Association of Colleges of Nursing/Bernard
Hodes Group)– No growth (to be conservative), although most research reports cite nurse
vacancy rates exceeding 8% and the Bureau of Labor Statistics (BLS) projected that more than 581,500 new RN positions will be created through 2018, which is a 22% increase in the RN workforce
• Let’s run through 3 scenarios:– Using no test (or an invalid test) for 3 years– Using Prophecy Assessments with a 50% cutscore for 3 years– Using Prophecy Assessments with a 20% cutscore for 3 years
Running Through the Math…
Changing the Workforce One Hire at a Time
0%10%20%30%40%50%60%70%
Year 1 Year 2 Year 3
Job Performance Levels of the Workforce After 3 Years of Not Using a Valid Test
Percentage of Above-Average Nurses in the WorkforcePercentage of Below-Average Nurses in the Workforce
Changing the Workforce One Hire at a Time
0%10%20%30%40%50%60%70%
Year 1 Year 2 Year 3
Job Performance Levels of the Workforce After 3 Years of Using Prophecy (w/50% Selection Ratio)
Percentage of Above-Average Nurses in the WorkforcePercentage of Below-Average Nurses in the Workforce
How to Build the Ideal Nurse Hiring Assessment March 23, 2012
Prophecy Healthcare // AONE Annual Meeting & Expo 15
Changing the Workforce One Hire at a Time
0%10%20%30%40%50%60%70%
Year 1 Year 2 Year 3
Job Performance Levels of the Workforce After 3 Years of Using Prophecy (w/20% Selection Ratio)
Percentage of Above-Average Nurses in the WorkforcePercentage of Below-Average Nurses in the Workforce
Wrap-up• The most important factor in building and using a valid assessment process is
measuring the whole person (otherwise, valuable competencies are not considered in making critical employment decisions)
• The next most important factor is measuring the various competencies of that “whole person” using tests that have a proven, scientific connection to those various skills and competencies.
• The next most important factor is measuring those various competencies in reliable (consistent) and efficient (fast) ways.
• The final consideration is properly balancing (weighting) the various tests in a way that matches the job
• Prophecy does all four.