Measuring Post-Measuring Post-Licensure CompetenceLicensure Competence

The Nursing Performance ProfileThe Nursing Performance Profile

Research TeamResearch Team

Janine Hinton RN, Ph.DJanine Hinton RN, Ph.D Mary Mays Ph.DMary Mays Ph.D Debra Hagler RN, Ph.DDebra Hagler RN, Ph.D Pamela Randolph RN, MSPamela Randolph RN, MS Beatrice Kastenbaum RN, MSN, Beatrice Kastenbaum RN, MSN,

CNECNE Ruth Brooks RN, MS, BCRuth Brooks RN, MS, BC Nick DeFalco RN, MSNick DeFalco RN, MS Kathy Miller RN, MSKathy Miller RN, MS Dan Weberg RN, MHIDan Weberg RN, MHI

SupportSupport

Funded by NCSBN CRE GrantFunded by NCSBN CRE Grant Supported by:Supported by:

Scottsdale Community CollegeScottsdale Community College Arizona State UniversityArizona State University Arizona State Board of NursingArizona State Board of Nursing

Statement of the ProblemStatement of the Problem

A valid reliable practice A valid reliable practice assessment is needed to assessment is needed to support intervention on the support intervention on the public’s behalf when the pattern public’s behalf when the pattern of nursing performance results of nursing performance results in or is likely to result in patient in or is likely to result in patient harmharm

Literature ReviewLiterature Review

Medical errors a leading cause Medical errors a leading cause of death of death (IOM, 2000)(IOM, 2000)

Written tests do not directly Written tests do not directly measure performance measure performance (Auerwarakul, (Auerwarakul,

Downing, Jaruratamrong, & Praditsuwan, 2005)Downing, Jaruratamrong, & Praditsuwan, 2005)

Multiple observations of a Multiple observations of a nurse’s performance have nurse’s performance have provided evidence of competent provided evidence of competent practice practice (Williams, Klaman,& McGaghie, 2003)(Williams, Klaman,& McGaghie, 2003)

Literature ReviewLiterature Review

High-fidelity simulation technology High-fidelity simulation technology allows the creation of reproducible allows the creation of reproducible scenarios to evaluate nursing scenarios to evaluate nursing performance performance (Boulet et.al., 2011; Kardong-Edgren, Adamson, & (Boulet et.al., 2011; Kardong-Edgren, Adamson, &

Fitzgerald, 2010)Fitzgerald, 2010)

Nursing and Health care leaders Nursing and Health care leaders have called for performance have called for performance assessments to evaluate assessments to evaluate competence and support competence and support remediation remediation (Benner, Stupen, Leonard, & Day, 2010; IOM, (Benner, Stupen, Leonard, & Day, 2010; IOM,

2011)2011)

Purpose of studyPurpose of study

To develop and evaluate a high-To develop and evaluate a high-stakes simulation testing stakes simulation testing process to measure minimally process to measure minimally safe nursing practice safe nursing practice competence and identify competence and identify remediation needs. remediation needs.

MethodologyMethodology

Needed process to apply Needed process to apply sophisticated measures of validity sophisticated measures of validity and reliabilityand reliability Participants appeared in 3 simulation Participants appeared in 3 simulation

videosvideos 3 subject matter expert rated each video 3 subject matter expert rated each video

on 41 measures of competencyon 41 measures of competency Raters blind to participant ability, Raters blind to participant ability,

experience, order of testingexperience, order of testing Videos presented a range of safe and Videos presented a range of safe and

unsafe performanceunsafe performance Obtained ratio level data suitable for Obtained ratio level data suitable for

parametric, inferential statistical analysisparametric, inferential statistical analysis

Filming Participant Filming Participant DemographicsDemographics Criteria—newly licensed RNs less Criteria—newly licensed RNs less

than 3 years nursing experience than 3 years nursing experience (N=21)(N=21)

Average age=32Average age=32 95% female95% female 58% white, 16% black, 26% hispanic58% white, 16% black, 26% hispanic 79% AD; 21% BSN79% AD; 21% BSN Less than 3 years experience—Less than 3 years experience—

mean experience=1.05 yearsmean experience=1.05 years Majority had some experience with Majority had some experience with

simulation 74% simulation 74%

Rater DemographicsRater Demographics

Criteria--BSN and 3 years Criteria--BSN and 3 years experience and work in a role experience and work in a role that involves evaluating others that involves evaluating others (N=4)(N=4)

Average experience=12.5Average experience=12.5 Age 31-51Age 31-51 White, femaleWhite, female Education: 3 BSN, 1 MSEducation: 3 BSN, 1 MS

Instrument DevelopmentInstrument Development

Developed and established initial Developed and established initial validity/reliability before fundingvalidity/reliability before funding

TERCAP served as the theoretical TERCAP served as the theoretical framework framework (Benner et.al.,2006; Woods & Doan-Johnson, 2003)(Benner et.al.,2006; Woods & Doan-Johnson, 2003)

Survey items on NCSBN’s Clinical Survey items on NCSBN’s Clinical Competency Assessment of Newly Competency Assessment of Newly Licensed Nurses were adapted Licensed Nurses were adapted (NCSBN, (NCSBN,

2007)2007)

Mapped to QSEN competenciesMapped to QSEN competencies

Categories of Items Categories of Items (TERCAP)(TERCAP) Professional ResponsibilityProfessional Responsibility Client AdvocacyClient Advocacy AttentivenessAttentiveness Clinical Reasoning, noticingClinical Reasoning, noticing Clinical Reasoning, understandingClinical Reasoning, understanding CommunicationCommunication PreventionPrevention Procedural CompetencyProcedural Competency DocumentationDocumentation

Example of One Item Example of One Item Category CompetenciesCategory Competencies

PreventionPrevention Infection control Infection control 2 client identifiers2 client identifiers Appropriate positioningAppropriate positioning Safe environmentSafe environment

Scoring—4 possibilitiesScoring—4 possibilities

Performance or action is consistent Performance or action is consistent with standards of practice and free with standards of practice and free from actions that may place the from actions that may place the client at risk for harmclient at risk for harm

Fails to perform or performs in a Fails to perform or performs in a manner that exposes the client to manner that exposes the client to risk for harmrisk for harm

No opportunity to observe in the No opportunity to observe in the scenarioscenario

BlankBlank

Scoring testScoring test

No weighted itemsNo weighted items No pass fail standardNo pass fail standard Description of Nurse’s Description of Nurse’s

performance across 9 performance across 9 categories of competencycategories of competency

Final rating of each item based Final rating of each item based on inter-rater agreement—at on inter-rater agreement—at least 2 of 3 agreeleast 2 of 3 agree

ScenariosScenarios

3 sets of 3 scenarios scripted=93 sets of 3 scenarios scripted=9 Adult acute care, common Adult acute care, common

diagnosesdiagnoses Each scenario had opportunities Each scenario had opportunities

to observe all performance to observe all performance itemsitems

Each sim patient had hospital-Each sim patient had hospital-like chart with information—labs, like chart with information—labs, history, MAR, ordershistory, MAR, orders

Simulation Testing/RatingSimulation Testing/Rating

21 nurse performers= and 63 videos21 nurse performers= and 63 videos Scenario Set 1=5 participantsScenario Set 1=5 participants Scenario Set 2=8 participantsScenario Set 2=8 participants Scenario Set 3=8 participantsScenario Set 3=8 participants

Each video evaluated by 3 ratersEach video evaluated by 3 raters 189 rating instruments189 rating instruments 41 items rated on each instrument41 items rated on each instrument 7,749 ratings7,749 ratings

Analysis ProceduresAnalysis Procedures

Predictive Analysis Software Predictive Analysis Software (v 18.0.3 SPSS (v 18.0.3 SPSS

Inc., Chicago, IL)Inc., Chicago, IL)

Frequency analysis to identify Frequency analysis to identify instrument properties:instrument properties: Used as intended Used as intended Interrater reliabilityInterrater reliability Sensitive to common practice errors Sensitive to common practice errors

(construct validity)(construct validity) Cronbach’s alpha (intercorrelation Cronbach’s alpha (intercorrelation

among items) was used to measure among items) was used to measure internal consistencyinternal consistency

Analysis Procedures ContAnalysis Procedures Cont

ANOVA used toANOVA used to Assess ability of instrument to Assess ability of instrument to

distinguish between experienced distinguish between experienced and inexperienced nursesand inexperienced nurses

Assess potential bias created by Assess potential bias created by administration methodsadministration methods

ResultsResults

Less than 1% of items left blank Less than 1% of items left blank or not observed—or not observed—indicates indicates scenarios comprehensivescenarios comprehensive

Interrater reliability—across all Interrater reliability—across all 41 items at least 2 raters agreed 41 items at least 2 raters agreed on 99.12%on 99.12%

Internal consistency Cronbach’s Internal consistency Cronbach’s alpha=0.91-0.84 on 41 items alpha=0.91-0.84 on 41 items combined and separatecombined and separate

ResultsResults

Construct validity—pass rates Construct validity—pass rates should mirror those in other should mirror those in other studiesstudies Infection control—pass rate 57% Infection control—pass rate 57%

mainly due to lack of hand mainly due to lack of hand hygienehygiene

Documentation—pass rate 29%--Documentation—pass rate 29%--area of frequent concern in area of frequent concern in practicepractice

ResultsResults

Criterion validityCriterion validity 2 groups by nursing experience2 groups by nursing experience

<1 year or<1 year or 1-3 years1-3 years

2 way mixed ANOVA2 way mixed ANOVA Experienced nurses made fewer errors than Experienced nurses made fewer errors than

new nurses (p<0.001)new nurses (p<0.001) Significant in 6 of 9 categoriesSignificant in 6 of 9 categories

AttentivenessAttentiveness Clinical Reasoning (noticing)Clinical Reasoning (noticing) Clinical Reasoning (understanding)Clinical Reasoning (understanding) CommunicationCommunication Procedural CompetencyProcedural Competency DocumentationDocumentation

CategoryInexperienced

NursesExperienced

Nurses

M (S) M (S) p value

Professional Responsibility

-.33 (.58) -.22 (.49)

*

Client Advocacy -.57 (.81) -.25 (.50) *

Attentiveness -.76 (1.00) -.17 (.38) 0.002

Clinical Reasoning - Noticing

-1.19 (1.33) -.47 (.81)

0.01

Clinical Reasoning - Understanding

-1.67 (1.28) -.81 (.95)

0.005

Communication -1.48 (1.44) -.75 (.97) 0.03

Prevention -1.57 (1.50) -1.31 (.82) *

Procedural Competency

-2.76 (2.32) -1.19 (1.37)

0.002

Documentation -3.33 (.73) -2.61 (1.20) 0.02

Comparison of Groups by Comparison of Groups by CategoryCategory

NPP ResultsNPP Results

Inexperienced 0.5 year

1 year experience

Inexperienced 0.5 yr

2 year experience

ResultsResults

Test BiasTest Bias Scenario was not significantScenario was not significant Categories was significant—some Categories was significant—some

competency categories more competency categories more difficultdifficult Communication, prevention, Communication, prevention,

procedural competency and procedural competency and documentation more difficultdocumentation more difficult

ResultsResults

Test Bias continuedTest Bias continued Scenario set—significant only for Scenario set—significant only for

documentation which may be documentation which may be easier on Set 1easier on Set 1

Order of testing and practice effect Order of testing and practice effect not significantnot significant

Location of testing not significantLocation of testing not significant

SummarySummary

Instrument has adequate validity and Instrument has adequate validity and reliabilityreliability

Raters used instrument as instructed Raters used instrument as instructed and in a reproducible mannerand in a reproducible manner

Items were highly interrelatedItems were highly interrelated Sensitive to common errorsSensitive to common errors Inexperienced nurses made more Inexperienced nurses made more

errorserrors Test not biasedTest not biased Plots permit users to visualize Plots permit users to visualize

performanceperformance

ImplicationsImplications

Provides a valid explicit measure of Provides a valid explicit measure of performance that regulatory Boards performance that regulatory Boards could use along with other data to could use along with other data to determine if practice errors are a determine if practice errors are a one-time occurrence or a pattern of one-time occurrence or a pattern of high risk behaviorhigh risk behavior

Potential uses in education and Potential uses in education and practice to assess performance and practice to assess performance and effect of educational interventioneffect of educational intervention

LimitationsLimitations

Volunteer subjects—not random or Volunteer subjects—not random or representativerepresentative

Sample size too small to support Sample size too small to support confirmatory factor analysis of the confirmatory factor analysis of the instruments construct validityinstruments construct validity

Tailored to specific context and Tailored to specific context and purposepurpose

Limitations of simulation—non-verbal Limitations of simulation—non-verbal and skin change cues missing—and skin change cues missing—suspend disbelief suspend disbelief

Future ResearchFuture Research

Funded by NCSBN for Phase IIFunded by NCSBN for Phase II Criterion Validity by comparing RN Criterion Validity by comparing RN

self and supervisor ratingsself and supervisor ratings Compare to education, Compare to education,

certification certification Broader cross section of Broader cross section of

experienced nurses recruitedexperienced nurses recruited

ReferencesReferences

Auewarakul, C., Downing, S. M., Jaturatamrong, U, and Auewarakul, C., Downing, S. M., Jaturatamrong, U, and Praditsuwan, R. (2005). Sources of validity evidence for Praditsuwan, R. (2005). Sources of validity evidence for an internal medicine student evaluation system: An an internal medicine student evaluation system: An evaluative study of assessment methods. evaluative study of assessment methods. Medical Medical Education, 39Education, 39, 276-283., 276-283.

Benner, P., Sutphen, M., Leonard, V., Day, L. (2010). Benner, P., Sutphen, M., Leonard, V., Day, L. (2010). Educating Nurses: A Call for Radical TransformationEducating Nurses: A Call for Radical Transformation. . Stanford, CA: Jossey-Bass.Stanford, CA: Jossey-Bass.

Boulet, J. R., Jeffries, P. R., Hatala, R. A., Korndorffer, J. Boulet, J. R., Jeffries, P. R., Hatala, R. A., Korndorffer, J. R., Feinstein, D. M., & Roche, J. P. (2011). Research R., Feinstein, D. M., & Roche, J. P. (2011). Research regarding methods of assessing learning outcomes. regarding methods of assessing learning outcomes. Simulation in Healthcare, 6Simulation in Healthcare, 6(7), supplement, 48-51.(7), supplement, 48-51.

Institute of Medicine (IOM) (2011). Institute of Medicine (IOM) (2011). The Future of The Future of Nursing: Leading Change, AdvancingNursing: Leading Change, Advancing Health. Health. Washington, DC: National Academies Press. Washington, DC: National Academies Press.

ReferencesReferences

Institute of Medicine. (2000). Institute of Medicine. (2000). To err is human: Building a To err is human: Building a safer system.safer system. Washington, DC: National Academies Washington, DC: National Academies PressPress

Kardong-Edgren, S., Adamson, K. A., & Fitzgerald, C. Kardong-Edgren, S., Adamson, K. A., & Fitzgerald, C. (2010). A review of currently published evaluation (2010). A review of currently published evaluation instruments for human patient simulation. instruments for human patient simulation. Clinical Clinical Simulation in Nursing, 6Simulation in Nursing, 6(1). (1). doi:10.1016/j.ecns.2009.08.004doi:10.1016/j.ecns.2009.08.004

Williams, R. G., Klamen D. A., & McGaghie, W. C. Williams, R. G., Klamen D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. of bias in clinical performance ratings. Teaching andTeaching and Learning in Medicine, 15Learning in Medicine, 15(4), 270-292.(4), 270-292.