The Jimmy A Young Memorial Lecture

July 17, 20147:00 to 8:30 AMMarco Island, FL

1

The NBRC has honored Jimmy’s memory and the contributions he made to respiratory care through this program since 1978 .

Jimmy Albert Young, MS, RRT1935 –1975

2

Jimmy Albert Young, MS, RRT was one of the profession’s most outstanding and dedicated leaders

– 1935 – born in South Carolina– 1960 – 1966 – served as Chief Inhalation

Therapist at the Peter Bent Brigham Hospital in Boston

– 1965 – earned the RRT credential, Registry #263

– 1966 – 1970 – served in several roles including director of the education program at Northeastern University in Boston

– 1970 – became director of the Respiratory Therapy Department at Massachusetts General Hospital

– 1973 – became the 22nd President of the American Association of Respiratory Care

– 1975 – was serving as an NBRC Trustee and member of the Executive Committee when he passed away unexpectedly

In a 15-year career,-achieved the RRT-directed an education program-directed a hospital department-served as AARC President-served as an NBRC trustee

3

The Clinical Simulation Examination

Then and Now

4

Presenter• Robert C Shaw Jr PhD RRT FAARC

– NBRC Assistant Executive Director and Psychometrician

5

Conflict of InterestI have no real or perceived conflicts of interest that relate to this presentation. Any use of brand names is not meant to endorse a specific product, but to merely illustrate a point of emphasis.

Learning Objectives

• Compare elements of the current RRT credentialing system to elements of the system that is planned for January 2015

• Compare the value of information that has been provided by results from the Clinical Simulation Examination to other elements of the RRT credentialing system

• Describe features of the 20-problem Clinical Simulation Examination for which candidates should be prepared by January 2015

OBJECTIVE 1

Compare elements of the current RRT credentialing system to elements of the system that is planned for January 2015

Compare Current to Future

RRT Program Elements

Number

CurrentJanuary

2015Hours of testing time 9 7Examinations 3 2Testing sessions for a candidate who passes on first attempts 3 2Sets of test scores 4 2Passing points 4 3Multiple-choice items to assess competencies broadly 240 140Patient management problems to assess competencies deeply 10 20Typical number of sections in a patient management problem 10 (8-12) 5 (4-6)

OBJECTIVE 2

Compare the value of information that has been provided by results from the Clinical Simulation Examination to other elements of the RRT credentialing system

QUESTION

Is there a measurement reason for the Clinical Simulation Examination to exist?

RESEARCH HYPOTHESIS 1

Scores from the Clinical Simulation Examination added information beyond the information from multiple-choice examination scores when predicting membership in three groups for candidates who sought the RRT credential.

What defined the three groups of candidates?

Credential Status Label for group

Examination Outcome

CRT Written Registry Clinical Simulation

CRTcertification

pass

fail fail

certification +1 pass fail

fail pass

RRT registration pass pass

Defining the Population

• Date range for examination attempts– October 22, 2009 through February 27, 2012

• A subset of 9,081 candidates had achieved CRT and made a first attempt at the remaining examinations for RRT (and were not outlying cases)– Written Registry– Clinical Simulation

• Information gathering (IG)• Decision making (DM)

Statistical Model and Method

• Step-wise discriminant analysis with automatic variable selection– Predict group membership from multiple variables,

each of which is continuously distributed– Dependent variable

• certification, certification+1, and registration groups– Independent variables

• First run included four sets of scores– CRT, Written Registry, Clin Sim IG, and Clin Sim DM

• Second run included two scores – CRT and Written Registry

Standardizing Examination Scores

• Raw score ranges– CRT = 0 to 140– Written Registry = 0 to 100– Clinical Simulation, varied by test form

• IG = a variable-min to a max in the range of 200-300• DM = a variable-min to a max in the range of 140-170

• Each raw score was converted to a z-score where z = (x – mean) / S

Results from Run 1

Step Test ScoresWilk’s Lambda -

Proportion of unexplained variance

F Test for Entry

Value p

1 Clinical Simulation DM .421 6251.4 <.0001

2 Written Registry .296 3803.5 <.0001

3 Clinical Simulation IG .295 2540.7 <.0001

4 CRT .295 1907.8 <.0001

Predictions about memberships in the registration group were accurate for 92.4% of the cases

Discriminant Score Equation

• Discriminant score =1.026 (Clin Sim DM z-score)+ 0.975 (Written Registry z-score)+ 0.091 (CRT z-score)- 0.010 (Clin Sim IG z-score)- 0.689

• Clin Sim DM and Written Registry scores were nearly equal and the dominant contributors to predictions about group memberships

Results from Run 2

Predictions about memberships in the registration group were accurate for 85.4% of the cases

Step

Test Scores

Wilk’s Lambda -Proportion of unexplained

variance

F Test for Entry

Value p

1 Written Registry .447 5607.3 <.0001

2 CRT .433 2361.6 <.0001

.42

.30

92.4%

Conclusions• The research hypothesis was accepted

– Scores from the Clinical Simulation Examination add information about RRT achievement beyond what is available from multiple-choice examination scores

• If the Clinical Simulation Examination was removed from the system, there would be a 7% loss of accurate RRT classifications– Incompetent candidates would become RRT– Competent candidates would be denied RRT

RESEARCH HYPOTHESIS 2

Although there were four sets of test scores, three tests, and two types of tests, RRT competencies were based on only one cognitive construct.

Examination Type CharacteristicsCharacteristic Multiple-Choice Clinical Simulation

Option-response scoring dichotomous (0 or 1) polytomous (-3 to 3)

Linkages between stimulus-response elements

independent items independent problems,dependent sections

Potential for branching units to which a subset of candidates are directed

no yes

Cut point determination method

external to test development

Integrated with test development

Cost to produce $ $$

Risks from Using Multiple Examinations with Different Characteristics

IGDM

CRTWR

Simulation Multiple-Choice

Type of Examination

CRTIG

DM

WR

Advanced Entry

Level of Examination

Statistical Model and Method

• Principal components analysis with cross-validation– Explore the underlying variance structure within four

sets of test scores• CRT• Written Registry• Clinical Simulation

– IG– DM

– Is useful for confirming a hypothesis, in this case the assertion that there is a common characteristic expressed by the four test scores

Preliminary Result 1As an indicator of sampling adequacy

-KMO should be at least .50

-Sig value should indicate statistical significance

As indicators of positive cross-validation

-KMO values should be about the same

  Sampleswhole random

split 1random

split 2sample size 9,081 4,557 4,224

Kaiser-Meyer-Olkin Measure of

Sampling Adequacy.775 .777 .772

  Chi-Square 18259.56 9358.25 8913.45

Bartlett's Test of

Sphericity

df 6 6 6

Sig. .000 .000 .000

Preliminary Result 2As indicators of making a sufficient contribution to the principal component solution

-Communality values should be at least .50, otherwise a variable should be removed

As indicators of positive cross-validation

-Values across each row should be similar

Communality Values Extraction Samples

Scores whole random split 1 random split 2 CRT .768 .765 .773

Written Registry .765 .771 .759

Clin Sim IG .589 .595 .583

Clin Sim DM .701 .713 .690

Primary ResultThe threshold for a consequential eigenvalue is 1.0

or

Components at the inflection point and beyond lack consequence

Conclusions-The research hypothesis was accepted

There was only one principal component to which all four sets of test scores were linked

-Potential risks associated with using a multiple-examination system were avoided

Summary from Both Studies

• Within the population of new RRTs each year, accurate classifications occur more often because there are multiple examinations

• Risks associated with a credentialing system based on multiple examinations were avoided

Study Limitations

• These were population studies involving a recent period of more than 2 years

• Unless characteristics of candidates or examinations change, I expect these results will generalize into the future– Candidates: program admission criteria, program

duration, program intensity – Examinations: number of instruments, types of

measurements

OBJECTIVE 3

Describe features of the 20-problem Clinical Simulation Examination for which candidates should be prepared by January 2015

Rationale for Changing the Simulation Examination

• Instant scoring demands selection of problems for each new test form that have not changed– After a decade, keeping examination content

current became an increasing challenge

32

Solution• Give the examination committee smaller

content elements from which test forms are assembled– Halve the number of sections in problems– Double the number of problems

• Hold testing time the same at 4 hours

33

ENHANCE PSYCHOMETRIC PROPERTIES

As long as other changes will be made . . .

34

Standardize Test Forms More Thoroughly

35

Type of ProblemSpecifications

Current 10-Problem Future 20-ProblemA1. COPD Conservative Care 1 or 2 2A2. COPD Critical Care 1 or 2 2B. Trauma 1 or 2 3C. Cardiovascular 1 or 2 3D. Neurological / Neurosurgical 1 or 2 2E. Pediatric 1 2F. Neonatal 1 2G. General Medical / Surgical optional 4

Problems Each Candidate Will See• 4 about COPD• 4 about children• 4 about general medical / surgical• 3 about trauma• 3 about cardiovascular• 2 about neuro

– Likely one neuromuscular and one neurologic

Simulation Examination Scores

Advantages of a one score and one cut system

• A test with more items and more points than its predecessors will yield more accurate scores as indicators of candidates’ abilities– Pass and fail decisions become more accurate

• Accuracy is gained without an increase in test administration time– Fee for the Clinical Simulation Examination stays

the same

A Potential Disadvantage of a Combined Score

• Compensation can occur unless the cut score policy is changed– Someone within a few points of passing based on

decision making performance could pass by acquiring a higher percentage of available information gathering points

New Cut Score PolicySection Type

Cut Score Range

Current New

DM 60% to 70% 60% to 70%

IG 60% to 70% 77% to 81%

The cut score for a test form must be the sum of MPLs from the two types of sections such that those section MPLs fall within the two ranges shown in the table

Implementation has mandated addition of options labeled as required among positively-scored options in IG sections

Conforming to the PolicyOne IG Section

Option Current January 2015

1 -2 -2

2 1 1R

3 2R 2R

4 -1 -1

5 2R 2R

6 -1 -1

7 -2 -2

8 1 1

9 2R 2R

10 -2 -2

11 1 1

12 -1 -1

MPL, Max, % 6, 9, 67% 7, 9, 78%

WHY THE CUT SCORE POLICY CHANGE MATTERS

Illustrations that follow came from one test form

DM % Score

9080706050403020100

DM Score DistributionFr

eque

ncy

200

150

100

50

0

Std. Dev = 11.42 Mean = 60N = 2331.00

MPL range remains 60%-70%

IG % Score

9080706050403020100

IG Score DistributionFr

eque

ncy

400

350

300

250

200

150

100

50

0

Std. Dev = 5.76 Mean = 80N = 2331.00

MPL range has been 60%-70%

SCATTERPLOT OF IG & DM SCORES

Ref Lines @ IG & DM MPLs

IG SCORES

240220200180160140120100806040200

DM

SC

OR

ES

130120

110

100

90

80

70

60

50

40

30

20

100

PassNo case in this quadrant

SCATTERPLOT OF IG & DM SCORES

Ref Lines @ -0.05 Z MPLs

IG SCORES

240220200180160140120100806040200

DM

SC

OR

ES

130120

110

100

90

80

70

60

50

40

30

20

100

People in this quadrant would pass under the current system

Pass

Highlights for Students• The numbers of problems by patient type will be

constant for each candidate• Testing time remains 4 hours

– 22 problems will be presented– Results will be based on responses to 20 problems

• As a result of a problem-splitting procedure– Some problems will not offer IG sections– Candidates will see the same number of IG sections

across the whole examination as they currently see

Highlights for Students (cont.)• Responses will be summed across IG and DM

sections that a candidate enters to produce one score to which a cut score will be compared– The cut will equal the sum of MPL values across

sections along the critical path• Compared to the current examination,

responses in IG sections will be consequential– Reduced tolerance for errors

QUESTIONS