Managing Multiple Credentialing Programs

December 2018

Robert C. Shaw, Jr., PhD National Board for Respiratory Care

Lori M. Tinkler, MBA

National Board for Respiratory Care

2

Published by Institute for Credentialing Excellence 2025 M Street, NW, Suite 800 Washington, DC 20036 Copyright © 2018 Institute for Credentialing Excellence Printed in the United States of America

3

Table of Contents

Abstract ......................................................................................................................................................... 4

Introduction.................................................................................................................................................... 5

Adding a new program .................................................................................................................................. 6

Describing the interrelationships between programs .................................................................................... 6

Managing resources ...................................................................................................................................... 8

Financial .................................................................................................................................................... 8

Staff ........................................................................................................................................................... 8

Vendor ....................................................................................................................................................... 9

Examinations and their administrations .................................................................................................. 10

Examination committees and banks ....................................................................................................... 11

Project management ............................................................................................................................... 11

Panels ..................................................................................................................................................... 12

Volunteer turnover ................................................................................................................................... 12

Representing credential holders on the board ............................................................................................ 13

Standardizing those things that can be unified across programs ............................................................... 14

Choosing examination formats and methods to ensure equity across test forms ...................................... 14

Summary of key points ................................................................................................................................ 15

4

Abstract

This white paper describes one organization’s experience managing multiple credentialing programs. The

authors have in mind an audience of those who lead organizations responsible for more than one

credentialing program or who may add a program. Recognizing that every organization’s circumstances

are unique, the authors hope that readers will find value in the ideas described and in linkages of these

ideas to the National Commission for Certifying Agencies (NCCA) Standards.

The NBRC awards eight credentials based on results from six examinations. Each examination is the

responsibility of a committee that follows standard policies and procedures of the board of trustees. The

board consists of 31 trustees, each of whom is appointed to one or two examination committees, except

for the public advisor. Not only do trustees make policies, but they also serve as content experts on

panels during examination cycles. These panels are supplemented with other consultants who become

another part of the system to be managed.

The paper will address the following topics:

adding a new program,

describing the interrelationships between programs,

managing resources,

representing credential holders on the board,

standardizing those things that can be unified across programs, and

choosing examination formats and methods to ensure equity across test forms.

White papers are products of the ICE Publications Committee and offered as a complimentary resource

for ICE members.

5

Introduction

The National Board for Respiratory Care (NBRC) operates programs culminating in eight credentials

based on results from six examinations. Examinations have varying eligibility pathways in recognition of

the diversity of preparation of qualified applicants. Earning NBRC credentials demonstrates that allied

health professionals who perform respiratory care in the United States are committed to public safety.

Within the NBRC’s program portfolio, some were started by the NBRC, while others were started by other

organizations and then transferred to the NBRC. Milestones occurred as follows:

1960s: The first two credentials are awarded to respiratory therapists in general practice.

o 1961: The Registered Respiratory Therapist (RRT) program was the first program.

o 1969: The Certified Respiratory Therapist (CRT) program was started by the professional

association of respiratory therapists, which is now called the American Association for

Respiratory Care (AARC).

The CRT scope of practice was first a subset of RRT practice; today, the CRT

documents a lower proficiency level compared to the RRT within the same scope

of practice.

The CRT program was transferred to the NBRC’s control in 1975; the program

was first linked to state licensure in 1976.

Eventually, every state except Alaska linked licensure to the CRT; a handful of

states have linked licensure to the RRT.

1980s to present: The remaining credentials are awarded in specialty areas of respiratory care

that subsequently emerged.

o 1982: The Certified Pulmonary Function Technologist (CPFT) and the Registered

Pulmonary Function Technologist (RPFT) programs were transferred from the National

Society of Cardiopulmonary Technology.

o 1991: The Neonatal/Pediatric Specialty (CRT-NPS, RRT-NPS) program was started by

the NBRC at the AARC’s request.

o 2008: The Sleep Disorders Specialty (CRT-SDS, RRT-SDS) program was started by the

NBRC at the AARC’s request.

o 2012: The Adult Critical Care Specialty (RRT-ACCS) program was started by the NBRC

at the AARC’s request.

Each credentialing program is NCCA accredited; the first accreditation occurred in 1977 when the NCCA

itself began. After its incorporation in 1960, the NBRC could only start a new program when asked to do

so by the AARC or one of its three physician-sponsoring organizations: the American Thoracic Society

(ATS), the American Society of Anesthesiologists (ASA), and the American College of Chest Physicians

(CHEST). The physician-sponsoring organizations and the AARC lend support by nominating trustees of

the NBRC. There are 31 trustees, including one public advisor, 15 physicians, and 15 therapists, some of

whom have documented competence in at least one specialty area.

The NBRC financially supports itself through fees collected from candidates and credential holders.

6

Adding a new program The NBRC began running multiple programs through transfers in two cases and additions in other cases.

The organization follows a process that is written into its policy when a formal request for a new program

is made. The process involves the following studies, any of which could stop the pursuit of a new

program:

viability,

personnel, and

job analysis

To begin a viability study, the NBRC assembles a focus group of stakeholders. The focus group’s goal is

to evaluate whether consensus exists about the need for a new credentialing program and the likely

impact the program could have. A set of questions is proposed; brainstorming of responses follows until

all ideas have been documented. Multi-voting follows so responses that are most strongly endorsed can

be observed. The proposed questions vary with the proposed program. Our most recent experiences

have involved proposals for emerging specialty areas of practice. Responses to approximately a dozen

questions were evaluated, some of which were as follows:

“What do specialists do that generalists do not do (or do not do as well)?”

“To what extent are specialists uniquely prepared to anticipate patient’s needs?”

“How would the presence of credentialed specialists affect staffing?

If the proposal survives the viability study, then a personnel study follows. The goal is to evaluate whether

enough individuals will pursue a new credential over a multi-year period. A two-pronged sampling plan

can help: one focused on individuals and another focused on employers. Some questions we have asked

individuals during this phase include “Do you work in the specialty?” and “Would you seek a new

credential?” Some questions we have asked employers include “Would a specialty credential add value?”

and “Do you expect that your institution would support a new credential?”

If sufficient endorsement is observed in personnel study results, then the NBRC performs a job-analysis

study. We know that most do not see a job-analysis study as a test of new program feasibility. However,

should a group of content experts be unable to brainstorm a body of job-related content, or should job

incumbents’ responses to the body of content fail to endorse a sufficient proportion of the content, then

program development would stop.

Every program request that has made it to the third step has proceeded to full development in our

experience, but the NBRC did have one proposed program stop after the first step. In that case, the

program (RRT-ACCS) proposal returned about 20 years after the specialty had more fully emerged. A

second attempt at the steps outlined above produced a new credentialing program.

Describing the interrelationships between programs

Some NBRC credentials are based on passing one examination, while other credentials are based on

passing multiple examinations. These systems have contributed to the operation of multiple credentialing

programs. To understand the difference between these programs, compare the eligibility criteria plus the

required examinations for each credential shown in Table 1.

7

Table 1. Credential eligibility criteria and required examinations

Credential Eligibility Examination(s)

Take one examination

CRT Graduation from an accredited education program Therapist Multiple-Choice

CPFT1 Experience or CRT or RRT credential

Pulmonary Function Technology

RPFT1 Experience or CRT or RRT credential

Pulmonary Function Technology

Take two examinations

RRT Graduation from an accredited education program Therapist Multiple-Choice Clinical Simulation

CRT-SDS Six months beyond achievement of the CRT Therapist Multiple-Choice Sleep Disorders Specialty

Take three examinations

RRT-ACCS A year beyond achievement of the RRT Therapist Multiple-Choice Clinical Simulation Adult Critical Care Specialty

RRT-NPS RRT Therapist Multiple-Choice Clinical Simulation Neonatal/Pediatric Specialty

RRT-SDS Three months beyond achievement of the RRT Therapist Multiple-Choice Clinical Simulation Sleep Disorders Specialty

Details about the Therapist Multiple-Choice Examination and the Pulmonary Function Technology

Examination are worth discussing to explain how managing multiple programs has recently eased. When

originally conceived, the CRT credential covered a subset of content covered by the RRT credential.

Some candidates were only educated on the CRT content for a couple of decades, so they took one

examination. Other graduates were educated on the broader RRT body of content. Graduates of these

programs who chose to pursue the RRT credential took the same multiple-choice examination taken by

CRT candidates plus a second multiple-choice examination before taking the Clinical Simulation

Examination. Nearly two decades ago, the Commission on Accreditation for Respiratory Care (CoARC)

stopped accrediting education programs that only prepared candidates for the CRT credential. A few

testing cycles later, content expectations for CRTs and RRTs had merged, as revealed by subsequent

job-analysis studies. The second multiple-choice examination was removed from the RRT credentialing

system and replaced by a high cut score on the Therapist Multiple-Choice Examination. Candidates who

pursue the CRT credential and become licensed in most states still take only one examination.

Candidates who pursue the RRT credential now take two examinations instead of three examinations.

Likewise, the CPFT credential was linked to a subset of content covered by the RPFT credential. Over a

series of job-analysis results, a merger of content was observed. This led to the removal of what had

1 CPFT and RPFT credentials are attained by personnel who perform the tests of pulmonary function (e.g.,

airflow measurements, volumes, diffusing capacity) that are necessary to diagnose lung diseases. Within the same scope of practice, the CPFT documents a lower proficiency level compared to the RPFT. While each healthcare facility sets its own hiring criteria, a person with the RPFT credential is often in charge of a whole lab, while a person with the CPFT credential works with patients while testing is done.

8

been the RPFT examination. Candidates now take one examination; their scores are compared to a low

cut score and a high cut score. Achieving the low cut recognizes a new CPFT, while achieving the high

cut recognizes a new RPFT. The RPFT and RRT credentials document higher levels of proficiency today

rather than mastery of a broader scope of content.

Managing resources Financial Although the NBRC enjoys the support of four sponsoring organizations, financial support is neither given

nor needed. Fees collected from candidates and credential holders provide the necessary support. NCCA

Standard 4: Financial Resources states, “The certification organization must have sufficient financial

resources to conduct ongoing, effective and sustainable certification and recertification

activities.” There is a subtle shift in the language in this standard. Many standards hold each program

accountable, while Standard 4 holds the organization accountable.

The NBRC’s situation illustrates why this shift in syntax makes sense. Starting with the addition of the first

specialty program described in the Introduction, each program has fallen short of entirely paying for itself.

Speaking to Standard 4, the difference between the resources expended to sustain each specialty

program and the resources brought in by the program come mostly from fees paid by those who have or

will achieve the CRT and RRT credentials. Because any in this population could attain specialty

credentials in the future, this scenario is judged equitable.

The comparatively high number of people who seek initial CRT and RRT credentialing also leads to the

release of more test forms each year in support of these credentials than are released for any specialty

program. Our intent is to prevent the overexposure of examination content when we step up the annual

release of test forms. A specialty program bringing in less revenue than is expended to sustain it still

releases a new test form each year because board members know that doing so is vital to ongoing

content relevance, test score accuracy, and public protection.

Having added two programs in the last decade, we will speak to the budgetary impact of those additions.

Adding these programs has increased total expenses for the organization, as each examination

committee now requires more consultants than in the past; plus, there are more committees. However,

our board is large, containing 30 content experts. Reassignment of some of them to newly added

programs has avoided a complete one-for-one addition of volunteers whose travel expenses must be

covered. Hence, while having a large board increases the expense of hosting a board meeting, it has

given us capacity that we have tapped to add programs.

Staff Two elements of NCCA Standard 5 state that each program must identify personnel who conduct

certification activities and that the program must oversee those personnel. The board hires a chief

executive officer (CEO), who, in turn, hires staff to fill various roles. The board reviews the CEO’s

performance each year. Anticipating that it may help other organizations to learn about vital personnel,

the following is a breakdown of the people that the NBRC employs to operate its six examinations and

eight credentialing programs:

9

Chief

o Executive Officer (CEO)

o Financial Officer (CFO)

Executive Support

o Executive Assistant

o Administrative Assistant/Meeting

Planner

o Accounting/HR Coordinator

o Marketing/Communications Coordinator

Information Systems

o Principal Software Engineer

o Systems Administrator

Examinations

o Vice President (in-house

psychometrician)

o Research Associate

o Examinations Coordinator (2)

o Assessment Coordinator

Credentialing Operations

o Vice President

o Certification Services Coordinator

o Certification Services Specialist

o Customer Care Specialist (4)

The vice president for credentialing operations touches candidates and credential holders across all

programs. The vice president for examinations touches each form of each examination along with the job-

analysis, cut-score, and criterion-validation studies done for the programs.

No staff person is fully dedicated to just one examination or just one credentialing program in our

organization, which helps ensure that the needs of each examination and program are met. The closest

we come to dedicated staff occurs when each of the two examinations coordinators is assigned as the

primary person responsible for assembling forms of an examination. Each coordinator is responsible for

three examinations. The two examinations coordinators regularly cover for each other while using the

same checklists and procedures, so such crossover occurs without feeling extraordinary.

Crossing over to work outside one’s normal scope extends to other personnel in the examinations division

when it comes to covering face-to-face meetings and soliciting new items. Our board meets twice a year,

so examination committees meet three to four days leading up to the board meeting. However, some

examination committees only need to meet once a year, which helps staff manage coverage. There are

still some days when three examination committees meet simultaneously, at which point our research

associate becomes the person who facilitates item approval like our examinations coordinators do.

Cross-covering occurs in the credentialing operations division as well. While a pool of customer care

specialists takes phone calls from candidates and credential holders, other operations personnel can

cover calls. Likewise, the fulfillment tasks done by the certification services specialist can be done by

others from the operations area.

Finding smart, adaptable people who can learn to anticipate needs is key to making the system work

since it relies heavily on cross-coverage.

Vendor Absent from the list of NBRC staff are vendor personnel who administer examinations at nearly 300 sites

and support the network that allows those administrations to occur. Most sites are in the United States.

The vendor provides scoring services whose results are posted to item banks that the vendor hosts.

NBRC staff are authenticated to access those banks so they can assemble new test forms and maintain

item content. If the NBRC had no in-house psychometrician or in-house examinations coordinators to

assemble test forms, then a vendor could provide those services. Development and maintenance of a

database of credential holders and candidates plus related software are supported by a vendor on an as-

10

needed basis. Standard 10: Confidentiality holds NBRC staff and staff employed by the vendor(s)

responsible for maintaining confidentiality about examinations, candidates, and credential holders.

Examinations and their administrations Credentials that candidates earn are based on the results of six examinations. All but one of these

examinations are assembled from multiple-choice items, each of which contains four options. The

exception is the Clinical Simulation Examination, which is assembled from patient management

problems.2 If a proposal for a new program included an examination format other than one that contained

multiple-choice items or patient management problems, then it would be further scrutinized regarding its

likely budgetary impact.

Each of the six examinations is administered on demand, regardless of these two formats. We could

envision a testing-window scenario working, but we opted for on demand to maximize access for

candidates. Linkages to licensure for some examinations especially encouraged this choice. Pretesting of

items and problems is deployed for all examinations, with the primary intent to enable the delivery of

results at the end of each administration session. The benefit of pretesting is an elevation in the quality of

items that contribute to candidates’ scores.

Avoiding administration windows and instead deploying items throughout the year means that we do not

concern ourselves with calendar effects on items. Item sequences are automatically scrambled among

candidates: sometimes an item is placed early on a test form, and sometimes it is placed it toward the

middle or the end. Hence, we need not concern ourselves with preserving the home position of an item

on a new test form to be confident that it will perform the same. While on-demand test administrations

throughout the year produce their own challenges, administering all examinations under this model

simplifies the work of managing multiple programs.

Challenges created by on-demand testing with no waiting period are the potential for item overexposure

plus an increased likelihood that a repeat candidate may pass because of more experience taking the

tests rather than because he or she remediated and became a stronger practitioner. Starting in 2020, the

NBRC will follow a new policy about waiting between repeat attempts. Candidates may still make the first

couple of attempts without waiting until the probability of a false-negative outcome shrinks to zero.

Thereafter, candidates may still test on demand, but they will wait 120 days between attempts when

seeking CRT or RRT credentials or 180 days when seeking a specialty credential.

2 These problems present scenarios describing patients during a clinic visit, in the emergency department, just after hospital admission, or days into a hospital stay. Problems contain between four and six sections. Each section contains at least four options, although some may contain more than a dozen options. Each option is given a score of -3, -2, -1, 0, 1, 2, or 3. Some sections prompt candidates to gather more information, while other sections present opportunities to make clinical decisions. The problem cut score is the sum of positive option scores the examination committee has labeled as required or negative option scores labeled as forgiven, either of which is selected by minimally competent candidates. The examination cut score is the sum of the twenty problem cut scores.

11

This policy change illustrates another key point behind managing multiple programs. We have learned to

give ourselves time to implement changes in an orderly manner and to communicate about those

changes with stakeholders.

Examination committees and banks The content of each examination is the responsibility of an examination committee. The names of these

committees and the number of members on each committee are shown in Table 2. Note that some

individuals serve on two committees.

Table 2. Examination committee composition

Examination committee name Credentialing outcome

Number of therapists/ specialists

Number of physicians

Therapist Multiple-Choice CRT, RRT eligibility 6 4

Clinical Simulation RRT 6 6

Pulmonary Function Technology CPFT, RPFT 5 4

Adult Critical Care Specialty RRT-ACCS 5 5

Neonatal/Pediatric Specialty RRT-NPS 5 4

Sleep Disorders Specialty CRT-SDS, RRT-SDS 4 4

Examination committee members can access each new test form as it is assembled as a part of the

approval process, but they cannot access an entire item bank. Such access is restricted to staff in the

examinations division. Among the NBRC’s credentialing programs are several interrelationships

concerning eligibility and proficiency levels. From the perspective of a credential holder, this means taking

another examination to achieve a specialty credential or to document a higher proficiency level. The

board follows a policy that prohibits retesting candidates over a point of content at the same complexity

level. Such a policy has led to a practice in which no items are simultaneously shared across

examinations. We have had items cross banks when testing cycles begin and end. However, once

moved, the item has a new home, so there is no need to ensure subsequent revisions in one bank are

made in another bank.

Project management Purposefully disruptive projects linked to the start of a testing cycle, like a job-analysis study and cut-

score study, are planned for these six examinations more than a decade into the future. Doing so typically

allows us to tackle just one such project in a given year, but some years, there are two projects.

Experience tells us that this cadence spreads the burden for volunteers and staff, so these projects can

succeed without stressing the system. Another way that we create capacity for these projects is to

implement fewer test forms toward the end of testing cycles while letting each one run longer.

12

Panels When we start a new testing cycle, examination committee members may participate in the job-analysis

and cut-score activities. However, panels are expanded to include others who represent stakeholders.

When we perform a cut-score study, we specifically add others who recently achieved the credential to

ensure that people like future candidates contributed to the outcome.

NCCA Standard 13: Panel Competition provides guidance about managing examination committees and

other groups that are formed. Standard 13 states, “The certification program must use panels of

qualified subject-matter experts to provide insight and guidance and to participate in job analysis,

standard setting, and other examination-development activities.” Our observations from renewals of

NCCA accreditations for NBRC programs at the end of 2017 encourages us to share what became a

crucial point. It is no longer sufficient to list the name, academic degree, credential, and geographic

location of each panel member. The NCCA is likely to expect to see information supporting the conclusion

that each member was qualified. Hence, we also reported information about years of experience, name of

employer, and position or title.

While preparing for two cut-score studies linked to specialty programs in 2018, we decided to collect still

more information about panel member experience. Anticipating that a sole response about experience

was tied to overall experience in the mind of a panel member, participants in 2018 cut-score studies also

told us the number of years they have worked in the specialty and the years since their competence in the

specialty was documented. For example, someone could have 12 years of overall experience, 5 years

working in a specialty, and documented competence in the specialty 2 years ago. Collecting such

information permits a more nuanced way to describe qualifications.

Volunteer turnover Before leaving the discussion about panels, we will point out a statement in the Commentary section of

Standard 13: “A system of terms of service that includes a rotation schedule for panel membership

is a useful means of ensuring broad input into the examination program.” Just as it is useful to plan

years in advance about when to sequence the start of each new testing cycle among our multiple

programs, it helps to plan the replacement of a small number of examination committee members

regularly. There is a competing goal to maintain continuity among groups charged to do examination-

related work. We strive to typically replace no more than a couple of members on an examination

committee of eight to ten people.

We can share a strategy the NBRC has deployed to fill openings with qualified volunteers. We have

recently surveyed populations of credential holders so those who are interested in serving as an item

writer, examination committee member, or credential maintenance assessment panel member could

nominate themselves. After collecting demographic information and creating a scoring rubric to

emphasize characteristics we valued, a ranked list became a vital resource when filling openings.

We think that having multiple examination committees likely helps us retain the capable volunteers we

rely on. Part of the payoff for them is the opportunity to interact directly with high-performing, committed

professionals like themselves. Still, those other people are likely to do at least some things differently, so

navigating those differences while working together on items and problems is uniquely challenging and

rewarding for our volunteers. We also organize group lunches and dinners while ensuring their travel

expenses are covered, all of which helps us retain volunteers. We do not want to give the impression that

13

no examination committee member resigns before his or her term ends, because a few have. However,

most complete their full terms.

Representing credential holders on the board Within the respiratory care profession, there are three organizations, each responsible for a different

mission: the AARC is the professional membership organization and advocates for professionals, the

CoARC accredits education programs, and the NBRC credentials individuals. Standard 2: Governance

and Autonomy states, “The certification program must be structured and governed in ways that are

appropriate and effective for the profession, occupation, role, or specialty area; that ensure

stakeholder representation; and that ensure autonomy in decision-making over all essential

certification activities.” Maintaining autonomy over decisions about credentialing programs while

avoiding undue influence as required by essential element A is straightforward when there are three

independent organizations.

Essential element B of the Standard 2 requires an accredited organization to identify its status as a legal

entity. The NBRC is a nonprofit, as defined by section 501(c)(6) of the Internal Revenue Code. Other

essential elements mandate that the board must represent credential holders. Table 3 summarizes the

policy about populating the board to ensure that people who hold each of the credentials awarded by the

board are represented. Here again, having a board of 31 people gives us the capacity to ensure that each

credential is continually represented as trustees rotate on and off.

Table 3. Examination committee composition

Clinician type Sponsoring group At least one Number Voting Elected

Physician American College of Chest Physicians (CHEST)

Pulmonologist or critical care specialist

5 Yes Yes

American Society of Anesthesiologists (ASA)

Anesthesiologist 5 Yes Yes

American Thoracic Society (ATS)

Pulmonologist or critical care specialist

5 Yes Yes

Therapist

Specialty Therapist

American Association for Respiratory Care (AARC)

NBRC Credential:

CRT or RRT

CPFT or RPFT

CRT-SDS or RRT-SDS

RRT-NPS

RRT-ACCS

15 Yes Yes

Public Advisor 1 Yes Yes

Total 31

14

Standardizing those things that can be unified across programs The NBRC has about 50 volunteers touching the content of six examinations each year. Some NBRC credentials require taking multiple examinations, so consistency in the way that we present content across examinations is vital. Standardizing content format across examinations is a priority because allowing variation where it does not belong could be unfair to candidates. One way that the NBRC encourages consistency in the presentation of content is by managing a standing committee composed of the chairperson and vice chairperson of each examination committee. This committee is chaired by the immediate past president of the board. In addition to optimizing test form production procedures with staff, the committee of examination committee chairmen is responsible for an internal document called the Standardized Testing Guide. Attributes of the Guide include the following:

it provides examples of examination content and how each should be presented in items and

problems;

it summarizes evidence-based protocols that have been widely accepted as guides to clinical

practice;

it describes standardized presentations of clinical information about patients, drug names, and

data tables;

it becomes a reference for staff as items and problems enter the system and as examination

committees edit examination content; and

it contains about 30 pages.

The Guide is updated twice a year during the two weeks when meetings of examination committees,

standing committees, and the board occur. In recent memory, the Guide has been revised in small ways

during each meeting. Staff then use the new Guide over the next six months.

Choosing examination formats and methods to ensure equity across test forms A program may choose one of two methods when facing the challenge of deploying multiple test forms

within a testing cycle during which the same specifications for test form assembly are used. One can

transform scores from different forms so they are reported on a single scale, or one can manage test form

assembly so the scale of raw scores remains constant. The NBRC has chosen the latter method since the

year 2000, which was when computerized administrations began. Choosing a constant set of raw scores

is only available when the scored portion of each new test form is entirely assembled from items whose

keys are already validated and whose statistical properties are confidently known. A pretesting system is

required to make this happen. Either method can accomplish the goal described by Standard 21:

Examination Score Equating, which requires certification programs to “demonstrate that different forms

of an examination do not advantage or disadvantage candidates because of differences in the

content framework and/or difficulty of particular forms.”

Any who consider the choice between these two equating methods should understand that when

deploying the assembly management method, raw score characteristics (e.g., mean, standard deviation,

and standard error of measurement) may not persist into the next testing cycle. In fact, the start of each

15

cycle is an opportunity to strengthen the accuracy of test scores. Choosing to do so leads to the selection

of items that are more difficult and more strongly discriminate. If one can assemble the first test form(s) of

the next cycle from items that are generally more difficult and discriminate more strongly than occurred in

the previous cycle, then one should also expect the following changes to the scale of raw scores:

a lower mean because candidates will have more difficulty earning points,

a higher standard deviation describing scores further from what will be a lower mean, and

a lower cut score than was used in the previous cycle.

The third point is complicated to predict, because a cut-score study will be done that influences where the

cut score is located. Assuming cut-score judges properly detect that items in the next form are more

difficult than the past, the cut score should wind up below previous score cutoffs. If the previous cut score

is used instead, then the pass rate will decrease. Being fully informed of these contingencies should

encourage optimal choices about whether to push for more accurate scores at the beginning of a new

testing cycle.

Bringing these points back around to an environment in which there are multiple examinations starting

new testing cycles on a staggered schedule almost every year, we will argue that each program deserves

to be independently evaluated as it enters a new cycle. Some of our programs have capacity to increase

raw score accuracy, while other programs already produce raw scores that are practically as accurate as

they can be. The larger point is that those who operate multiple programs may choose what happens to

the raw score scale when each program transitions from one testing cycle to the next. When working with

a vendor, it is reasonable to describe the preferred outcome regarding raw scale stability or ask for a

recommendation.

Summary of key points The purpose behind this white paper was to describe how the NBRC has managed multiple credentialing

programs while illustrating where NCCA Standards intersect the discussion.

The NBRC recognizes the achievement of generalists and specialists with eight credentials based on the

results of six examinations. The first of these credentials was awarded in 1961. Programs were added

through transfers from other organizations or because a specialty credentialing program was added by

the NBRC at the AARC’s request.

Respiratory care has a formalized separation of organizations that advocate for professionals, accredit

education programs, and credential individuals. Such separation makes it easier, in our opinion, to keep

the NBRC board focused on its mission and head off undue influence from those who are not purely

motivated.

The primary reason the NBRC manages multiple examinations is that multiple credentials rely on results

from two or three examinations. Some examinations document the transition from generalist to specialist.

Other examinations assess competencies in two ways: breadth of knowledge through a multiple-choice

examination and clinical simulation through patient management problems.

Running multiple credentialing programs requires effective resource management while understanding

that those resources include finances, staff, volunteers, and a vendor. The NBRC definitively makes the

16

case that it is free from unwanted outside influence by supporting itself financially without relying on

outside money.

Volunteer management is affected by the expectation that board trustees will serve on at least one

examination committee (which will be supplemented by consultants). The NBRC employs an in-house

psychometrician plus staff who assemble test forms. In-house staff process applications and operate a

call center for inquiries from candidates and credential holders. Any of these functions could be provided

by a vendor, such as the one the NBRC uses to administer and score examinations, but we have decided

that in-house staff should directly control examination assembly plus candidate and credential-holder

communication.

We have found it valuable to plan transitions from testing cycle to testing cycle across programs many

years in advance, so, in aggregate, the organization typically only manages one big project a year, with a

maximum of two projects. Doing so avoids overtaxing the time of volunteers. We have recently found it

useful to solicit and rank potential volunteers to identify the best available people and to widen the scope

of our searches for these vital volunteers.

Board trustees have valued working directly on examinations linked to the awarded credentials. We have

been able to maintain continuous representation of each credential on the board up to this point in part

because there are 31 trustees.

Acknowledging that each credentialing program and each examination is unique in many respects, the

NBRC strives to standardize those things that can be standard. Having a committee that represents each

examination is key to ongoing standardization efforts. It has helped to create and continually revise a

document that is used internally to encourage standardized presentations of examination items and

problems.

The NBRC has chosen to create equivalent test forms within testing cycles instead of transforming scores

from different forms to a second scale. Choosing this alternative has meant that a separate choice must

follow about whether to instigate change in the scale of raw scores or to maintain stability as each new

test cycle starts.

Readers who have follow-up questions about the management of administrative aspects of the NBRC’s

multiple programs should contact Lori Tinkler (lori.tinkler@nbrc.org). Follow-up questions about

measurement should be directed to Robert Shaw (robert.shaw@nbrc.org).