a ten-year review of cbt

29
September 30, 2004 2004 CLEAR Conference ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence A Ten-Year Review of CBT By Casey Marks, Ph.D., NCSBN Geraldine Piskorski, MBA, ASCP Board of Registry Anthony R. Zara, Ph.D., PV Presented at the 2004 CLEAR Annual Conference: 21 st Century Regulation: “Show Me” What Works September 30, 2004

Upload: akio

Post on 22-Feb-2016

46 views

Category:

Documents


0 download

DESCRIPTION

A Ten-Year Review of CBT. By Casey Marks, Ph.D., NCSBN Geraldine Piskorski, MBA, ASCP Board of Registry Anthony R. Zara, Ph.D., PV Presented at the 2004 CLEAR Annual Conference : 21 st Century Regulation: “Show Me” What Works September 30, 2004. Session Agenda. Speaker backgrounds - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

A Ten-Year Review of CBT

ByCasey Marks, Ph.D., NCSBN

Geraldine Piskorski, MBA, ASCP Board of RegistryAnthony R. Zara, Ph.D., PV

Presented at the 2004 CLEAR Annual Conference: 21st Century Regulation: “Show Me” What Works

September 30, 2004

Page 2: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Session Agenda

1. Speaker backgrounds2. Before transition to CBT3. Transition4. Operations5. Refinement6. CBT lessons learned7. Conclusions

Page 3: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Speaker Background

Casey Marks, NCSBN

Geraldine Piskorski, ASCP Board of Registry

Anthony Zara, Pearson VUE

Page 4: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

What We’re Trying to Accomplish

1. Share experiences

2. Compare and contrast large vs. small programs’ issues

3. Share solutions

4. Outline successful practices

Page 5: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Some Industry Trends

1. Exploding numbers of programs2. More complicated item types3. Security concerns4. Globalization5. Maturing technologies, e.g., Internet6. Increased choices for program sponsors

Page 6: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Issues Before Transition to CBT

Small program• ASCP/BOR background• Size and depth of item bank • Variable vs. fixed length test• Opportunity to review• Size of Examinee

population• Availability of computer

testing facilities• Cost of delivery

Large program• NCSBN background• NCLEX facts• Pre-CAT research program• Barriers?

Page 7: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Foundational Pre-CBT Research

• Dimensionality• IRT calibration• Scale stability• Comparability of results• Early on, much psychometric research was necessary; now

a bit less, but still need to validate findings with your program.

• Mode of administration– What flavor of CBT? (CAT not right for everyone)

Page 8: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Operational Research

• Examinee interface• Stopping rules• Test length• Item pre-testing plan• Test timing• Candidate impact• Beta testing

Here is where your program’s specifics get analyzed and determined

Page 9: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Transition Issues

Small program• Transition from testing

two days each year to year-round testing

• Staffing• Stakeholder

communications

Large program• CBT benefits• Create roadmap to

success• Organizational

processes• Stakeholder

communications

Page 10: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Benefits of CBT

• More frequent administrations – (daily testing)

• Measurement accuracy and efficiency• Advanced item types can be used• Scheduling on demand• Increased security• Instant candidate results• Professional testing environment• Reduced time between retakes

Page 11: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Roadmap for CBT

• Needs analysis• Stakeholder analysis• Building consensus• Working decision processes• Communications• Leverage the experience of others when

analyzing your program.

Page 12: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Organizational Processes

• Organizational Decision Process– Understanding needs– Options for solutions

• Contracting Process– RFP or directed proposals– Provider options– Focus on outcomes

Page 13: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Operations Issues

Small program• Psychometrics• Administration• Stakeholder support

Large program• Batch to continuous

model– Test development– Psychometrics– Administration– Stakeholder support– Staffing

Page 14: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Operations

• Batch to Continuous: Different Mindset – Test Development

• Constant pool rotation -- ability to introduce content more quickly

• More items needed• Continuous item development/reviewing• Item writer recruitment

Page 15: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Operations

• Batch to Continuous:– Psychometrics

• Constant item maintenance• Different item exposure concerns• Candidate volume spikes/dips due to content changes

and passing standard changes• Ability to remove underperforming

problematic/underperforming items during administration

Page 16: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Operations

• Batch to Continuous:– Test Administration

• Individual issues/incidents• Ongoing test center monitoring• Ratio of investigations to candidates increases• Time from exam registration to result• Time-dependent registration policies

– Application process– Fee changes– Rescheduling issues

Page 17: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Operations

• Interface with stakeholders– Test accommodations– Communicating program changes– Procedure/documentation changes

• Stakeholder Support– Direct support– Information/communication

• Accessible • Timely• Constant• Rumors/urban legends

Page 18: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Operations

• Batch to Continuous:– Program Staffing

• Dedicated• Specialized• Outsourcing

– Customer Service• The on-demand mentality• Expectations

Page 19: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

CBT Program Refinement

Small program• Immediate pass/fail

results at test center• Providers - more

options• Quality of graphics

Large program• Ongoing research• Item types• Providers• Service

Page 20: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Refinement - Research

• Ongoing Research– Methodology and content– Staff and/or test service– External

• Implementation dates seemingly “arbitrary”

Page 21: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Refinement – Item Types

• Advanced item types– Media– Stimuli– Response formats

Page 22: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Refinement - Providers

• Service Provider Change– Between 1994 and 2002 same CBT provider– New test development and administration provider

• (2 for 1) in 2002– New center network– New test driver/candidate interface– Candidate support– Data reconciliation between test services– New application/administration/reporting/results

processing systems– Seamless…

Page 23: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Program Refinement - Service

• Innovative item types (beyond MCQ)– Utilizing same exam administration parameters

• More frequent administrations – 2 to 4 to 8 per year

• Additional testing time– Opportunity to demonstrate competence under

optimal exam termination rules

• International administration– Outside U.S. and territories starting in 2005

Page 24: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

CBT Lessons Learned

Small program• Importance of good

communication with all stakeholders

• Size of examinee population

• Improved benefits to examinees

Large program• Benefits obtained worth

it?• Program complexities• Stakeholder

satisfaction?• Service?• Etc.

Page 25: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Challenges

• Large program, yet perfection is needed/expected daily…– More moving pieces, more expertise

needed– Change is harder to integrate– The more complex the program, the more

difficult to educate stakeholders– Timing of program changes never

convenient for all stakeholders

Page 26: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

General Conclusions

• Any large vs. small program things?

• Any “gotchas”?

• Need to consider all program stakeholders when considering procedure changes

Page 27: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Conclusions

• CAT is a good thing for the NCLEX…– Boards of Nursing enjoy:

• Convenience and speed of system• The ability to focus on exam content not psychometric and/or exam

administration– Candidates like:

• Quick registration, scheduling and results receipt • Scheduling options• Personalized service

– NCSBN appreciates:• Security benefits• The ability to test content in new and novel ways

Page 28: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Discussion

Page 29: A Ten-Year Review of CBT

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

Contact Information

Casey Marks, Ph.D.Director of Testing ServicesNCSBNemail: [email protected]

Gerry Piskorski, MBADirector, Applicant, Certificant and IT ActivitiesASCPemail: [email protected]

Anthony R. Zara, Ph.D.Vice President Testing ServicesPearson VUEemail: [email protected]