9/27/2019

1

Comparability or equivalency: The parallels between remote proctoring and test centre exam delivery through data and lessons learned.

Isabelle Gonthier, PhD, Yardstick Assessment Strategies (YAS)

Michelle Porter, International Code Council (ICC)

Presenters

Isabelle Gonthier, President & COO, YAS

Michelle Porter, Director Assessment Center, ICC

ICC Certification Program

Exam Delivery

Remote Proctoring – PRONTO

Test Center

Comparison of test center and remote proctoring delivery

Candidate performance

Candidate satisfaction

Lessons Learned

Agenda

9/27/2019

2

Who is ICC?

A member-focused association with over 64,000 members;

Develop model codes and standards used in the design, build and compliance process to construct safe, sustainable, affordable and resilient structures.

Certification exams started being offered through CBT in 2006

Over 440 exam titles currently available and delivered on computers in test centres

82 exam titles are available online through remote proctoring (with more to be made available in the next 6 months).

Remote proctoring was launched in August 2017.

ICC Certification Program

What is PRONTO?

PRONTO stands for ‘Proctored Remote Online Testing Option’

Secure exam delivery service with live remote proctoring

Available 24/7, candidates can plan and take their exam at the day and time most convenient to them (on-demand)

Provides immediate exam results upon completion

97 exam titles available through PRONTO

Approximately 36% of exams available in both test centres and PRONTO are

taken via PRONTO

PRONTO - Remote Proctoring

Process

Candidate connects to proctor (after tutorial and system requirementscheck)

Proctor verifies candidate ID

Proctor asks candidates to show the entire room and the desk (takingwebcam and do a 360 degree scan) and confirms allowed material

Candidate connects to exam through URL provided by the proctor

Proctor monitors exam taking through the entire session

Candidate behaviour

Candidate computer

Breaks are allowed during the testing session but only for a maximum of 5 minutes

Room needs to be rescanned when candidate returns to the exam

PRONTO - Remote Proctoring

9/27/2019

3

Over 300 secure test centres across the US within the Pearson VUE network

Candidates schedule an appointment with Pearson VUE through a single sign-on portal - myICC

Results available upon exam completion

Process

Candidate check-in and ID verification

Candidates put personal belongings in locker

Security check: Metal detector (randomly) and pocketturning/emptying

Proctor verifies candidate material

Proctor assign a seat to candidate

Proctor monitor candidates in test centre through the entire session

Test Centre Delivery

Test Delivery Requirements

REQUIREMENT TEST CENTRE PRONTO

Open-book? Yes Yes

Government-issued photo ID Yes Yes

Restroom or other breaks

allowed?

Yes Yes, for a maximum of

5 minutes

Writing utensils allowed? Yes; whiteboard provided No

ADA accommodations Yes; multiple Limited

Candidate-owned hardware No Yes

Candidate performance: How do test centre delivery and virtual proctoring

delivery compare?

Item performance

Pass and fail results

Time spent on items

Candidate satisfaction: What do candidates have to say and how does it

compare between test centre delivery or virtual proctoring delivery?

Incidents: What type of incidents are more prevalent during test delivery, are

they similar or different?

Test Centre vs Virtual Proctoring

9/27/2019

4

Residential Building Inspector ExaminationA Residential Building Inspector will be responsible for performing inspections of structures to determine compliance with the various Building Codes and Standards adopted by his / her jurisdiction. At this level of certification, the Inspector shall be able to inspect one-and-two family dwellings, townhomes not more than three stories in height, and accessory structures.

Exam format60 MCQs

2 hoursOpen book

Exam deliveryPearson VUE test centres PRONTO (Remote Proctoring)

Our Case Study

Comparison of performance on overall exam showed no significant difference between performance test center and PRONTO

Candidate Performance - Items

Periodx̅

Test

Center

x̅PRONTO

Overall Exam Conclusion

September 2017 –

December 2018

70.7% 70.2%t = 0.252, p >

0.05

No significant difference at

0.05

Candidate Performance - Pass/Fail

Comparison of performance on overall exam showed no significant difference between performance test center and PRONTO

Most candidates take the exam on PRONTO between 8am – 8pm (over 80%)

Note: All candidates that took the exam between 2am and 5 am failed the exam

Most candidates take their exam at 8am in test centers, with a slight increase around noon time.

0.00

50.00

100.00

150.00

200.00

250.00

300.00

350.00

400.00

450.00

7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00

Number of Candidate per Hour

Test Centres

0

20

40

60

80

100

120

12

-1 a

m

1-2

am

2-3

am

3-4

am

4-5

am

5-6

am

6-7

am

7-8

am

8-9

am

9-1

0 a

m

10

-1

1 a

m

11

-12

pm

12

-1p

m

1-2

pm

2-3

pm

3-4

pm

4-5

pm

5-6

pm

6-7

pm

7-8

pm

8-9

pm

9-1

0 p

m

10

-1

1 p

m

11

-12

am

Number of Candidates per Hour

PRONTO

9/27/2019

5

Candidate Performance - Pass/Fail

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00

Pass rate per hour

Test Centre

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Pass Rate per Hour

PRONTO

Average passrate for PRONTO: 40%

Average passrate in test centres: 42%

Candidate Performance - Pass/Fail

No difference observed between test centre and PRONTO delivery on time spent per items on the exam

Candidate Performance - Time spent on items

Periodx̅

Test Centerx̅

PRONTOOverall Exam

Individual Questions

September 2017 –

December 2018

110.74 s per question

117.94 s per question

t = -1.27, p > 0.05

No significant difference at a =

0.05

9/27/2019

6

Candidate Satisfaction

Mean

Test Center

Mean

PRONTO

Overall Satisfaction 82.8% 82.5%

Test Center Location/

Connection was easy82.5% 75.04%

Reservation Procedure

was easy77.44% 75.98%

Test Centre/ Proctor

was knowledgeable85.68% 82.66%

Security of the

registration process85.98% 85.94%

User Friendly 83.08% 81.82%

Examination

functioned as

explained

82.46% 84.14%

Candidate Satisfaction – PRONTO comments

Positive Comments

• Great to sit at home instead of driving a long time to a testing center

• Great system

• Great process, felt more comfortable using own office as opposed to going to a test center

• A lot more comfortable than an exam room; easier to think

• Great interface; proctor was thorough and professional

• No work missed due to ability to schedule the exam at any time

Negative Comments

� Took a long time to get logged and start the exam

� Tutorial on how to use the features should be before the exam starts.

� Spent too much time trying to figure out how to go through the test and how to mark things

� Drawings difficult to navigate

� Connection issues during the testing event

Candidate Satisfaction – PRONTO comments

“I like the format

very much. Love that

I can take the exam

when I am at my

best.”

“This was my first

test ever so I am

good with how it

worked just wished I

had passed.”

“Thank you for

creating a secure and

easy environment.”

“This was an all

around good

experience and I will

use PRONTO for all

of my future ICC

exams”

“Taking exams at anytime

works with my schedule

and it saves driving time

to test centers which are

about an hour away from

my house.”

9/27/2019

7

Risks to Manage: Exam Delivery

Remote Proctoring

• Security

• Cheating

• Item harvesting

• Unauthorized items

• Environment

• Disruptions

• Technical issues

• Equipment and bandwith

• Waiting time to connect with

proctor

Test Centers

• Security

• Cheating

• Item harvesting

• Unauthorized items

• Environment

• Disruptions

• Technical issues

• Proctor to candidate ratio

Phased approach: Implementing PRONTO for a subset of exam titles

Dual approach at the onset: Delivery through test centres AND through virtual

proctoring (not one or the other)

Clear communications with candidates: Ensure they understand the

requirements for remote proctoring (e.g., wired internet connection vs wi-fi)

Strong marketing and education campaign to promote PRONTO: more of this

is always required

Tutorials and videos: the more hands on examples the better

Lessons Learned

9/27/2019

8