Upcoming Caveon Events

• NCME Conference, April 26 – 30th, San Francisco Test Security I: Policy Issues: Technical/Statistical/Methodological Issues

– Presenters include: Dr. John Fremer, Caveon

• USDLA Conference,  April 28- May 1, St. Louis

Will the real learner please stand up?  – Presenters include Caveon’s Jamie Mulkey, Ed.D. and Patrick Martin

• Caveon Webinar Series: Next session, May 2

What You Need to Know about High Stakes Cheating in Your Schools– Presenters include former State Assessment Director, Dr. Mike Stetter,

and Dr. John Fremer, President, Caveon Consulting Services

Latest Publications

• Handbook of Test Security – Now available for purchase! We’ll share a discount code before end of session.

• TILSA Guidebook for State Assessment Directors on Data Forensics – soon to be released

Caveon Online

• Caveon Security Insights Blog– http://www.caveon.com/blog/

• twitter– Follow @Caveon

• LinkedIn– Caveon Company Page– “Caveon Test Security” Group

• Please contribute!

• Facebook– Will you be our “friend?”– “Like” us!

www.caveon.com

“The Good and Bad of Online Proctoring”

Harry LaymanExecutive DirectorDigital Assessment PlanningThe College Board

April 17, 2013

Caveon Webinar Series:

Dr. David FosterCEOCaveon Test Security

Agenda for today

• Recap of Structure and Content of recent paper “Online Proctoring Systems Compared”

• Introduction and Definitions for Online Proctoring Discussion• Test Security Threats Addressed by Online Proctoring• Online Proctoring Features and Threat Impacts• Seven Critical Online Proctoring Best Practices• Odds and Ends

– Research Support– Future Features– Trends

• Lessons for Test Program Users

Online Proctoring Systems Compared:Purpose of Paper

• Published by Dave Foster and Harry Layman– http://bit.ly/proctoring

• Background of paper• Recent paper provides a framework for describing different

offerings and several tables of feature comparisons• Lots of detail, not entirely complete – will remain a work in

progress• A good starting point as you begin to think about features /

benefits and your specific risk profile and threats of concern

Online Proctoring Systems Compared: Structure and Content

• Overview of Online Proctoring – what it is, what it is not, and primary value proposition

• An enumeration of 8 vendor products / services included in the scope of the review– Mix of capabilities and approaches; offerings not homogenous/equivalent and so

comparisons need to be more holistic– We have not yet investigated all options equally, so the “comparisons” are

informative but not complete; look for updates over time

• Seven groups of features or characteristics of Online Proctoring offerings organized into a series of comparison tables

• This presentation will “hit the highlights,” setting online proctoring in the general context of test security threats, and provide a high-level discussion of the attributes and features of Online Proctoring systems that can mitigate those threats.

Online Proctoring Is / Is not:

• Online Proctoring refers specifically to using an internet-based approach to remotely monitor individual test administrations, replacing traditional approaches that have relied on “eyes in the room”

– Not CCTV, roving personnel, or using volunteers in user-selected locations (libraries, etc.)

• Online Proctoring is NOT a “find-your-own-proctor” model as used for decades in distance education exams

Online Proctoring Provides:

• Key value proposition in a nutshell:

– Independent, bias-free proctors

– Professional proctors (training, supervision, logging, accountability, etc)

– Technology-assisted audio, visual and activity monitoring support (e.g., log files, video, lock-down, keystroke monitoring, test session control, real-time intervention, etc.)

General Proctoring Requirements

• Online Proctor During Entire Exam • Continuous Internet • Encryption for Data Transfer • Schedule Availability • Proctor Management

– Supervised – Training – Career Path – Certification

• Program Customization

Security Threats / Scope

• Online Proctoring is primarily concerned with

– Threats during test administration; and hence

– Attempts to either capture test content, or

– Get inappropriate assistance in any way.

Security Threats / Scope

• Online Proctoring is only concerned with a subset of the overall set of program vulnerabilities, threats and risks. What are they?

• The subset are those threats that are detectable by the online proctor (or onsite proctor) and over which he or she has some control

Test Security Threats: A Finite Universe

• Only Six Categories of Cheating Threats– Using pre-knowledge of test content– Colluding with others– Using cheating aids– Using a proxy test taker– Hacking into scoring system– Copying answers from other test takers

• Only Six Categories of Stealing Threats– Hacking into a system and stealing test files– Capturing content by digital photography devices– Capturing content by electronic recording of the screen– Memorizing content– Transcribing content verbally (on paper or recording device)– Getting content from test program insider

Test Security Threats: A Finite Universe

• Only Six Categories of Cheating Threats– Using pre-knowledge of test content– Colluding with others– Using cheating aids– Using a proxy test taker– Hacking into scoring system– Copying answers from other test takers

• Only Six Categories of Stealing Threats– Hacking into a system and stealing test files– Capturing content by digital photography devices– Capturing content by electronic recording of the screen– Memorizing content– Transcribing content verbally (on paper or recording device)– Getting content from test program insider

Down to Six! Test Security ThreatsRelevant for Online Proctors

• Cheating Threats– C1. Colluding with others– C2. Using cheating aids– C3. Using a proxy test taker– C4. Copying answers from other test takers

• Stealing Threats– S1. Capturing content by digital photography devices– S2. Transcribing questions verbally (on paper or

recording device)

C1. Colluding with Others

• How can a proctor effectively detect and deal with collusion?

– Camera that can see a large portion of the room– Camera that can pick up sounds and talking– Ability to pause test session, and perhaps suspend it– Ability to contact test taker, record/store

session and time-stamp incident– Proctors should be prohibited

from viewing test taker screen

C2. Using Cheating Aids

• How can a proctor effectively detect and deal with a test taker using a cheating aid?

– Camera that can see a large portion of the room, particularly desk, computer, keyboard, test taker head/arms/hands

– Strong lockdown program to prevent access to online aids– Ability to contact test taker, pause

test session, and perhaps suspend it– Ability to record/store session and

time-stamp incident

C3. Using a Proxy Test Taker

• How can a proctor effectively detect and deal with a proxy test taker?

– Strong initial authentication– Continuous authentication methods– Ability to contact test taker, pause test session, and

perhaps suspend it– Ability to record/store session and time-stamp incident

C4. Copying Answers from Another Test Taker

• How can a proctor effectively detect and deal with a test taker copying from another?

– Camera that can see a large portion of the room, including other test takers allowed in the room

– Ability to contact test taker, pause test session, and perhaps suspend it

– Ability to record/store session and time-stamp incident

S1. Capture Test Content with Digital Photography

• How can a proctor effectively detect and deal with the digital photography of the test questions?

– Camera that can see a large portion of the room, along with head/hands/torso of test taker

– Prevent proctor from viewing test screens– Ability to contact test taker, pause test session, and

perhaps suspend it– Ability to record/store session and time-stamp incident

S2. Transcribing Questions Verbally

• How can a proctor effectively detect and deal with the transcription of test questions?– Camera that can see a large portion of the room, along

with head/hands/torso of test taker– Prevent proctors from viewing the test taker screen– Ability to contact test taker, pause test session, and

perhaps suspend it– Ability to record/store session and

time-stamp incident

Seven Critical Online Proctoring Best Practices

1. Use wide-view camera with microphone

2. Require lockdown program

3. Prevent proctor view of test screens

4. Allow and require proctor to communicate with test taker

5. Allow proctor to control the test session

6. Use strong authentication methods

7. Record, store and time-stamp test session, along with proctor logs

First: Use a Capable Webcam

• Is external to the computer; connected by 2-4 feet of USB cable and set back

• Has large field of view (approx. 80° up to 360°)• Allows proctor to see testing workstation and head,

torso, arms, hands of test taker• Camera angle and placement

does not allow viewing the

workstation screen• Has integrated microphone

Second: Use a Capable Lockdown Program

Operating System/Computer/Browser• Prevent right-click • Prevent min/max windows

• Prevent printing • Prevent Copy/Paste

• Prevent function keys • Prevent running of applications

• Prevent important key combos • Prevent launch of applications

• Hide Taskbar and Desktop • Prevent communication tools

• Hide menus and icons • Prevent browser control/navigation

Third: Prevent Proctor View of Test Workstation Screen

• What are the reasons a proctor should not view the content of a test taker’s workstation screen?

– Security Reason #1: Viewing the screen may encourage collusion between proctor and test taker

– Security Reason #2: Seeing the items of a high-stakes exam on a proctor’s screen may encourage the theft of the test content

– Privacy Reason: It’s not appropriate for a proctor to view how a test taker answers test questions

Fourth: Allow Proctor to Communicate with Test Taker

• Chat• Canned Messages• Technical Support call if necessary

Fifth: Allow Proctor to Control Test Session

• Control allows proctor to deal effectively with detected threats– Pause the test

• cover or remove the current question• require acknowledgement and compliance from the test

taker• provide a warning

– Suspend the test (can restart at later time)– Cancel the test

• Proctor rules for these actions are subject to customization by the testing program

Sixth: Use Strong Authentication Methods

• Strong– Fingerprint readers– Facial recognition– Keystroke pattern recognition– Voice recognition

• Avoid weak authentication methods– Viewing of government-issued IDs through webcam

Seventh: Record and Store Sessions

• Record entire test session and proctor logs/communications

• Time stamp incidents to make them easy to locate

• Provide easy-to-use review application for both video of test session and proctor logs

• Store long-term or until no longer needed

Best Practices Effects on Security Incidents

• If these best practices are followed what benefits can be expected?

– Better detection of threats or attempted breaches– Professional, un-biased, and competent response by

proctors– Deterrence: Fewer incidents going forward– Stronger evidence basis for decision making and taking any

action, including legal action

Research Support

• Reliable data – on site or online – is hard to come by

• Limited data available for online proctoring; what is there is encouraging

• More reporting by participating organizations will increase the data pool

• Need studies also from academic researchers

It’s Only Going to Get Better: Future Proctoring Features

• Improvements (e.g., better cameras; better proctor training, etc.)

• New features: Automated proctoring technologies– Inappropriate keystroke detection – Audio level detection and display – Detection of suspicious response patterns with Real-Time

Data Forensics– Better facial, voice recognition software on the horizon

More Trends

• More providers, may see increasing stratification between cost and quality / professionalism (as with other security services)

• New technologies may allow better detection, or lower costs – or both– Augmenting human monitoring of online video with real-

time “artificial intelligence” could allow for greater proctor-to-examinee ratios with even better security

Lessons for Program Users

• Do it right or be an easy target• Strong security is always needed for tests where

cheating and theft occur, but not everyone has the same set threats or concerns

• Different audiences may have different levels of technical sophistication, risk, time sensitivity, etc.

• Online proctoring when done properly can be as capable as any other well-run proctoring method at reducing test security risks

HANDBOOK OF TEST SECURITY

• Editors - James Wollack & John Fremer• Published March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains

– Certification/Licensure– Clinical– Educational– Industrial/Organizational

• Don’t forget to order your copy at www.routledge.com– http://bit.ly/HandbookTS (Case Sensitive)– Save 20% - Enter discount code: HYJ82

Questions?

Please join our “Caveon Test Security” LinkedIn group and submit questions into the discussion box or contact richelle.gruber@caveon.com

THANK YOU!

- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”

Harry LaymanExecutive DirectorDigital Assessment PlanningThe College Board

Dr. David FosterCEOCaveon Test Security

Again, find our Online Proctoring Systems Compared paper at http://bit.ly/proctoring