Steve Addicott, Vice PresidentDennis Maynes, Chief Scientist

Caveon Test Security

Caveon Webinar Series:

Six Security Challenges to Your High Stakes Test Programand How Data Forensics May Help Thwart Them

January 22, 2014

Upcoming Caveon Events

www.caveon.com

Caveon Webinar Series: Next session, February 19 Protecting Your Tests Using U.S. Copyright Law

ATP Innovations In Testing Annual Conference • March 2-5 in Scottsdale, AZ• Check out our sessions here: • http://www.caveon.com/atp-2014-innovations-in-testing-caveon-s

essions/• Visit us in Booth 33 or make an appointment to talk to us about

your specific test security or test development concerns.

Agenda for Today

• Magnitude of the Challenges

• Six Challenges

• Potential Solutions/Approaches

• Role of Data Forensics

• Summary

Magnitude of the Problem

2012 ATP Security Committee Survey Results

• Exact matches of exams on the internet? • 41% of test sponsors (who completed the survey)

• $88,000,000 - $223,000,000!!!!• Overall cost estimate for replacing compromised

exams

• Intangible Losses• Validity of certificates• Credibility of program• Confidence in certificate holders

Six Challenges

1. Proxy test taking 2. Braindump usage 3. Test theft 4. Technology 5. Stakeholder support 6. Test administration models

“Caveon Speaks Out on IT Exam Security” http://www.caveon.com/articles/it_exam_security.htm

Proxy Test Taking

•2007: Contracted with a proxy test taker for $1,000• In a few weeks, the certificate was “awarded.”• Data analysis discovered

• The test site:• registered with a false mailing address• affiliated with a mobile site• operated by the proxy test taking organization

• Tests at five more test sites were “very similar” / “in collusion”• Estimated number of proxy-taken exams was 500 in 6 months

• We infer that:• This organization was paid $1 million for

proxy test taking services for a single exam title in one year.

From the Internethttp://www.certtoday.com

We Believe

• Proxy test takers• Legitimate test sites, but…

• Front room and back room• Operate multi-nationally• Super-human performance• Branching out to other certifications • Sophisticated

• “Whack-a-mole” – they move on

Braindump Usage

Braindump/Theft Usage Case 2012

• Test taker 313 took the exam on 1/25 at 10 am• 97% of the live items were disclosed on 1/25 at 4 pm.• The items were “near-exact” (recorded and transcribed)

• Four test takers from the same company (296, 297, 310, and 311) took the exam on 1/23 and 1/24. • Theft probably occurred on 1/23.

• Eleven more took the exam between 1/25 and 2/29.

• Assuming independence, the similarity had a vanishingly small probability (<10-38).• The imputed answer key had 10 wrong answers for 60

questions.• It’s more likely for the Powerball winner to win the next 4

jackpots!

We Believe

• Braindump usage is rampant (may exceed 1 in 6 test takers)

 • Not just for “profiteers” anymore—small groups

• Some braindumpers have gotten smarter.• Are reacting to new test design tactics

• Some braindumpers are naïve.• Education is key.• Invalidating scores will deter braindump usage.

Test Theft

• Testking.com and pass4sure.com • dominant web-based providers of stolen

content.

• More popular on Google than the word, “braindump” – Google Trends 1/2014.

A Real-Life Example

• Medical certification program

• Administration to 3,500 candidates on Saturday

• Anonymous email on Wednesday– “I thought you should know…”– ENTIRE ITEM BANK ATTACHED!!

About Stolen Tests

• Exact copies with answers• Copies of digital files (hacking)?

• Near-exact copies without answers• Digital recording with answer key imputation?

• Reconstructed copies• Recalled or memorized questions?

• Theft triggers• Announced exam republications• When pass rates drop

• Publication of stolen content appears to take about two weeks

Technology

• Bluetooth-enabled ear pieces

• Spy cameras

• Other communication tools

Technology

Technology

Technology

Technology

Technology

Stakeholder Support

In Our Experience

• Legal departments are reluctant to invalidate scores and to revoke certifications

• Many partnering organizations are opposed to sanctions

• Executive “buy in”-- Leadership may not understand the extent of fiscal and ancillary losses

• Poor communication plans – Internal & externalEnsuring that tests measure what they are intended to measure will yield positive effects for the candidates and the sponsoring organizations.

Stakeholder Support Can Be Won

Although the number of individuals who pass their exams as a result of fraudulent exam prep or test taking behavior is very small, it can have a big impact on the value of your certification.  EMC is committed to providing the highest level of exam security and does take action when fraudulent exam practices are uncovered.  Every month we perform a statistical analysis of all exam result(s).  Any exam results found to be questionable - with a high probability of being the result of exam fraud - we revoke.  We have been doing this for over two years with great success.

-Liz Burns, EMC Proven Professional Program Manager, posted on the EMC Community Network, August 27, 2009

Test Administration Models

• Security breaches are more likely when…• Tests are administered 24/7

• CBT vs. Paper/Pencil doesn’t matter• Franchised test sites are used• Test prep schools run test sites• Rules are suspended at conferences

• Item compromise is more likely to occur by theft than exposure

We Believe

• The publish-and-forget approach is inherently insecure when tests are administered 24/7.

• Different test administration models may require different security measures and approaches than those taught in schools or used by traditional scheduled testing administrations.

• Test security costs vary with different test administration models.

Test Security is a Process, Not a State

Protect

Detect

Respond

Improve

Measure and Manage

Protect Against Security Breaches

• Test taker and test developer agreements• Education for test takers• Require participation in security

investigations• Messaging

• Cisco Exam Compliance Video Tutorials• https://learningnetwork.cisco.com/community/certifications/

policies_reference_tools/earned-it-videos

• Security Audits of Policies and Procedures• Background checks of test site personnel• Security training of test site personnel• Registered copyrights• Deter through enforcement actions

Detect and Respond

• Detect using data forensics• Similarity to detect sites operated by

proxies, braindump users, and coaching schools

• Latency to detect proxies and braindump users

• EVT™ items to detect braindump users • Respond to potential breaches when

detected• Policies need to clearly support using

statistics• Just-in-time analysis or delayed scores

remove messiness of score invalidations

Exam Inoculation

• Active area of research• “Inoculate the exam” against test

fraud• Does not require score invalidation

or test site shutdowns• Requires frequent republication of tests

• Use innovative measurement techniques (EVT) to detect when to republish

• Use continuous test development model so that new items are always available when the exam must be republished

• Will require adjustments to processes used by the psychometric and test development staff

Data Forensics Detection

Statistical Anomalies

Testing Irregularities

Security Violations

Security Breaches

Test Fraud

Type I Versus Type II Errors

• Focus on test score validity, not candidate behavior.

• Type I error: Improperly deciding a the test score is invalid.

• Type II error: Failing to detect when the test score is invalid.

• Using low probabilities decreases Type I errors and increases Type II errors.• This is a conservative approach.• Errors of allowing invalid scores to stand

are preferred over invalidating valid scores.

Communicating with Stakeholders

• Set appropriate expectations• Clearly convey what data forensics can

and cannot do• Policies of “zero tolerance” and “see no

evil” are not reasonable.

• Present and report key metrics• Number of invalid tests which were

detected• Number of test sites which appear to be

errant• Number of test questions which needed

to be replaced

Questions?

Caveon Online

• Caveon Security Insights Blog• http://www.caveon.com/blog/

• Twitter - Follow @Caveon• LinkedIn

• Caveon Company Page/Caveon Test Security Group/Caveon Security Minute Group

• Facebook• “Like” us!

www.caveon.com

Thank you!

Steve AddicottVice Presidentsteve.addicott@caveon.com@SdAddicott

Dennis MaynesChief Scientistdennis.maynes@caveon.com@DennisMaynes

- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”