six security challenges to your high stakes test program
TRANSCRIPT
Steve Addicott, Vice President
Dennis Maynes, Chief Scientist
Caveon Test Security
Caveon Webinar Series:
Six Security Challenges to Your
High Stakes Test Program and How Data Forensics May Help
Thwart Them
January 22, 2014
Upcoming Caveon Events
www.caveon.com
Caveon Webinar Series: Next session, February 19 Protecting Your Tests Using U.S. Copyright Law ATP Innovations In Testing Annual Conference • March 2-5 in Scottsdale, AZ • Check out our sessions here: • http://www.caveon.com/atp-2014-innovations-in-testing-caveon-
sessions/ • Visit us in Booth 33 or make an appointment to talk to us about
your specific test security or test development concerns.
Agenda for Today
• Magnitude of the Challenges
• Six Challenges
• Potential Solutions/Approaches
• Role of Data Forensics
• Summary
Magnitude of the Problem
2012 ATP Security Committee Survey Results
• Exact matches of exams on the internet?
• 41% of test sponsors (who completed the survey)
• $88,000,000 - $223,000,000!!!!
• Overall cost estimate for replacing compromised exams
• Intangible Losses
• Validity of certificates
• Credibility of program
• Confidence in certificate holders
Six Challenges
1. Proxy test taking
2. Braindump usage
3. Test theft
4. Technology
5. Stakeholder support
6. Test administration models
“Caveon Speaks Out on IT Exam Security”
http://www.caveon.com/articles/it_exam_security.htm
Proxy Test Taking
•2007: Contracted with a proxy test taker for $1,000
• In a few weeks, the certificate was “awarded.”
• Data analysis discovered
• The test site:
• registered with a false mailing address
• affiliated with a mobile site
• operated by the proxy test taking organization
• Tests at five more test sites were “very similar” / “in
collusion”
• Estimated number of proxy-taken exams was 500 in 6
months • We infer that:
• This organization was paid $1 million for proxy
test taking services for a single exam title in one
year.
We Believe
• Proxy test takers
• Legitimate test sites, but…
• Front room and back room
• Operate multi-nationally
• Super-human performance
• Branching out to other certifications
• Sophisticated
• “Whack-a-mole” – they move on
Braindump Usage
Braindump/Theft Usage Case 2012
• Test taker 313 took the exam on 1/25 at 10 am • 97% of the live items were disclosed on 1/25 at 4 pm.
• The items were “near-exact” (recorded and transcribed)
• Four test takers from the same company (296, 297, 310, and 311)
took the exam on 1/23 and 1/24. • Theft probably occurred on 1/23.
• Eleven more took the exam between 1/25 and 2/29.
• Assuming independence, the similarity had a vanishingly small
probability (<10-38). • The imputed answer key had 10 wrong answers for 60 questions.
• It’s more likely for the Powerball winner to win the next 4 jackpots!
We Believe
• Braindump usage is rampant (may exceed 1 in 6 test
takers)
• Not just for “profiteers” anymore—small groups
• Some braindumpers have gotten smarter.
• Are reacting to new test design tactics
• Some braindumpers are naïve.
• Education is key.
• Invalidating scores will deter braindump usage.
Test Theft
• Testking.com and pass4sure.com
• dominant web-based providers of stolen
content.
• More popular on Google than the word,
“braindump” – Google Trends 1/2014.
A Real-Life Example
• Medical certification program
• Administration to 3,500 candidates on Saturday
• Anonymous email on Wednesday – “I thought you should know…”
– ENTIRE ITEM BANK ATTACHED!!
About Stolen Tests
• Exact copies with answers • Copies of digital files (hacking)?
• Near-exact copies without answers • Digital recording with answer key imputation?
• Reconstructed copies • Recalled or memorized questions?
• Theft triggers • Announced exam republications
• When pass rates drop
• Publication of stolen content appears to take about two weeks
Technology
• Bluetooth-enabled ear
pieces
• Spy cameras
• Other communication
tools
Technology
Technology
Technology
Technology
Technology
Stakeholder Support
In Our Experience
• Legal departments are reluctant to invalidate scores and to revoke certifications
• Many partnering organizations are opposed to sanctions
• Executive “buy in”-- Leadership may not understand the extent of fiscal and ancillary losses
• Poor communication plans – Internal & external
Ensuring that tests measure what they are
intended to measure will yield positive effects for
the candidates and the sponsoring organizations.
Stakeholder Support Can Be Won
Although the number of individuals who pass their exams as a
result of fraudulent exam prep or test taking behavior is very
small, it can have a big impact on the value of your
certification. EMC is committed to providing the highest level of
exam security and does take action when fraudulent exam
practices are uncovered. Every month we perform a statistical
analysis of all exam result(s). Any exam results found to be
questionable - with a high probability of being the result of
exam fraud - we revoke. We have been doing this for over two
years with great success.
-Liz Burns, EMC Proven Professional Program Manager,
posted on the EMC Community Network, August 27, 2009
Test Administration Models
• Security breaches are more likely when…
• Tests are administered 24/7
• CBT vs. Paper/Pencil doesn’t matter
• Franchised test sites are used
• Test prep schools run test sites
• Rules are suspended at conferences
• Item compromise is more likely to occur by theft
than exposure
We Believe
• The publish-and-forget approach is inherently
insecure when tests are administered 24/7.
• Different test administration models may require
different security measures and approaches than
those taught in schools or used by traditional
scheduled testing administrations.
• Test security costs vary with different test
administration models.
Test Security is a Process, Not a State
Protect
Detect
Respond
Improve
Measure and Manage
Protect Against Security Breaches
• Test taker and test developer agreements
• Education for test takers
• Require participation in security investigations
• Messaging
• Cisco Exam Compliance Video Tutorials • https://learningnetwork.cisco.com/community/certifications/policies_r
eference_tools/earned-it-videos
• Security Audits of Policies and Procedures
• Background checks of test site personnel
• Security training of test site personnel
• Registered copyrights
• Deter through enforcement actions
Detect and Respond
• Detect using data forensics
• Similarity to detect sites operated by proxies,
braindump users, and coaching schools
• Latency to detect proxies and braindump
users
• EVT™ items to detect braindump users
• Respond to potential breaches when detected
• Policies need to clearly support using statistics
• Just-in-time analysis or delayed scores remove
messiness of score invalidations
Exam Inoculation
• Active area of research
• “Inoculate the exam” against test
fraud
• Does not require score invalidation
or test site shutdowns
• Requires frequent republication of tests
• Use innovative measurement techniques (EVT) to
detect when to republish
• Use continuous test development model so that new
items are always available when the exam must be
republished
• Will require adjustments to processes used by the
psychometric and test development staff
Data Forensics Detection
Statistical Anomalies
Testing
Irregularities
Security
Violations
Security
Breaches
Test Fraud
Type I Versus Type II Errors
• Focus on test score validity, not candidate
behavior.
• Type I error: Improperly deciding a the test score
is invalid.
• Type II error: Failing to detect when the test score
is invalid.
• Using low probabilities decreases Type I errors
and increases Type II errors.
• This is a conservative approach.
• Errors of allowing invalid scores to stand are
preferred over invalidating valid scores.
Communicating with Stakeholders
• Set appropriate expectations
• Clearly convey what data forensics can and
cannot do
• Policies of “zero tolerance” and “see no evil”
are not reasonable.
• Present and report key metrics
• Number of invalid tests which were detected
• Number of test sites which appear to be errant
• Number of test questions which needed to be
replaced
Questions?
Caveon Online
• Caveon Security Insights Blog
• http://www.caveon.com/blog/
• Twitter - Follow @Caveon
• Caveon Company Page/Caveon Test Security
Group/Caveon Security Minute Group
• “Like” us!
www.caveon.com
Thank you!
Steve Addicott
Vice President
@SdAddicott
Dennis Maynes
Chief Scientist
@DennisMaynes
- Follow Caveon on twitter @caveon
- Check out our blog…www.caveon.com/blog
- LinkedIn Group – “Caveon Test Security”