caveon webinar series: key security lessons learned at atp's innovations in testing conference...
DESCRIPTION
The annual Association of Test Publishers Innovations in Testing Conference was held last week in Fort Lauderdale, Florida and Caveon was there! As expected, test security was front and center on the conference program, as more than 17 sessions included test security topics. Dr. John Fremer, President of Caveon Consulting Services, joined by Nikki Eatchel, Vice President, Program Management, Questar Assessment, Inc and the 2011-2012 ATP Security Committee Chairperson, for an informational webinar that explores key takeaways and lessons learned on security from this year’s ATP conference. If you missed the conference or some of the important sessions on test security, this is one way to stay updated on the latest and greatest industry security trends If you have any questions or would like to watch the recording of this session, please contact Richelle Gruber at [email protected] Thank you!TRANSCRIPT
![Page 1: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/1.jpg)
Key Security Lessons Learned at ATP’s “Innovations in Testing” 2013
Nikki Shepherd Eatchel Dr. John FremerVice President PresidentProgram Management Caveon Consulting ServiceQuestar Assessment
February 21, 2013
Caveon Webinar Series:
![Page 2: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/2.jpg)
Upcoming Caveon Events
• National Council on Measurement in Education Conference, San Francisco, April 26-30
• United States Distance Education Association Conference, St. Louis, April 28 - May 1
• National Conference on Student Assessment, National Harbor, June 19-21
![Page 3: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/3.jpg)
Caveon Online• Caveon Security Insights Blog
– http://www.caveon.com/blog/
• twitter– @Caveon– Follow us!
• LinkedIn– Caveon Company Page– Caveon Test Security Group
• Please contribute!
• Facebook– Will you be our “friend?”– “Like” us!
www.caveon.com
![Page 4: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/4.jpg)
Agenda for today
• ATP Test Security Committee & Live Lab• Truth & Consequences, Conducting Effective Test
Security Investigations• The Handbook of Test Security• Data Forensics, Opening the Black Box• The Game’s Afoot: Sleuths Match Wits• Tell it to the Judge, Winning with Data Forensics
Evidence in Court • Balancing Test Security and Accessibility
![Page 5: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/5.jpg)
But, before we get started…
Caveon is proud to announce we are celebrating
10 years in business in 2013!
![Page 6: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/6.jpg)
Security Committee Briefing & Live Lab
Goals for the Year
• Security Survey• Test Security Options• Enforcement• Candidate Rights and Responsibilities• Newsletters• Live Lab
![Page 7: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/7.jpg)
Survey Activities
Activities Completed
• Three surveys designed by the ATPSC • Survey review and feedback provided by the
Institute for Credentialing Excellence (ICE)• Survey Distributed, 117 respondents
– 79 Certification/Licensure– 16 Education– 22 Vendor
• Initial analysis complete
![Page 8: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/8.jpg)
Sneak Peak at the Data Trends
General Information
• Wider variety of respondents for 2012 survey
• Increased focus on security by all respondent organizations
• Large percentage of “Don’t Know” responses
![Page 9: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/9.jpg)
• Larger percentage of respondents with formal, internal security plans
• Significant gap in formal, external security processes
• Lack of standardization implementation
• Few audits
Sneak Peak at the Data Trends (Cont.)
![Page 10: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/10.jpg)
Additional ATPSC Activities
• Security Options Document• DMCA Takedown Effort• Candidate Rights and Responsibilities
![Page 11: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/11.jpg)
Additional ATPSC Activities (Cont.)
Security Options Document• Assessment Models• Delivery Channels
Currently available at
www.testpublishers.org
![Page 12: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/12.jpg)
Additional ATPSC Activities (Cont.)
DMCA Takedown Effort• 6 ISP’s were Identified hosting 17 Infringing
Websites– One ISP – all websites removed– Approximately 30% of the websites no longer exist and
affiliate websites were also removed
Candidate Rights and Responsibilities• 8 organizations participated• Available May 2013
![Page 13: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/13.jpg)
Truth & Consequences: Conducting Effective Test Security Investigations
Benjamin Mannes, CPP, SSI, CHS-III
Director, Test Security
American Board of Internal Medicine
Marc J. Weinstein, Esq.
Partner
Dillworth Paxson, LLP
![Page 14: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/14.jpg)
Truth & Consequences: Conducting Effective Test Security Investigations (Cont.)
• The Proliferation of Risk
• Core Elements of Exam Integrity– Protect– Investigate– Enforce
![Page 15: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/15.jpg)
Truth & Consequences: Conducting Effective Test Security Investigations (Cont.)
Identification and Vetting• Strong web presence• Ongoing review and analysis• Vetting
Analyze• Collusion• Sharing• Inappropriate Study Techniques• Violations• Proxy
![Page 16: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/16.jpg)
Truth & Consequences: Conducting Effective Test Security Investigations (Cont.)
Investigation• Gather• Preserve
![Page 17: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/17.jpg)
Truth & Consequences: Conducting Effective Test Security Investigations (Cont.)
Field Interviews• Purpose• Participants• Nonverbal communication
![Page 18: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/18.jpg)
Truth & Consequences: Conducting Effective Test Security Investigations (Cont.)
Enforcement• Options• Organizational Goals• Policy/Procedures, Laws/Regulations• Civil versus Criminal
Join the conversation at:
@ExamIntegrity on twitter
LinkedIn Group: “Exam Integrity”
![Page 19: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/19.jpg)
HANDBOOK OF TEST SECURITY
• Editors - James Wollack & John Fremer• Participating Authors – Deborah Harris, William
Hatherill, David Foster• Slated for Publication – March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains
– Certification/Licensure– Clinical– Educational– Industrial/Organizational
![Page 20: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/20.jpg)
HANDBOOK – PRIMARY AIMS
• Insights of Psychometricians and Policy-Makers• Best Practices for Designing Secure Tests• Analysis of Security Vulnerabilities• Practical Strategies for Cheating Prevention and
Detection • Lessons Learned
– Actual security violations– Security initiatives
![Page 21: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/21.jpg)
HANDBOOK – FIVE TAKEAWAYS
• Security Vulnerabilities for All Genres of Testing• Critical Importance of Security Planning• Practical and Proven Prevention & Detection
Strategies• How Security Breaches Have Been Dealt With• Lessons We Have Learned From Past Instances of
Testing Misbehavior
![Page 22: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/22.jpg)
HANDBOOK – FIVE PREDICTIONS
• In Many High Stakes Testing Programs– Cheating detection statistical analyses will be performed
routinely– Computer-based testing will increasingly become the norm– Technology developments will be critically important to test
security– Internationalization of testing programs will increase– “Test Security Manager” will become a recognized and valued
position
![Page 23: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/23.jpg)
DATA FORENSICS, OPENING THE BLACK BOX
• John Fremer• Neal Kingston• James Wollack• Dennis Maynes
![Page 24: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/24.jpg)
DATA FORENSICS, OPENING THE BLACK BOX (cont.)
• Features of “Mature” Forensics Approaches– Explicit standards– One or more associations– Formal training programs– Regular conferences– One or more journals– Body of published literature– Regular positions in many agencies and companies
![Page 25: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/25.jpg)
• Useful Statistics– Similarity – Unusual number of same answers – Aberrance – Answering hard questions correctly while
missing easy ones– Gains (or drops) Atypical score changes over time– Erasures – Unusual patterns such as almost always
wrong to right– Fast Responding – Much faster than others –
superhuman?– Shared Information – Biometrics or personal information
– Email or home addresses, photos, etc.
DATA FORENSICS, OPENING THE BLACK BOX (cont.)
![Page 26: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/26.jpg)
THE GAME’S AFOOT: SLEUTHS MATCH WITS
• Kim Brunnert – Elsevier• Joy Matthews-Lopez and Paul Jones – National
Association of Boards of Pharmacy• Lawrence Rudner – Graduate Management
Admissions Council• Dennis Maynes – Caveon Test Security
![Page 27: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/27.jpg)
THE GAME’S AFOOT (cont.)
• Overview of Problem• In 2010, a test preparation course
– Harvested items– Imputed a key– Distributed the content and key
• Test Program Manager Reviewed Score Results• Many Scores at 95 %
![Page 28: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/28.jpg)
THE GAME’S AFOOT (cont.)
• Live Data Set• Three Teams• Illustration of Problem
– 32 identical tests with a score of 95%– Probability of occurrence = 1 in 4 trillion
• 100 Item Test• 387 Test Takers
![Page 29: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/29.jpg)
THE GAME’S AFOOT (cont.)
• Similarities of Team Approaches– Examination of test scores and histograms– Estimation of imputed key– Inspection of item performance– Filtering and splitting of test takers and items
• Differences among Team Approaches– Types of classification rules and models– Degree of confidence in the number of test takers
involved– Use of IRT models –may not apply when disclosed
test content involved
![Page 30: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/30.jpg)
THE GAME’S AFOOT (cont.)
• Summary of Project– Optimal solution is not obvious– Several models can and were used– Problem is harder
• Only some items and keys are compromised• Imputed answer keys are the actual keys
– More research is needed• Detection of imputed answer keys• Bayes classifiers (need probability models given
disclosure)– Harvesting and disclosure one of most damaging
security breaches
![Page 31: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/31.jpg)
TELL IT TO THE JUDGEWinning With Data Forensics Evidence in Court
• Aimee Hobby Rhodes, JD• James Wollack, Ph.D.• Rachel Schoenig, JD• Jennifer Ancona Semko, JD• Steve Addicott
![Page 32: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/32.jpg)
TELL IT TO THE JUDGE (cont.)
• Global Financial Certification Program• Newly Using Data Forensics• Strange Results Observed• Test Program Manager Investigates• Ethics Board Votes to “De-certify”• Candidate Takes Case to Court
![Page 33: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/33.jpg)
TELL IT TO THE JUDGE (cont.)
1. Use an Expert who is truly an Expert
2. Use a Trusted and Tried Methodology
3. Report Fairly
4. Set Up Procedures Properly
5. Be Objective
6. Be Reasonable in What You are Concluding
![Page 34: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/34.jpg)
TELL IT TO THE JUDGE (cont.)
7. Use Your Expert as a Teacher – Explaining what was done and Why the Evidence is Compelling
8. Ensure that your Expert has all the Facts
9. Take into Consideration all Reasonably Available Evidence
10. Apply the Terms of your Candidate Agreement Reasonably and in Good Faith
![Page 35: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/35.jpg)
Balancing Security and Accessibility
Sheryl Lazarus
Martha Thurlow
National Center on Educational Outcomes (NCEO)
University of Minnesota
![Page 36: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/36.jpg)
Balancing Test Security and Accessibility (Cont.)
Accessibility Issues• Paper-Based• Computer-Based
Students can unintentionally be denied meaningful access to examinations.
![Page 37: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/37.jpg)
Balancing Test Security and Accessibility (Cont.)
Typical Accommodation Options
• Test Administrator Access• Visual Cues• Teacher Highlighting• Accessibility Software
![Page 38: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/38.jpg)
Balancing Test Security and Accessibility (Cont.)
Best Practices
• Test Security Guidelines Addressing Accommodations• Test Security Agreements Addressing Accommodations• Consideration of Student IEP, 504, or LEP• Security Training• Protocols for Access to Testing Materials • Protocols for Distribution of Testing Materials
![Page 39: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/39.jpg)
Balancing Test Security and Accessibility (Cont.)
References
Standards for Educational and Psychological Testing
(APA/AERA/NCME, 1999)
Testing and Data Integrity in the Assessment of Student Achievement (NCME) – Draft document
Operational Best Practices for Statewide Large-Scale Assessment Programs (ATP/CCSSO, 2010)
![Page 40: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/40.jpg)
RESOURCES
• CCSSO/ATP – Operational Best Practices• ATP Security Committee• Caveon Test Security
– Blog– Webinars
• (Just released) NCME Guidelines• (Just released) TILSA Guidebook for State
Assessment Directors on Data Forensics
![Page 41: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/41.jpg)
CAVEON’S 10 YEAR ANNIVERSARY VIDEOTake a look and celebrate with us!
http://lnkd.in/XFBKR7
![Page 42: Caveon Webinar Series: Key Security Lessons Learned at ATP's Innovations in Testing conference 2013](https://reader036.vdocuments.us/reader036/viewer/2022070318/557a12b4d8b42a37638b484a/html5/thumbnails/42.jpg)
THANK YOU!
Nikki Shepherd Eatchel Dr. John Fremer
Vice President President
Questar Assessment Caveon Consulting Services
neatchel@questarai [email protected]
- LinkedIn Group – Test Security- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog/- LinkedIn Group – Caveon Test Security