caveon webinar series - lessons learned at the 2015 national conference on student assessment -...

www.caveon.com 1

Caveon Webinar Series

July 22, 2015

Walt Drane, MS DOEJamie Mulkey, Caveon Test Security

Steve Addicott, Caveon Test Security

Lessons Learned at the National Conference on Student Assessment

www.caveon.com 2

Agenda for Today

• Important themes • Keynote• Security sessions

– Walt– Jamie

• New Resources• Q&A

www.caveon.com 3

State Assessment Director Breakfast

• Wow!• State Assessments =

Politics• Common Core or No

Common Sense?• Exodus from

Consortia

www.caveon.com 4

Impact of Technology on Security• Devices• Technology

Enhanced Items• Automated

Scoring• Computer

Adaptive Testing• Others?

www.caveon.com 5

Keynote – Dr. Andreas SchleicherHow the U.S. and Other Countries Compare:

Lessons Learned from International Education Data

What is the OECD?

Thirty-four democracies work together promoting economic growth, prosperity, & sustainable development

What is PISA?

Test that evaluates 15-year-old students in reading, mathematics, and science, worldwide

www.caveon.com 6

Keynote Key Messages

• “Math skills” vs math skills

• Problem solving skills = Career success

• Great teachers require development

Developing Teaching as a profession

Recruit top candidates into the profession

Support teachers in continued

development of practice

Retain and recognise effective teachers – path for growth

Improve the societal view of teaching as a profession

Mean mathematics performance, by school location, after accounting for socio-economic status77 Capacity at the point of delivery

www.caveon.com 8

Test Security Sessions…

We noticed….there weren’t many!!

www.caveon.com 9

Essential Tips for Leveraging Your

Vendor’s Expertise to Strengthen the Security

of Statewide Assessments

www.caveon.com 10

State Assessment Needs

• Valid and reliable score results

• If results aren’t valid, then actions may be taken – or not taken – that actually harm rather than help

www.caveon.com 11

Holistic Test Security FrameworkDet

er

Detect

Decide (now and for the future)

www.caveon.com 12

Vendor’s Role in DETERRING Negative Activities?Security Audits• Use the Vendor to “Audit” the Security Audit

Process• Thinking Outside the Box

Support/Consulting on policies and procedures• Use vendor to “audit” current policies and

procedures• Student Assessment Handbook, Appendix F, and

Legislation

Invalidations as a deterrent• Use the Vendor during the appeals process

www.caveon.com 13

Vendor’s Role in DETECTING?

Data Forensics

• Systems that utilize sophisticated statistical analyses of test-response data to identify patterns indicative of test fraud (e.g. cheating and piracy).

• Erasure Analysis for paper and pencil assessments

• Analysis of test responses of on-line Assessments

www.caveon.com 14

State’s Expectation of Vendor When DECIDING?To inform our decisions, we expect from our vendors

• Statistical analyses for invalidationsoConsistently use the data you haveoPlan for future challenges (more than

data)

• Information to support go/no go decisions for investigationsoPlay to win - don't pick a fight you are

not reasonably sure you can win

• Deep-dive data analysis and interpretation

www.caveon.com 15

Test Security’s Changing Landscape in Computerized

Testing Environments

www.caveon.com 16

Security in Computerized AssessmentsWhy should we still be concerned about test security with the move to computerized assessments?

When Threats Occur

Before During After

Unauthorized access Computer hacking Keystroke logging Storing outside delivery system Testing interruptions Testing window Memorable items Student workstations Electronic devices Social media

www.caveon.com 18

Policies and Procedures

– How do state assessment programs implement computerized assessments? What policies, procedures, and practices have they put in place?

– Focus on Exam Administration:• Assessment monitoring• Proctor training• PITA Hotline• Online training modules (25% of which

deal with test security)• Resources for

staff/teachers/coordinators

www.caveon.com 19

Test Security Investigations

How can you conduct legally defensible investigations that deliver appropriate consequences?

• Having the right documentation in place

• Collecting all related information • Conducting interviews as needed• What are some of the consequences

www.caveon.com 20

New Resources Made Available at NCSA

www.caveon.com 21

New Resources Available

http://bit.ly/1CHUvsi

www.caveon.com 22

New Resources Available

http://bit.ly/1J71f8q

www.caveon.com 23

New Resources Available

http://bit.ly/1VlyARX

www.caveon.com 24

New Resources Available

http://bit.ly/1RBCKWm

www.caveon.com 25

Thank You!

Walt DraneState Assessment Director, Mississippi DOEWDrane@mde.k12.ms.us

Steve AddicottVice President, CaveonSteve.addicott@caveon.com

Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”

Jamie MulkeyVice President, CaveonJamie.mulkey@caveon.com