miami-dade county public schools september 27, 2012
TRANSCRIPT
Miami-Dade County Public SchoolsSeptember 27, 2012
Test Chairpersons’ Orientation
AgendaGeneral Session (8:15 – 11:30 a.m.)
Welcome
Dr. Sally A. Shay, Student AssessmentAccountability: School Grades and Annual Yearly Progress
Gisela Feild, and Vilma Rubiera, Assessment, Research, and Data Analysis
2012-13 Testing Calendar
Updates to the Testing Programs
VPK Assessment
Dr. San Juanita de la Cruz, Early Childhood Education
AgendaGeneral Session (Continued)
Test Security, Reporting Violations, and Responding to State Inquiries
District Investigations:
Detective Louis Nunez, Miami-Dade School Police
General Session (12:30 – 1:30 p.m.)
Introduction to File Download Manager
Tiffany Davis, American Senior High School and Betty Viada, Miami Lakes Educational Center
Breakout Sessions (1:45 – 3:30 p.m.)
Test Chair 101: Basics on preparing test materials
Zulema Lamazares, Henry Flagler Elementary
Edusoft: Testing Platform for Baseline and Interim Assessments
Rudy Azcuy, Instructional Technology
On-line CELLA: Administration for ESOL PlacementDeland Innocent, Bilingual/World Languages
Agenda
Accountability
Gisela Feild, Administrative Director, Assessment, Research and Data Analysis
and Vilma Rubiera, Project Manager, Data Analysis
http://oada.dadeschools.net/TestingCalendar/TestingCalendar.asp
2012-13 Testing Calendar
7
2014 and Beyond
Program Updates:
Florida Comprehensive Assessment Test(FCAT/FCAT 2.0)
and
End-of-Course (EOC) Assessments
Next Generation Sunshine State Standards(NGSSS)
Maria C. Bruguera, Director I
Mara Ugando, Staff Specialist
Phasing in FCAT 2.0 Writing Phasing-in FCAT 2.0 Reading computer-based test
(CBT) in Grades 7 and 9 Phasing in FCAT 2.0 Mathematics computer-based
test (CBT) in Grade 5 FCAT (SSS) Reading Retake (graduation test) will not
longer be offered in the 2012-13 school year and beyond.
Civics End-of-Course (EOC) will be field tested in selected middle schools
FCAT 2.0 and EOC Updates
EOC Requirements based on cohort year:
Algebra 1, Biology 1, and Geometry EOC Assessments will be administered in grades 6 -12, to enrolled students in an eligible courses in winter, spring, and summer.
Grade 9 students in 2012-13 and younger: must pass the Algebra 1, Biology 1, and Geometry EOC assessments to earn high school credit.
Grade 9 in 2011-12 (Grade 10 students): must pass the Algebra 1 EOC to earn high school credit. In addition, the Biology 1 and Geometry EOC scores will count as 30% of the final course grade.
Grade 9 in 2010-11 (Grade 11 students): the Algebra 1 EOC score will count as 30% of the final course grade.
The US History EOC will count as 30% of the final course grade for ALL students in Grades 9 -12 enrolled in an eligible course. This 30% rule will remain in effect this school year and beyond.
EOC Requirements
Transition Schedule
General Specifications (Windows/PC and Apple/Macintosh)
Each computer must meet or exceed the following General Specifications in addition to the applicable Operating System Specifications below:
500 MB available disk space
1024x768 screen resolution
Keyboard and mouse (or other pointing device)
High-speed internet connectionOperating System SpecificationsWindows/PC Windows XP
(Service Pack 3)
Pentium III 733 MHz processor
512 MB RAM
Windows (Service Pack 1)
Pentium III 1 GHz processor
1 GB RAM
Windows 7
Pentium III 1 GHz processor
1 GB RAM
Apple/Macintosh
Mac OS X 10.4
G3, G4, G5 500 MHz processor
512 MB RAM
Mac OS X 10.5
Intel, G4, G5 867 MHz processor
512 MB RAM
Mac OS X 10.6
Intel processor
1 GB RAM
2012–2013 Computer-Based Testing System Minimum Specifications
The Certification Tool is to be completed by each school that will offer computer-based assessment(s) in the 2012-13 school year. (Please refer to Weekly Briefing #12320.)
Six sections to complete online:I. Computer SpecificationsII. Student Scheduling Plan III. Annual Checklist IV. Administration – Specific ChecklistV. School Computer-Based Assessment TeamVI. Submit to District Office
The Certification Tool Webcast Training is available at: http://oada.dadeschools.net/Screencasts/2013EOC/2012-13CBTSCREENCAST.html.
Certification Tool
The Certification Tool is to be completed twice in the 2012-13 school year.One Certification Tool for FALL and WINTER assessments: Fall 2012 FCAT/FCAT 2.0 Retake: Reading and
Mathematics Winter 2012 EOC Assessments: Algebra 1, Biology 1,
and Geometry
One Certification Tool for both SPRING assessments:Spring 2013 FCAT /FCAT 2.0 Retake: Reading and
Mathematics Spring 2013 FCAT 2.0 Reading and Mathematics Spring 2013 EOC Assessments: Algebra 1, Biology 1,
Geometry, US History, and Civics Field Test (selected schools only)
Certification Tool, cont.
PearsonAccess – www.pearsonaccess.com/fl A website used for almost all test preparation (e.g., PreID),
setup, administration, and reporting tasks.Correct test administration must be selected each time
you log in.Requires a username and password.Links to support materials: www.FLAssessments.com/
PearsonAccess will timeout after 14 minutes of inactivity.
PearsonAccess
PearsonAccess Training CenterAccessed from the Training
Center tab on PearsonAccess Home page.
Provides an opportunity to practice PearsonAccess tasks.
Manage the Infrastructure Trial in preparation for testing.
Requires a username and password.
PearsonAccess, cont.
Platform for Florida’s high-stakes computer-based statewide assessments.
Software application installed either on a file server or on each computer that will be used for testing.
Secure, locked-down browser.
Runs on tablets (except for iPads) but not recommended.
Note: This is the URL used for testing this Spring; it is different from the URL used for the Infrastructure Trial.
TestNav 6.9
Platform used to deliver accommodated forms (e.g., large print, zoom, color contrast, screen reader, assistive devices) for Florida’s high-stakes computer-based statewide assessments.
Must be installed on all computers that will be used by eligible students requiring accommodated CBT forms.
Local installation is required.New updated version must be installed.
Available for download at
www.FLAssessments.com/downloads. Note: Audio card and headphones are required for the computers that will run the Screen Reader accommodation.
TestHear
Proctor Caching is a process of loading or “caching” test content locally on a computer at the school level.
Does not require a separate caching server and can run on any workstation on the network that meets minimum requirements.
Proctor Caching software is provided by Pearson.Reduces test delays due to network congestion.Provides students with a more seamless testing
experience.Required for all FCAT/FCAT 2.0/EOC computer-based testing in Florida.
Proctor Caching
The Infrastructure Trial simulates a computer-based test using mock test items that will allow schools to test hardware and network capacity prior to the CBT operational administration. It should take approximately thirty minutes for
users to log in, supply random answers, and submit the test.Please note that the content of the test used for
the Infrastructure Trial is NOT intended for student practice.
Schools must set up the Trial to use all computers scheduled to be used concurrently during the Fall / Winter / Spring CBT administrations.The Infrastructure Trial is managed in the Training
Center. TestHear accommodated forms are available for the
Infrastructure Trial. These forms are larger (50 MB) than regular TestNav forms (2 MB).
Infrastructure Trial
Electronic Practice Assessment ToolProvides students an opportunity to practice using
the computer-based platform prior to testing. Available for download at
www.FLAssessments.com/ePATs. Download ePAT launcher separately from test
content.Accommodated ePATs are also available. Launcher
is bundled with the practice test content for accommodated forms.
Scripts and instructions for downloading ePATs are under the Resources tab.
Practice Test (ePat)
www.FLAssessments.com/ePATs
Resources tabProvide additional exposure
to the format of computer-based assessments
Viewing the tutorials does not replace the required practice test (ePAT) sessions
Student Tutorials
Student Tutorials, cont.
Current policy states that individuals who handle secure materials for statewide assessments (FCAT/FCAT 2.0/EOC) must have state- or district-level teaching certifications.
If there are instances in which a non-certified person will assist the test chairperson with preparing test materials, the District-Level Certification Form must be completed and submitted for approval.
District Level Certification Form
*Race to the Top Hard to
Measure:Physical
Education/Health and Visual
Arts
Kathleen Sierra, Supervisor
26
Funded through the Race to the Top (RttT) Grant – Three Year Project
Goals of the InitiativeDevelop high quality, standards-based assessments in
non-core content areasProvide Florida public and charter school districts with an
extensive bank of assessment itemsHigh Quality, standards-basedField testedVetted by Florida Educators
Once Item bank is fully operational, Florida Districts, schools, and teachers will access to the bank to generate assessments to meet their needs.
RttT District Developed Assessments
for Hard to Measure Content Areas
How were courses chosen?Through student enrollment and teacher
assignments
Content Areas being developed:Career and Technical EducationHealth Education K-12Physical Education K-12Performing ArtsVisual ArtsWorld Languages
RttT District Developed Assessments for
Hard to Measure Content Areas (cont.)
Timeline:2011-12 and 2012-13 school years
Develop Item SpecificationsDevelop Test Items, Blueprints, and test forms
2013-14 school yearField test
2014-15 school yearItem Bank fully operational for all Florida public and
charter school Districts
29
RttT District Developed Assessments for
Hard to Measure Content Areas (cont.)
Interim Assessment Program
Formative Classroom Assessment
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
To determine students’ baseline knowledge of the Next Generation Sunshine State Standards in Reading, Mathematics, Science and Social Studies.
To provide teachers with a quick snapshot of students’ initial knowledge and abilities to focus instruction.
To provide for state required progress reports in schools classified under the Differentiated Accountability System as Focus, Focus DD, Focus DDD, Prevent, and Priority.
Program guide and instructions for downloading the assessments are available on our website at http://oada.dadeschools.net/IAP/IAP.asp
Purpose of the BaselineBenchmark Assessments (BBA)
To monitor student progress on the Reading, Mathematics, Science and Social Studies Next Generation Sunshine State Standards (Fall, Winter, and Spring)
To provide teachers with classroom assessment tools that will provide reliable student-level benchmark information to inform instruction
To provide students with information on their progress on specific benchmarks
To provide schools with a reliable and valid assessment tool for state progress reporting.
Purpose of Interim Assessment
Tests
Interim Assessment tests administered three times per yearBBA, fall, winter, and spring
Alignment with the Next Generation Sunshine State Standards and District’s pacing guides
Benchmark Assessment Item Banks for Reading and Mathematics are available in ExamView
Data analysis and disaggregation, score reports, and item analysis are available in Edusoft
Overview of the InterimAssessment Program
Administration Windows Subjects and Grade LevelsBBA
August 20, 2011 – September 21, 2012
- Reading : Grades 3-10, - Mathematics: Grades 3-8; Algebra 1 ,
Geometry, - Science: Grades 5, 8, and Biology 1- United States History- Civics
Fall
October 29 – November 9, 2012
- Reading : Grades 3-10- Mathematics: Grades 3-8; Algebra 1,
Geometry, - Science: Grades 5, 8, and Biology 1
Winter
January 9 – January 20, 2012
- Reading : Grades 3-10, - Mathematics: Grades 3-8; Algebra 1,
Geometry, - Science: Grades 5, 8, and Biology 1- United States History
Spring
March 18 – March 22, 2012
- United States History- Civics
Interim Assessment Administration
Windows 2012-2013
All eligible students* must take the Interim Assessment in Reading, Mathematics, and Science.
All students enrolled in:Algebra I**Geometry**Biology I**U.S History**Civics**
Reading, Grades 3-10,*Mathematics, Grades 3-
8*Science: Grades 5 and
8*
*Students MUST take the test corresponding to the grade level in which they are listed in ISIS (unless they are enrolled in an EOC course in same content area).
**Or an equivalent course (see Appendix D, page 42 of the CBT Supplemental Program Guide)
Students to be Tested
Content- Area Paper-Based Tests Computer-Based
Tests Reading Grades 3-5, and 8 Grades 6, 7, 9, 10 , and
RetakeMath Grades 3, 4, 6-8 Grade 5
Algebra I Geometry
Science Grades 5, 8 Biology I
Social Studies U.S. History* Civics
U.S. History*
*Paper-based for Baseline only, computer-based in subsequent administrations.
Administration Format
* Longer assessments may be administered over a two-day time period.
Before the Test Administration Time After the Test
Approximately 10 minutes to passout testing materials
Reading*: Approximately 70
minutes
Approximately 10 minutes to collecttesting materials
Mathematics*: Approximately 75
minutes
Science*: Approximately 112
minutes
CivicsApproximately 60
minutes
U.S History*Approximately 90
minutes
Approximate Testing Times
Satisfactory Progress
This student demonstrated a satisfactory level of achievement on the content focus of the Next Generation Sunshine State Standards assessed during this instructional period. To attain high levels of achievement in this content area, the student must receive continued instruction on the challenging content and skills across the benchmarks designated for this grade level.
Limited Progress This student demonstrated a limited level of achievement on the content focus of the Florida Sunshine State Standards assessed during this instructional period. To attain high levels of achievement in this content area, the student must receive targeted interventions and remediation in the areas of concern, and continued instruction on the challenging content and skills across the benchmarks designated for this grade level.
Insufficient Progress
This student demonstrated an insufficient level of achievement on the content focus of the Florida Sunshine State Standards assessed during this instructional period. To attain high levels of achievement in this content area, the student must receive intensive interventions and remediation in the areas of concern, and continued instruction on the challenging content and skills across the benchmarks designated for this grade level.
Students who are likely to
score in FCAT levels 3 and
above
Students whose
performance was not
sufficient to predict success
Students who are likely to score below FCAT level 3
The Performance Levels for grades 3-10 reading, 3-8 mathematics, and algebra are outlined below.
Performance Levels
Useful reports that can be retrieved via Edusoft:
Class List Reports
Item Analysis Report
Item Response Report
Student Performance Report
Subgroup Report
Custom Group Report
Performance Band Report
Score Reports
To allow teachers to monitor student progress on the Reading and Mathematics Next Generation Sunshine State Standards
To provide teachers with classroom assessment tools that will provide reliable student-level benchmark information to inform instruction
To provide students with information on their progress on specific benchmarks
Item Bank Resources:
Item Bank Overview
Item Bank Newsletter
Download Examview Test Generator
Benchmark Assessment Item Bank
Felicia Mallory, Executive DirectorDenetra Collins, Staff Specialist
Comprehensive English Language
Learners (CELLA)
CELLA is a four-skill English language assessment measuring student proficiency in listening, speaking, reading, and writing.
Students in grades 1 – 12 are administered theListening, Reading, and Writing sections in groupsSpeaking on a one-on-one basis
Kindergarten students are assessed in all four areas on a one-to-one basis.
Unlike the FCAT, CELLA is not a timed test.
Overview of CELLA
Purposes of CELLAThe test results can provide:
Data useful for charting student progress over time,
Diagnostically useful information about students’ strengths and weaknesses in English,
The language proficiency levels of individual students can be used in making decisions regarding placement into, or exit from English for Speakers of Other Languages (ESOL) programs.
Identifying the proper ESOL level,Determining whether the District and schools met
Annual Measureable Achievement Objectives (AMAOs)
AMAO 1:Progressing in English language acquisition annual increases in the number or percentage of students making progress in learning English
AMAO 2:
Exiting or reaching English language proficiencyannual increases in the number or percentage of students attaining English language proficiency by the end of each school year
AMAO 3:
ELL-Adequate Yearly Progress (AYP)AYP for the ELL subgroup in demonstrating proficiency on grade-level in reading and mathematics on the FCAT
Annual Measurable Achievement
Objectives (AMAOs)
Important Dates for 2013
CELLA
Administration:Individual Group: March 4 - April 5, 2013
National Assessment ofEducational Progress
(NAEP)
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
Provides district-level data to participating urban districts
Provide a common yardstick to measure student achievement in comparison to
Other urban districtsStatesLarge Central CitiesNational Public SchoolsCensus Regions
Features of NAEP-TUDA
A common nationwide measure of student performance
Reports changes over time.
Provides results for the nation, states, and some urban districts.
Secondary indicator for state progress and allows national comparisons between states and large urban districts
Used by the President, Congress, and state leaders to set education policy
Purpose of the National Assessment
of Educational Progress (NAEP)/TUDA
A representative sample of schools and students are selected. Randomly selected students in grades 4,8, and12 are assessed in reading, mathematics, economics, and writing. Assessments include multiple-choice and constructed responses. Assessments are administered by NAEP field staff Assessments are 90 to 120 minutes, which include testing and answering background questions Accommodations are offered for ELL and SWD students. Results are reported by scale scores, subscales, achievement levels, subgroups, and geography
Overview of the NAEP
OperationalComponent
s
Grades/Ages
Reading 4, 8, 12
Mathematics 4, 8, 12
Pilots Grades
Reading 4, 8
Mathematics 4, 8
Technology and Engineering Literacy (TEL)
8
Special Research Studies
Varies
2013 Assessment Components
January 28,2013to
March 8, 2013
NAEP 2013 Assessment Window
Event Date
Provide School Information Form registration
October 11, 2012
NAEP field staff will contact schools to schedule an assessment visit
Mid-December, 2012
Pre-Assessment packet arrives in selected schools
January 2, 2013
Pre-Assessment visits January 2 – February 15, 2013
NAEP Testing Administration Window
January 28 – March 8, 2013
Important Dates for NAEP 2013
Subject and Grade Release Date
Math – Grade 4 and 8Reading – Grades 4 and 8
November 2013
NAEP-TUDA 2013 Release Date
International assessment sponsored by the National Center for Education Statistics (NCES) of the United States Department of Education.
Measures students’ mathematical, scientific, and reading literacy and their preparedness for adult life.
Assesses students who are about age 15.Student must be born between July 1, 1996
and June 30, 1997.Forty-two students are sampled per selected
school to take a two-hour assessment of math, reading, and science questions.Includes a student questionnaire.
Program for InternationalStudent Assessment (PISA)
Florida Competency on Personal Fitness (FCEPF)
Denetra Collins, Staff Specialist
Mayda Cabeza, Director I
Stanford Achievement Test, Tenth Edition (SAT-
10)
What is the SAT-10?Who is administered the SAT-10?
Grades K - 2, district-wide Important dates to remember
April 9-12, 2013
SAT-10
How do SAT-10 materials arrive to schools?
Is the SAT-10 a secure test?
SAT-10 Test Chairperson Training
February 2013
School level training of test administrators and proctors
Test Administration
Untimed
SAT-10
School Listing of Results (June)Must be picked up at ITS
Individual Student Reports (August)Delivered to schools
ResultsNot available on the Parent PortalPosted to SPI under the tab FCAT Scores
UsesReferral of students to the district’s gifted
program (Plan A or Plan B)Early identification of students who would be
considered at high risk for problems associated with reading comprehension
SAT-10 Results
Grade 3 Opportunities for Promotion
Section 1008.25, Florida Statutes (F.S.), Eliminating Social Promotion.Spring FCAT 2.0 Reading Spring Grade 3 Reading Student PortfolioAlternative Standardized Reading Assessment (ASRA)—
ITBS, Level 9, administered the last two weeks of schoolAlternative Assessment for Grade 3 Promotion (AAGTP)
—SAT-10, Primary 3, administered in Summer SchoolGrade 3 Mid-Year Promotion (GTMYP)—ITBS, Level 10,
administered in November
Grade 3Opportunities for Promotion
Administered during the 2nd semesterParticipants: All grade 3 students complete a portfolioScored at the school siteRepresents the 2nd opportunity for a student to be
promoted to grade 4Student must demonstrate mastery of the Next
Generation Sunshine State Standards (NGSSS)Training conducted via Screencast
Grade 3Reading Student Portfolio
Administered the last two weeks of schools. Participants:
Students scoring FCAT 2.0 Reading level 1 , students who did not participate in the Spring FCAT, and
Students without a passing Reading Portfolio Represents the 3rd and last opportunity within the
school year prior to retention. This administration of the ITBS, Level 9 is scanned via
Edusoft therefore results are available immediately. Students must score at or above a given cut score on
the ITBS to be considered for promotion. Training conducted via Screencast
Alternative StandardizedReading Assessment (ASRA)
Administered at the end of the summer Reading camp. Participants: Retained grade 3 students SAT-10, Primary 3 is scored locally through ITS Represents 4th opportunity for students to be promoted to
grade 4 Students must score at or above a given cut score on the
SAT-10 to be considered for promotion Training conducted via Screencast
Alternative Assessment for
Grade 3 Promotion (AAGTP)
Administered mid-NovemberParticipants: Students who “just missed” passing the
AAGTP, or other students at the principal’s discretion ITBS, Level 10 is scored at the school site via EdusoftRepresents 5th opportunity for retained 3rd grade
students to be promoted to grade 4Students must have a passing score on the ITBS,
representing 4th grade work.
Grade 3 Mid-Year Promotion (GTMYP)
Iowa Tests of Basic Skills (ITBS), Grades K-8
andIowa Tests of Educational Development
(ITED), Grades 9-12
The Iowa Tests
Used for Plan B Gifted Placement;Extension of Services in the ESOL program;Optional for Charter Schools Only, ESOL program
placement Scanned and scored via Edusoft Download a copy of the Iowa Tests Guidelines at:
Iowa Tests
http://oada.dadeschools.net/TestChairInfo/InfoForTestChair.asp
High schools must evaluate the college readiness of each student prior to grade 12. Eligible grade 11 students with the following criteria should test.
Student who score a level 2 or 3 on the reading portion of the Grade 10 FCAT 2.0
Students who score Level 2, 3 or 4 on the Algebra I EOC in Grade 10 (New).
Results of the assessment will be used to advise students.
(New) Spring 2013, FLDOE will conduct a special concordance study between the Algebra I EOC and PERT Mathematics.Select statewide sample in grades 6-10 (cohort
required to pass)Select students will take both assessments giving them
opportunities to earn a passing score once concordant scores are set.
HB1255 Section 1008.30, F.S.
Postsecondary Education Readiness Test Customized for Florida and aligned to Florida’s
Postsecondary Readiness Competencies Readiness Scores
Reading, 104Writing, 99Mathematics, 113
What is the P.E.R.T.?
P.E.R.T. Placement Test (3 sub-tests) identifies appropriate course placement in Reading, Writing and Mathematics
Computer Adaptive Larger font available for accommodation
30 questions per subtest25 operational items for placement score5 field test items
P.E.R.T. Components and Design
Testing window begins February 2013. Must test eligible grade 11 students. However, students
may demonstrate college-readiness via the ACT, SAT, or CPT. Students who do not demonstrate college readiness MUST
enroll in the appropriate postsecondary course in their senior year.
Students may not retake the PERT until the postsecondary coursework is completed. Students have only 2 attempts to take the PERT during their high school career.
Schools who have an overage of test units may use them to test select students for Dual Enrollment.
P.E.R.T. Implementation in High
Schools
Roll out in the Fall 2012. Fully operational by November
Will house all PERT scores Miami-Dade College (MDC) will be able to access
PERT test scores for students participating in Dual Enrollment and incoming students interested in enrolling at MDC after graduation.
Miami-Dade County Public Schools will continue to periodically upload PERT test scores on SPI for school access.
P.E.R.T. Score Repository
Florida Voluntary Prekindergarten (VPK)
Assessment
Office of Early Childhood Programs
Dr. San Juanita de la Cruz,District Supervisor
(305) 995-7645
Dr. Sally A. Shay, District Director
Test Security, Reporting Violations, Responding to State Inquiries, Caveon
Data Forensics
Standards, Guidelines, and Procedures for Test Administration and Test Security available at http://oada.dadeschools.net/TestChairInfo/InfoForTestChair.asp
Adopted by School Board
General Guidelines Florida Test Security Statute and
Rule
Security
DO DO NOT Provide secure storage Restrict access
3 or fewer keys Handle secure materials,
including affixing of labels in a limited-access area
Secure visual barriers on computer workstations
Maintain documentationChain of custodyBook assignment to students
Train test administrators and proctors
Monitor students during lunch break or extended breaks
Visual barriers for computer-based testing
Allow unauthorized access to materials or content
Open secure materials prior to date allowed
Allow untrained or uncertified staff to test students
Coach or provide assistance during test administration
View, review, copy, or debrief test content
General Security Guidelines
All school staff are responsible for ensuring appropriate administration procedures and reporting any irregularities.
Reporting procedures Specified in Standards, Guidelines, and Procedures
for Test Administration and Test Security Test Chairperson / Principal Student Assessment and Educational Testing Regional Center or District Administrative Office Appropriate investigative unit
Reporting Irregularities and
Security Violations
The FLDOE has contracted with Caveon Test Security to provide its Caveon Data Forensics™ for all statewide assessments.
Caveon will analyze data to identify highly unusual test results for two primary groups: Students with extremely similar test responses; and Schools with improbable levels of similarity, gains
and/or erasures. Flagging only the most extreme results.
Caveon Data Forensics
Michigan Educator Survey*, July 2011
34% felt pressure to change grades for the better
29% felt pressure to cheat on standardized tests
21% know of an educator that changed scores on student tests
8% admitted to changing students’ grades due to outside pressure
*Detroit Free Press, July 26, 2011
Troubling News
Examples of “Unusual” BehaviorVery high similarity among pairs or groups of test
takers (may be across test groups)Very unusual number of erasures, particularly
wrong to rightVery substantial gains or losses from one occasion
to anotherFocused on impact on scores
Caveon Data Forensics Process
Most Powerful & “Credible” StatisticMeasures degree of similarity between two or more
test instancesAnalyze each test instance against all other test
instances in the schoolPossible causes of extremely high similarity:
Answer CopyingTest CoachingProxy Test TakingCollusion
Similarity
Based on estimated answer changing rates from:Wrong-to-RightAnything-to-Wrong
Find answer sheets with unusual WtR answersExtreme statistical outliers could involve tampering,
“panic cheating”, etc.
Important! No student–level score invalidations based on erasure analysis; erasure analysis limited to school-
level flagging for additional review.
Erasures
Predict score using prior year information.Measure large score increases/decreases
against predicted scoreWhich score truly reflects the student’s actual
ability or competence?Extreme Gains/Losses may result from:
Pre-knowledge, i.e., “Drill It and Kill It”CoachingStudent development—visual acuity
Important! No student–level score invalidations based on gains analysis; gains analysis limited to school-level
flagging for additional review.
Unusual Gains/Losses
Chance of being hit by lightning = 1 in one million
Chance of winning the lottery = 1 in 10 millionChance of DNA false-positive = 1 in 30 millionChance of students being flagged for similarity,
but doing own work = 1 in a trillion
A Quick Discussion of Extreme
Results…
Similarity Analysis onlyMost credible, strongestNo flagging for erasures or gains
Invalidate test scores with Similarity Index ≥ 12
Student-Level Analysis
0.5
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
10
.51
1.5
12
.51
3.5
14
.51
5.5
16
.51
7.5
18
.51
9.5
20
.52
1.5
22
.52
3.5
24
.52
5.5
26
.52
7.5
28
.52
9.5
30
.5
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
FCAT Spring 2011 Grade 8 (Reading)N=196,866, Above 12=163 (0.08%)
Mean = 0.3
Similarity Index
Fre
qu
en
cyA Visual Representation
1.5
3.5
5.5
7.5
9.5
11
.5
13
.5
15
.5
17
.5
19
.5
21
.5
23
.5
25
.5
27
.5
29
.5
0
1000
2000
3000
4000
5000
6000
7000
FCAT Spring 2011 Grade 8 (Reading)With first bar removed
Index ≥ 12 is 15 standard deviations above the mean of 0.3
Similarity Index Value
Fre
qu
en
cyA Visual Representation
What does the mean look like?
0 5 10 15 20 25 30 35 40 450
1
2
Index=0.25; Scores=303 & 309
No Match Same Correct Same Incorrect
Item Number
Another Look at the Mean…
What does an index > 12 look
like?
0 5 10 15 20 25 30 35 40 450
2
Index=0.25; Scores=303 & 309
No Match Same Correct Same IncorrectItem Number
0 5 10 15 20 25 30 35 40 45012
Index=15.7; Scores=303 & 309
No Match Same CorrectItem Number
A Comparison…
M-DCPS Student-Level Invalidations
Retake FCAT RMS EOC 0
100
200
300
400
500
600
700
800
900
782
392 397
33 11 1113 3 5
Caveon Student Invalidation Outcomes, 2011-12
Invalidations Appeals Released
Similarity, Gains, AND Erasures Flagged schools will lead to school district
reviewUnsatisfactory reviews may lead to an
inspection by the Inspector General’s Office
School-Level Analysis
Dis
tric
t-S
ch
oo
l
Su
bje
ct
Nu
mb
er
of
Tests
Pass R
ate
Pass I
nd
ex
Mean
Sco
re
M4 S
imil
ari
ty
Rate
In
dex
Era
su
res R
ate
In
dex
Gain
s R
ate
In
dex
xxxx M 338 0.43 0.0 286.20 34.9 0.6 9.1yyyy R 672 0.37 0.0 306.49 24.3 0.1 0.1zzzzz M 532 0.60 0.0 300.62 21.3 0.3 4.5aaaa M 664 0.67 0.2 310.77 17.5 0.1 3.5bbbb M 364 0.97 45.0 376.53 0.0 0.0 6.1cccc M 512 0.52 0.0 292.00 14.8 0.1 5.3dddd R 338 0.18 0.0 280.79 13.5 0.3 1.0eeee M 830 0.87 39.6 344.23 0.0 0.0 2.2ffff R 534 0.32 0.0 299.73 13.1 0.0 0.2gggg R 458 0.35 0.0 303.05 1.5 12.7 0.0hhhh M 197 0.28 0.0 265.98 12.5 2.4 10.3
Sample School-Level Analysis
This program is important and timely.
It’s about valid test results.
Take Away
M-DCPS School Police
Detective Louis Nunez
Assessment, Research, and Data Analysis Website: http://oada.dadeschools.net/
Test Chairperson Website: http://oada.dadeschools.net/TestChairInfo/InfoForTestChair.asp
Testing Calendar (check for updates regularly): http://oada.dadeschools.net/TestingCalendar/TestingCalendar.asp
Guidelines and Tips for School Test Chairpersons:
http://oada.dadeschools.net/TestChairInfo/27GuidelinesandTipsforSchoolTestChairpersons.pdf
Standards, Guidelines, and Procedures for Test Administration and Test Security:
http://oada.dadeschools.net/TestChairInfo/29StandardsGuidelinesandProceduresforTestAdministrationandSecurity.pdf
District Resources
Student Assessment and Educational Testing: Dr. Sally A. Shay, District Director
Florida Comprehensive Assessment Test (FCAT/FCAT 2.0) and End-of-Course Assessments (EOC): Maria C. Bruguera, Director I Mara Ugando, Staff Specialist
Stanford Achievement Test-Tenth Edition (SAT-10), Grade 3 Promotion Opportunities, and Post Secondary Education Readiness Test (PERT): Mayda Cabeza, Director I
Comprehensive English Language Learning Assessment (CELLA), Interim Assessment, and National Assessment of Educational Progress (NAEP): Felicia Mallory, Executive Director Denetra Collins, Staff Specialist
Race to the Top, Hard to Measure Kathleen Sierra, Supervisor
Test Distribution Center Magaly Hernandez, Supervisor
Contacts