welcome to the spring 2017 post-test administration dtc ... · welcome to the spring 2017 post-test...
TRANSCRIPT
Welcome to the Spring 2017
Post-Test Administration DTC Meeting
Agenda
Post Administration Review 9:00 AM – 12:00 PM
Testing window overview
What went well?
Opportunities for improvement
Testing Materials
Computer-based testing
Next Steps
CCRA Preliminary Results
District Advisory Committee Meeting 1:00 PM – 3:00 PM
State Department of Education
Dr. Jeanene Barnett, Deputy Superintendent of Assessment &
Accountability
Craig Walker, Executive Director of State Assessments
Angela Bilyeu, Assistant Executive Director of State Assessments
Maria Harris, Director of Assessment and Data Literacy
Sarah Owens, Mathematics Assessment Specialist
Elizabeth Warren, ELA & Social Studies Assessment Specialist
Cora James, Science Assessment Specialist
Rebecca Logan, Executive Director of NAEP & International
Assessments
Kurt Bernhardt, Executive Director of Education Technology
Measured Progress and eMetric
Measured Progress
Tammy Bullock, Director Client Services
Julie DiBona, Portfolio Manager, Oklahoma
Sharman Lyons, Senior Program Coordinator
eMetric
William Kinnison, Project Manager
General Discussion
What went well? What are our
opportunities for
improvement?
What was new this year?
Manuals - Redesign
MetaMetrics Linking Study – Embedded Items
Spanish Audio Files
Breach vs. Equivalent (EQ) Forms
Materials Return Process Poster
All materials being returned together
Automatic UPS pickup days
New eMetric features
Test Materials
Service Desk Calls
Initial Orders
Manuals
Test Administration
Return Materials
Total calls (4/3/17 – 4/28/17) – 3,738
Service Desk Calls
Top 5 Call Categories1. Student
Enrollment2. Online Testing3. Shipping4. Content5. Technical issues
Initial Orders
High schools that showed enrollments for
elementary school test materials
Online testing schools that showed orders
for paper materials
Missing students in precode led to fewer
PSTGs than expected
Manuals
New look and feel
Bookmarks were great for electronic devices
Challenges with combining sample items;
especially for grade 3
Missing information
Header names
Explanation of the “Form” bubble
Implications that CBT section two had sample
items
Test Administration
Repeated lettering of answer choices
Items for the linking study looked out of place
Calculator concerns
Breach forms available to be assigned at the
beginning of the test window
Returning materials
The process is going more smoothly this year!
Fewer district RS labels were issued this year
Common return issues or errors:
Photocopies of Class ID Sheets
Mixed grades in scored material envelopes
Security forms from different schools mixed into one district box
Oklahoma School Testing Program Online Assessments
OSTP Online AssessmentsOnline Testing Summary
Students Scheduled for Test
Tests CompletedTotal Tests Completed Spring 2017: 493,050Total Tests Completed Spring 2016: 699,347
Total Number of Students Scheduled 2017: 231,855Total Number of Students Scheduled 2016: 299,801
OSTP Online AssessmentsGrades 6-8 Online Testing Counts - Operational
OSTP Online AssessmentsGrades 6-8 Online Testing Counts - Breach
Grade 10/History Online Testing Counts - Operational
OSTP Online Assessments
*2016 includes Spring English II, English III, Algebra I, Algebra II, Geometry and Biology
Grade 10/History Online Testing Counts - Breach
OSTP Online Assessments
*2016 includes Spring English II, English III, Algebra I, Algebra II, Geometry and Biology
OSTP Online AssessmentsWeek 1
Week 2
Fri Thurs Wed Tues Mon
20,000
20,000
30,00030,000
OSTP Online Assessments
10,000
30,000 30,000
12,500
Fri Thurs Wed Tues Mon
Week 3
Week 4
OSTP Online AssessmentsOperating Systems Used
20162017
• Site Readiness Testing
• 1898 Orgs
• 809 Participated
• 482 Certified
• 0 Minutes Datacenter Downtime
OSTP Online AssessmentsRisk Mitigation
OSTP Online AssessmentsNew Features for Spring 2017
• Bulk Upload of Student Accommodations• Export Students by Test Completion Status• Dashboard• Bulk Add and Edit Users• Deactivate/Reactivate User Accounts• Bulk Class Creation• Test Administrator Technology Guide• Add DOB and Middle Initial to Test Tickets• iOS Automatic Assessment Configuration• Default to Operational Program• Bandwidth Test Tool Tip• Students Sorted by Last Name
DTC Request!
DTC Request!
DTC Request!
DTC Request!
DTC Request!
DTC Request!
OSTP Online AssessmentsBulk Upload of Student Accommodations
OSTP Online AssessmentsExport Students by Test Completion Status
OSTP Online AssessmentsDashboard
OSTP Online Assessments
Deactivate/Reactivate User AccountsBulk Add and Edit Users
OSTP Online AssessmentsBulk Class Creation
OSTP Online AssessmentsKnown Issues and Proposed Resolutions
Component Issue Resolution
Kiosk Slow download Under investigation
Kiosk Notes and bookmarks not saved on pause
Investigating options
Kiosk Shading on items difficult to see Support guides, content, hardware
Kiosk Installing & launching ChromebookApplication in kiosk mode
Training, online tutorial
Portal Grade vs course level class Investigating options
Portal Student transfers in the middle of a test
Investigating options
Reporting Key DatesGrade 3 RSA reports 7/19/17
Final reporting 10/11/17
I don’t understand, why is reporting so late this year?What is standards setting and why does this delay reporting?
Standard Setting
Newly developed Oklahoma Academic Standards
were instructionally implemented in 2016
necessitating the need for assessment standard
settings.
The primary goal of the standard setting is to
determine the knowledge, skills, and abilities
(KSAs) that are necessary for students to
demonstrate in order to be classified into each of
the performance levels.
Standard Setting
Standard setting for Grade 3 RSA will take place in June, followed by a larger standard setting for OSTP Mathematics, ELA and Science in August. A total of 11 panels for the 18 grade/content combinations with 11 members each will be convened
Upon completion of the standard setting meeting, Measured Progress will submit a report to the SDE that documents all aspects of the standard setting process. Includes: Process information
Cut scores
Reporting data implications
Panelist evaluation information
Cut Score Approval
Panelist recommendations will be presented to the
Commission on Educational Quality &
Accountability (CEQA) for final consideration.
SDE is planning to send a letter to schools
(separate from the reports) and develop other
tools to explain to parents the new level of
expectations for mastering our state standards
and new performance expectations on the
statewide annual assessments.
CCRA Preliminary Results
CCRA Program 2017
Preliminary Data as of 5/10/2017
Preliminary Participation Counts• Estimated CCRA Counts:
o Total # Juniors (11th): approx. 44,919
o Minus 1% (449): 44,470
o Answer Documents Scored: 35,181
o Percentage that participated as of
5/10/2017: 79%
• Increased by 3% from SY 15-16
• Make-up Testers (not included in
preliminary reporting)
o SAT – 283 students
o ACT – 1200 students
• Total Answer Documents
Scored as of May 10, 2017:o 35,181
• 17, 287 – Male
• 17, 667 – Female
• 227 – Unknown (ACT only)
Preliminary Participation Counts• Total Answer Documents
Scored as of May 10, 2017:o 33,753
• 16,597 – Male
• 16,929 – Female
• 227 – Unknown
• Estimated ACT Counts:o Total # Juniors (11th): approx. 42,614
o Minus 1% (449): 42,188
o Answer Documents Scored: 33,753
o Percentage that participated as of
5/10/2017: 79%
50%49%
1%
Totals
Males Females Unknown
Preliminary Results –ACT Composite Scores
• 3 out of
33,753
received a
36 (perfect
score)
• Average
Composite
Score: 18.9o SY 15-16: 18.9
0
500
1000
1500
2000
2500
3000
5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
Average Scores
per Content
Total Possible Points per Section:
36
Avg. English Section Score:
SY 16-17 17.8
SY 15-16 17.6
Avg. Math Section Score:
SY 16-17 18.5
SY 15-16 18.7
Avg. Reading Section Score:
SY 16-17 19.7
SY 15-16 19.6
Avg. Science Section Score:
SY 16-17 19
SY 15-16 18.9
Benchmark Scores*
• These college courses
include:o English Composition,
o College Algebra,
o Introductory Social Science
courses, and
o Biology
• Scores on The ACT ®
subject-area tests that
represent the level of
achievement required
for students to have a o 50% chance of obtaining a “B”
or higher or
o about a 75% chance of
obtaining a “C” or higher in
corresponding credit-bearing
first-year college courses
*Based on a sample of 214 institutions and more than 230,000 students from across the United States, the Benchmarks are median course placement values for these institutions and as such represent a typical set of expectations.
College & Career Readiness Benchmarks
• The ACT ® English
Subtest Benchmark: 18
• College Course:
English Composition
• Met or Above
Benchmark: 46%
• Below Benchmark: 54%
• SY 15-16o Met or Above: 48%
o Below: 56%
46%54%
Students
Met or Above Benchmark Below Benchmark
College & Career Readiness Benchmarks
• The ACT ® Mathematics
Subtest Benchmark: 22
• College Course:
College Algebra
• Met or Above
Benchmark: 25%
• Below Benchmark: 75%
• SY 15-16o Met or Above: 23%
o Below: 77%
25%
75%
Students
Met or Above Benchmark Below Benchmark
College & Career Readiness Benchmarks
37%
63%
Students
Met or Above Benchmark Below Benchmark
• The ACT ® Reading
Subtest Benchmark: 22
• College Course: Social
Sciences
• Met or Above
Benchmark: 37%
• Below Benchmark: 63%
• SY 15-16o Met or Above: 35%
o Below: 65%
College & Career Readiness Benchmarks
24%
76%
Students
Met or Above Benchmark Below Benchmark
• The ACT ® Science
Subtest Benchmark: 23
• College Course:
Biology
• Met or Above
Benchmark: 24%
• Below Benchmark: 76%
• SY 15-16o Met or Above: 23%
o Below: 77%
College & Career Readiness Benchmarks
• The ACT ® STEM
Benchmark: 23
• Met or Above
Benchmark: 8%
• Below Benchmark: 92%
8%
92%
Students
Met or Above Benchmark Below Benchmark
Preliminary Participation Counts• Total Answer Documents
Scored as of May 10, 2017:o 1,428
• 690 – Male
• 738 – Female
• Estimated SAT Counts:o Total # Juniors (11th): approx. 2,305
o Minus 1% (449): 2,282
o Answer Documents Scored: 1,428
o Percentage that participated as of
5/10/2017: 63%
48%52%
Totals
Males Females
Preliminary Results –SAT Total Scores
• Average
Composite
Score: 938
• Scale:
400-1600
0
10
20
30
40
50
60
400
540
610
650
690
730
770
810
850
890
930
970
1010
1050
1090
1130
1170
1210
1250
1290
1330
1370
1410
1450
1490
Chart Title
Series 1
Average Scores per Content
Score Range per Section
200 - 800
Avg. Score: Evidence-based Reading & Writing
480.4
Avg. Score: Math 457.6
Score Range per Test
10 – 40
Avg. Score:Reading
24.4
Avg. Score: Writing & Language Arts
23.7
Avg. Score: Math 22.9
College & Career Readiness Benchmarks Defined
• Students are considered college- and career- ready
when their SAT section scores meet both the Math
and the Evidence-Based Reading and Writing
(EBRW) benchmarks.
• Students with an SAT score that meets or exceeds
the benchmark have a 75% chance of earning at
least a C in first-semester, credit-bearing college
courses.o SAT Math Section: Algebra, Statistics, Pre-Calculus, or Calculus Classes
o SAT EBRW Section: History, Literature, Social Sciences, or Writing Classes
*
• How Color-Coded Score Ranges Are Computedo The values for green test and sub-score score ranges are based on the
average test scores and sub-scores of all test-takers who met or exceeded
the corresponding benchmark for their grade.
o Yellow and red score range values are based on scores of test-takers who
did not meet the corresponding benchmark for their previous grade.
• Their average test scores and sub-scores determine the highest score in
the red range.
• Students who score above this average and below the green range fall
in the yellow range.
College & Career Readiness Benchmarks Indicators
Green The section score meets or exceeds the benchmark.
Yellow The section score is within one year’s academic growth of the benchmark.
Red The section score is below the benchmark by more than one year’s academic growth.
College & Career Readiness Benchmarks
• SAT Evidence Based
Reading & Writing
Section Benchmark:
480
• College Course: History,
Literature, Social
Sciences, or Writing
Classes
• Met or Above
Benchmark: 45%
• Below Benchmark: 55%
45%55%
Students
Met or Above Benchmark Below Benchmark
College & Career Readiness Benchmark Indicators
• SAT Evidence Based
Reading & Writing
Section Benchmark
Indicators
• Green: 45%
• Yellow: 8%
• Red: 47%
45%
8%
47%
Students
Green Yellow Red
College & Career Readiness Benchmarks
• SAT Mathematics
Section Benchmark:
530
• College Course:
Algebra, Statistics, Pre-
Calculus, or Calculus
Classes
• Met or Above
Benchmark: 23%
• Below Benchmark: 77%
23%
77%
Students
Met or Above Benchmark Below Benchmark
College & Career Readiness Benchmark Indicators
• SAT Mathematics
Section Benchmark
Indicators
• Green: 30%
• Yellow: 8%
• Red: 62%
30%
8%62%
Students
Green Yellow Red
11th Grade Benchmarks
• SAT Evidence Based
Reading & Writing
Section Benchmark:
460
• College Course: History,
Literature, Social
Sciences, or Writing
Classes
• Met or Above
Benchmark: 53%
• Below Benchmark: 47%
53%47%
Students
Met or Above Benchmark Below Benchmark
11th Grade Benchmark Indicators
• SAT Evidence Based
Reading & Writing
Section Benchmark
Indicators
• Green: 53%
• Yellow: 13%
• Red: 34%53%
13%
34%
Students
Green Yellow Red
• SAT Mathematics
Section Benchmark:
510
• College Course:
Algebra, Statistics, Pre-
Calculus, or Calculus
Classes
• Met or Above
Benchmark: 29%
• Below Benchmark: 71%
29%
71%
Students
Met or Above Benchmark Below Benchmark
11th Grade Benchmarks
11th Grade Benchmark Indicators
• SAT Mathematics
Section Benchmark
Indicators
• Green: 29%
• Yellow: 10%
• Red: 61%
29%
10%61%
Students
Green Yellow Red
CCRA Contacts
Tony Cortez
ACT Account Executive
918-527-9948
Melany Franklin
Director
State & District Partnerships
Office:512-721-1822
Cell: 512-667-0151
2017-2018 DRAFT Testing Calendar
It’s all about
student learning.
Period.
We appreciate the opportunity to serve you
and the students of Oklahoma!