1

ATEAM

ATEAMKenneth Barbour

Faye Atkinson, Rhonda Carlyle, Lori Ivey, Phyllis Price, Jim Rhew, Kevin Skelton, George Stubblefield

2

Our Purpose

• Support the design Assessments of Student Achievement and implementation of Assessment

• Collect and Analyze Achievement Data• Report Student Achievement• Design, develop and implement a system to analyze

student academic performance outcome data for purposes of professional development, and technical assistance geared to improve instruction and student performance

3

Our Customers• Accountability Services

– Test Development– Testing Policy and Operations

• LEA Test Coordinators• Regional Accountability Coordinators (RACs)• Regional Computing Consultants (RCCs)• State Board of Education• Public• Local Education Agencies (LEAs)• Schools• Researchers• Department of Justice• Federal Reporting

4

Data Consumer List

• State Board of Education

• Legislative / Judicial Requests

• Federal Submissions

• Researchers

• LEAs / Schools

• Title I & III

• Public

5

Our Partners

• Portfolio Managers in other DPI Areas• Accountability Services Division• Technical Outreach for Public Schools (TOPS)• Regional Accountability Coordinators

– Regional Computing Consultants• Test and Accountability Coordinators/Directors

6

Our Governance• Accountability Services Division• Data Management Group (DMG)• US Department of Education • No Child Left Behind (NCLB)

– Adequate Yearly Progress (AYP)• Accountability Basics Control (ABCs) of Public

Education• Accountability Configuration and Control Board

(CCB) - Advisory• State Board of Education (SBE)• Legislature

7

Services of ATEAM

• Applications

• Analyses

• Reports

• Accountability Database (ADB)

• Technical Support– Customer Support (RITS - External)– Customer Support (RTS - Internal)

• Training

8

Products List• WinScan• ABCTools• Request Tracking System (RTS)• Regional Issue Tracking System (RITS)• NC WISE Test Upload System• Online Testing Irregularity Submission System (OTISS)• Individual Student Report (ISR)• Data Collection, Verification and Reporting (DCVR)• Collection Date Submission System • Accountability Database (ADB)• Writing Instruction System (WIS) - TOPS• NC Registration - TOPS• Online Testing Systems - TOPS • Online Ordering System - TOPS• Testing News Network (TNN) – TOPS• Request Tracking System - TOPS

9

ATEAM

Other DPI Authoritative Data Sources

NCLB

Public

ABCs

NC WISESchools

RACs/RCCs

ATEAM INTERFACES

Test Development

Pearson and Other Contractors

TOPS

Testing PolicyAnd Operations

District Accountability /

Testing Coordinator

80% of theReport Card

10

Accountability Services Structure• Accountability Division

– Lou Fabrizio, Director of Accountability, Policy and Communications

– Gary Williamson, Director of Accountability Operations– Tammy Howard, Chief, Test Development Section– Sarah McManus, Chief, Testing Policy and Operations

Section• ATEAM• Contractors

– TOPS (Technical Outreach for Public Schools) – Pearson– Other Contractors

11

Application: WinScan• Windows application used by the Central Office

of LEAs

• WinScan is a software program that is used to scan, score and process statewide tests

• WinScan identifies and corrects errors

• WinScan produces local reports– Class Rosters– Goal Summaries– Label Reports– Many Others

12

Application: ABCTools• Windows application used by the Central Office of LEAs• Data file loading for MYSQL database using managed file transfer• Manages accountability data for NCLB and ABCs with MYSQL data

warehousing system• Provides predefined SQL statements to assist users with queries• Provides student level audit of data elements used for NCLB and ABCs• Provides reports to assist school staff in identifying students at risk as

result of proficiency levels• Provides classroom scale score averages• Provides access SQL Query Tool• Provides precoding for answer sheets used for ABCs and NCLB• Provides verification reports and processes for test data submitted to

NCDPI for NCLB and ABCs • Provides links to accountability information

13

Application: Online Testing Irregularity Submission System (OTISS)

• A web based APEX Application

• Allows for the collection, approval workflow, tracking and archiving of testing irregularities

• Includes school level access

14

Application: Individual Student Report

• Windows application used by the Central Office of LEAs

• Prints parent/teacher reports• Subject to peer review by USED• NCLB regulates the reports• Includes scale scores, percentiles, exit

standards, Lexile measures, Quantile measures, achievement level descriptors, and sub score reporting

15

Application: Data Collection, Verification and Reporting (DCVR) andAccountability Database (ADB)

• Manages file transfer, data validation and reporting • Manages the collection of data sets from schools and LEAs,

validates and merges the data into an accountability database

• Provides interactive error reporting and data collection • Provides for the data cleaning and aggregation of data into

the accountability database (ADB) • Insures a unique ID across all the student information

collected from various data sources, including NC WISE • Encompasses all business logic to create the source data for

the ABCs and AYP reports • Provides the source data for all accountability data collected

for the EdFacts requirement

16

ADB / DCVR OVERVIEW

File Submission

And Validation

ETL Processes

ABCTools WinScan Other

Status Reports

DPIAuthoritative

Sources

Static Data And Configuration

Business Rule

Processes

Table HierarchyFollows BusinessRule Progression

To Enable Auditing And Analysis

ADB

Reports Analysis Data Requests LEA Exports

17

Application: Collection Date Submission System

• A web based APEX application• Allows the LEAs to enter their start of

school, 20th day, First Day of Fall testing, First Day of Spring testing, and last day of school dates

• Allows the ATEAM to use these dates to pull the enrollment and historical test data for the schools

• Decreases the load on the NC WISE servers and the LEA Testing Coordinators

18

Application: NC WISE Test Upload System

• Provides ability to import student test score information into NC WISE

• SCO file (from WinScan) is processed during a nightly batch job

• Summary / error reports are available through a web interface

• Also includes capabilities for import of SAT and CTE test data

19

Analyses:• Ad hoc analyses for customers• Growth Model development and validation• Reconciliation of reporting anomalies • Assist standardized test data certification• Apply linking tables to scales• Supply DPI Business Units with analyses results

– District School Transformation – Charter Schools– NC Virtual Public Schools– Title I & III– Others

20

Reports: Major Report Systems

• NCLB/AYP

• ABCs

• Disag

• NC School Report Card (SRC)

• Greenbook - TOPS

• EdFacts – (via EDEN)

• Considated State Performance Report (CSPR) (EdFacts)

21

Reports: Reports ListList of Reports for ABCs/AYP Release• Executive Summary• Low Performing Schools• Secondary Analyses

– Number and Percentage of School Level Targets Met by Subgroup

– Number and Percent of Schools by Percent of Targets Missed

– Number and Percent of Schools by Number of Targets Missed

– Number of Schools That Met AYP Using at Least One of the Following Special Flexibilities: Safe Harbor, Confidence Interval, Growth (Schools may be counted on more than one row)

– Number of Schools, Number Meeting AYP and Percent Meeting AYP by Number of Targets

– Number of Schools that did not make AYP: Number due to one subgroup

– Number of Schools that missed AYP due to two subgroups

22

Reports: Reports List Continued -

• Title I School Improvement List• LEA Improvement Status• LEAs in Corrective Action• Impact on Performance Using Retesting Results• 4-Year Cohort Graduation Rate (Current Year)• 4-Year Cohort Graduation Rate (Previous Year)• 5-Year Cohort Graduation Rate (Previous Year)• Evolution of the ABCs• AYP Schools Under Review• Possible ABCs Violations of the 95% Rule• Possible Violations of AYP 1% Cap

23

Report Tables List• Preliminary End-of-Grade General Test Multiple-Choice Test Results

Statewide Performance Summary in Reading and Mathematics Grades 3-8

• Preliminary End-of-Grade NCCLAS Test Results Statewide Performance Summary in Reading and Mathematics Grades 3-8

• Preliminary End-of-Grade NCEXTEND2 Test Results Statewide Performance Summary in Reading and Mathematics Grades 3-8

• Preliminary End-of-Grade NCEXTEND1 Test Results Statewide Performance Summary in Reading and Mathematics Grades 3-8

• End-of-Grade General Test Multiple-Choice Test Results, Statewide Characteristics and Performance of Grade 3-8 Students

• Preliminary End-of-Grade General Test Multiple-Choice Test Results Statewide Performance Summary in Science Grades 5 and 8

• 1985-86 to 2007-08 End-of-Course General Test Multiple-Choice Test Results, Statewide Performance Summary

• Preliminary End-of-Course NCCLAS Test Results Statewide Performance Summary

• Preliminary End-of-Course NCEXTEND2 Occupational Course of Study Test Results Statewide Performance Summary

24

Technical Support: Request Tracking System (RTS)

• Internal Request Tracking is an integral part of the total IT responsibility of the ATEAM

• It serves as the primary method to communicate changes requested to the “owner” of the work with applications and systems and the corresponding resolutions back to the customer

• It maintains consistency in logging and reporting requests, utilizing standardized procedures

25

Technical Support: Regional Issue Tracking System (RITS)

• External Request Tracking is an integral part of the total IT responsibility of the ATEAM

• It serves as the primary method to communicate regional changes requested to the “owner” of the work with applications and systems and the corresponding resolutions back to the customer

• It maintains consistency in logging and reporting requests, utilizing standardized procedures

26

Authoritative Sources

27

Technical Outreach for Public Schools (TOPS)• EOC On-line Testing• Paper and Pencil Testing• NCExtend1• Writing Instruction System (WIS)• Data Analysis Group• NC Registration• On-line Ordering System• Test Item Development• Test Item Bank• Testing News Network (TNN)• Logistics

– McKimmon Center– Blue Ridge Location– Warehouse (Other)

28

Writing Instruction System (WIS) - TOPS • Application resulting from the Grade 7 Online Writing Project • A pilot using the Moodle freeware system configured and hosted

at North Carolina State University (The pilot ended 6/30/09) • Consists of 3 online Professional Development modules for

Teachers, 2 on demand writing assignments and 2 content specific writing assignments for students in grades 4 and 7 – 7th graders were allowed to develop and upload the writing

assignments to an online portfolio – Teachers were then able to access and grade the assignments

online • Based on the results, the State Board of Education approved

the DPI recommendation to extend the pilot for an additional year and to open participation to additional LEAs. The project is extended through June 2010.

29

NC Registration - TOPS

• Schools and school systems use the system to indicate students: – who are eligible for testing– to schedule test administrations within

predefined testing sessions for groups of students

30

Online Testing System - TOPS

• Provide same assessment as paper and pencil online format

• Used for the eight EOC tests

• Browser based

• Need Adobe Flash to use

31

Online Ordering System - TOPS

• System developed that allows the LEA Test Coordinator to place, review, and update orders for testing materials

• System includes materials such as:– Braille– Read Aloud – One Item per Page – Large Print Test

32

Testing News Network (TNN) - TOPS

• TNN is a closed information and collaboration service sponsored by the Accountability Services Division and hosted by TOPS

• Users are testing and accountability staff of LEAs in NC Public Schools

• Provides secure information dissemination of local school system test coordinators

• Provides indexing of critical events, policies, reports, communications, process details and software upgrades for the testing program

33

Training System - TOPS

• NC Testing Program Online Training– Item Development Basics– Developing EOC Mathematics– Grades 3-5 Items– Grades 6-8 Items– Developing EOC Items– EOC English 10 Items– Additional Courses will be added

34

NC FALCON - TOPS Training

• North Carolina Formative Assessment Learning Communities Online Network

• Opening as a pilot in January, 2010 to train teachers, educators and administrators on how to do formative assessment

35

Archive System - TOPS

• Electronic archive of test materials for the NC Testing Program– EOGs – EOCs – NCEXTEND1 – NCEXTEND2 – Grade 10 Writing Assessment

36

Help Desk System - TOPS

• Provides HelpDesk Support for LEA / School Staff for all TOPS systems

37

Hardware Required for Standardized Testing and Accountability at the LEA Level

• In 1992, the NC General Assembly revised the NC Public School Testing Program legislation authorizing NCDPI to implement the end-of-grade and the end-of-course tests (reference Public School Laws of NC, Article 10A, Part 2, 115C-174.11)

• A key component to the successful implementation of the NC Testing Program is the timely scanning and scoring of the students' answer documents and the printing of the students' reports

• To accomplish these tasks, the NCDPI purchases, upgrades, maintains hardware, and develops software for computers, scanners, and printers

38

Software Required for Standardized Testing and Accountability at the LEA Level• ABCTools

• Adobe Acrobat

• Individual Student Reports

• Microsoft Office Professional

• Antivirus

• SFTP Secure Shell Client

• WinScan

• WinZip

39

EQUIPMENT ROLES EQUIPMENT RESPONSIBILITIES

Accountability IT Team: Equipment specificationsEquipment and/or software compatibilityGhost imagesEquipment set up assistance

Accountability Operations Director: Approve purchases

Accountability Purchasing Agent: Purchase orders

DPI Fixed Asset Clerk: Provide asset tagsMaintain official inventory (indicating where equipment is permanently housed)Provide inventory list for annual auditState Surplus contact

Accountability Equipment Acquisitions, Maintenance and Inventory Management:

Maintain internal inventory (indicating where equipment is temporarily housed)Assist with surplus/transfer formsLiaison between Accountability and Fixed Asset ClerkEnsure surplus/transfer forms completed and submitted as neededApprove temporary transfer of equipmentFacilitate annual audit

Lead RAC: Liaison between RACs and Accountability Inventory ManagementProvide RAC input on equipment issues

RCC: Equipment set up assistance (LEAs)Equipment maintenance assistance

RAC: Maintain regional equipment inventoryReconcile regional and fixed asset inventory annuallyRequest equipment transfers

LEA Test Coordinator: Day to day equipment maintenance and upkeepFamiliarity with equipment and its proper useReport damages/breakdowns to RAC/RCC

TOPS: Provide shipping of equipmentMaintain equipment poolRepair equipment

40

Training for: ABCTools • Monthly training for registered participants held at DPI • Demonstration of loading data into ABCTools, if

needed. ATEAM may have processes to move data into a set of folders for transformation.

• Demonstration of where to find data on web (Disag and others) and what the reports provide

• Demonstration on published public reports• Demonstration on Data Security• Demonstration of other data transformation use for

school improvement

41

Training for: Precode Plus

• Precodes answer sheets for EOC/EOG tests

• Designed to assist Testing Coordinators with specialized precoding needs

• Allows user to precode from roster listing, precode from a pick list of students or precode using an external file for a specific list of students

42

Training for: Reports • Historical Audit Rosters (Provides Scale Scores and Levels)• Membership/Course Codes (Provides membership rosters by subject)• Student Profile Report• Exit Standards Roster• Graduation Reports• ABC Summary• AYP Reports• Growth Summary Report (Classroom Scale Score Averages)• Students Repeating Courses Report• Student Rosters with Demographics from NC WISE• Classroom Scale Score Averages (Growth Summary from ABCTools)• Rerostering of WinScan Records for EOG/EOC • Goal Summary Reports in WinScan• Growth Target Reports (Why and why not to use these reports) • Detail AYP Reports for Web• Detail ABCs Reports for Web

43

ATEAM (Extra)

• Revised 01/21/10 RMC