public schools of north carolina state board of education department of public instruction sbe...

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

SBE Issues Session on Testing/Accountability Issues

Lou Fabrizio Director

Division of Accountability Services

December 3, 2003

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


AgendaI. Are schools administering the right assessments?

• Purposes of the testing program• Administering the right assessments

II. Are standards sufficiently rigorous?• Performance standards for EOG/EOC• ABCs growth standards• Minimum skills competency test

III. Are changes needed in the current processes? • Maximizing benefits and minimizing intrusion• Quality control measures• Different administrations

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


I. Administering the rightassessments: purposes of testing

• It’s the law - G.S. 115C-174.10:• To assure all HS graduates possess

essential skills and knowledge• To provide a means of identifying

strengths and weaknesses in education process in order to improve instructional delivery

• To make system accountable to the public for results

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


North Carolina Testing Program State Board of Education

Required Testing for 2003-041

Grade Level Reading Math Social Studies Science Writing Other

3

Pretest & EOG

Pretest & EOG

NC Alternate Assessments3

4 EOG EOG Narrative6


5

EOG

EOG


6 EOG

EOG NC Alternate Assessments3

7 EOG EOG Argumentative6 NC Alternate Assessments3

8 EOG

EOG


Computer Skills4

9 English I2,9

Algebra I2,9

ELPS8,9 Physical Science2,9

Competency Test5,7

10 High School Comprehensive

Test NC Alternate Assessment3

Geometry2,9

High School Comprehensive

Test NC Alternate Assessment3

Biology2,9

NC Alternate Assessment3

Informational6

11 Algebra II2,9

Chemistry2,9

12 Physics2,9

NC Alternate Assessment3

1Tests currently administered as part of the North Carolina Testing Program are located in the boxes. Field tests may be administered annually in selected

subjects and grades at randomly selected sites. Some North Carolina students also participate in the National Assessment of Educational Progress (NAEP), the PSAT, the SAT, and the ACT. The grade 3 pretest is administered during the first three weeks of school; the end -of-grade tests are administered during the last three weeks of school. The locally-selected test dates must be on consecutive school days. 2The end-of-course tests are administered where the courses are taught within the final 10 days of school (or the equivalent for alternative schedules).

3Students with disabilities and limited language proficiency who do not participate in the grade 3 pretest, the end-of-grade tests, or the writing assessment

(grades 4, 7 and 10) must be administered a state-developed alternate assessment. Currently the state-mandated alternate assessments are the North Carolina Alternate Assessment Academic Inventory (NCAAAI, a curriculum-based checklist) and the North Carolina Alternate Assessment Portfolio (NCAAP- Reading, Mathematics, and Writing) for students with serious cognitive deficits. The alternate assessments are implemented in response to the federal Individuals with Disabilities Education Act (IDEA) and No Child Left Behind Act of 2001. 4

Students who entered the eighth grade in 1996-97 (Class of 2001 and beyond) are required to meet the proficiency standard on the computer skills multiple-choice and performance tests as a graduation requirement. A student with a disability who is following the Occupational Course of Study, as a graduation requirement, is required to meet the computer skills proficiency standard as stated in the student’s Individual Education Program (IEP). 5Students are required to pass the competency tests in reading and mathematics to get a high school diploma for all students entering the ninth grade in 1994-

95 and beyond and who are following the Career Preparation, College Technical Preparation, or College/University Preparation course s of study. 6The writing assessment at grades 4, 7, and 10 is being implemented statewide effective with the 2003-04 school year. Also effective with the 2003-04

school year, the North Carolina Writing Assessment at grades 4, 7, and 10 will be scored using the North Carolina Writing Assessment Scoring Model . Scores from the Writing Assessments as grades 4, 7, and 10 will be included in the ABCs performance composite effective with the 2003-04 school year. 7Implementation of the North Carolina Exit Exam has been delayed due to action of the N.C. General Assembly.

8The Economic, Legal, and Political System (ELPS) End-of-Course test will be administered to all students who are enrolled in the ELPS course for credit.

9Students with disabilities and with limited English proficiency who are enrolled in a course for credit but are not able to participat e in the EOC tests are required to participate in the EOC NCAAAI pilot.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• Other NC statutory mandates:• G.S.115C-12 (9a): EOC Tests – “for the

minimum courses required for admissions to the [UNC] constituent institutions”

• G.S.115C-174.11:(b): Competency, Computer Skills(c)(1): EOG Tests, Writing(c)(2): NAEP

• G.S.115C-105.35: ABCs accountability model, Gr. 3 pretest

Administering the rightassessments: purposes (cont’d.)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• Federal mandates:• No Child Left Behind Act of 2001 (Title I):

• End-of-Grade (EOG),

• NC Alternate Assessment Portfolio (NCAAP),

• NC Alternate Assessment Academic Inventory (NCAAAI),

• HS Comprehensive Tests,

• National Assessments of Educational Progress (NAEP)

• NCLB Title III• IPT Language Proficiency Test

• Individuals with Disabilities Education Act (IDEA)• Alternate assessments and accommodations

Administering the rightassessments: purposes (cont’d.)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Administering the right assessments• Our current assessments meet federal and

state mandates, except state EOC requirement for tests in

• English II, III, and IV

• Foreign Languages (which ones?)

• Fourth Year Mathematics (which one?)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• Tests under development:• EOG Science (5, 8, and ? for NCLB)

• North Carolina Computerized Adaptive Testing System (NCCATS) in Reading, Mathematics and eventually Science (IDEA)

• G.S. 11C-12(9b) called for exit exam(s)• General Assembly suspended development

pending report on compliance with NCLB

. . . the right assessments (cont’d.)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• SB 1115, SL 2002-126: Exit Exam • “The Board shall not adopt any revisions prior to

reporting them…to the [Joint Legislative Education Oversight Committee (JLEOC)].”

• USED has not made a decision regarding • Algebra I and Biology; issue with the

Occupational Course of Study

• Algebra I exemption for students whose IEP identifies the student as learning disabled in mathematics and states that this disability will prevent the student from mastering Algebra I


PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• Exit exam options: • Revise HS Comprehensive Tests (what to

test?, grade?)• Compilation of EOC Tests• Use external measures, e.g., SAT, ACT

• HB 801 – Modify the state Competency Testing Program – this needs further study before the SBE can make a report to the JLEOC


PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• IDEA Reauthorization Impact

• Timeline?

• Policy Revisions?

• Changes to tests and accommodations?


PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Administering the right assessments: Considerations

• Too many tests?

• Testing the right areas?

• Should other tests be added?

• Should some tests be deleted?

• Algebra I exemptions?

• Occupational Course of Study?

• NC Computerized Adaptive Testing System?

• Exit Exam?

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• Performance standards are:

• an external interpretation of test results

• based on judgments• Tests themselves are:

• rigorous • aligned to the curricula

II. Are proficiency standards sufficiently rigorous?

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• Standards can be adjusted

• Various methods produce divergent results

• Best to adjust with new curriculum implementation

. . . standards sufficiently rigorous? (cont’d.)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


• Changes now will impact NCLB 100% proficient goal by 2013-14

• Changes will affect the Student Accountability Standards

• Changing standards vs. maintaining trends

• Operational vs. non-operational conditions for setting standards

• Using sample vs. census (statewide) data

Standard-setting Dilemmas

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


ABCs Growth Standards• K-8 model implemented in 1996-97 :

• Same students over time • Same subjects; Reading and

Mathematics• Some drift over time (normal and

desirable)• Changes would be more appropriate

after 2003-04 (when we will have two years in a row of students taking the new tests in Reading)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Growth Standards (cont’d.)• Prediction formulas in grades 9-12

• EOC: same students over time; different subjects (measures)

• Competency passing rate: same students over time, same measure

• ABCs Dropout Rate: different groups of students, 3 successive years

• College University Prep/College Tech Prep: different groups of students,

3 successive years

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Growth Standards (cont’d.)

• Results for students last year who scored at achievement levels III and IV during the prior school year – are they making expected growth with similar success rates as other students?

• Can we rely on the current formulas to do this?

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Growth Standards: Issues to consider• To continue with the same model or use

additional variables

• To use longer time frames for growth, to capture prior history

• To use more complex analytical methods (must have adequate measures and database for chosen framework)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Growth Standards: Guiding principles– should we continue to follow them?

• Same students (longitudinal)

• Equity or fairness (schools, relative to previous performance quartiles)

• Simplicity (conciseness)

• Precision or theoretical framework (reliability and validity)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Growth Standards: Other considerations

• Keep statewide growth as a benchmark, tailor standards for individual schools

• Retain ability for schools, LEAs to have some foreknowledge of standards (goals, targets)

• Retain the ability for LEAs to calculate and verify results locally with a variety of tools (software - DPI and commercial products), calculators

• Earliest timeframe for model implementation is 2005-06 due to model development and APA process

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Minimum Skills Competency Test Standard

• Mathematics and Reading Comprehension Tests at the 8th grade level

• Computer Skills Tests measure attainment of curriculum objectives through 8th grade

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Competency Standard: Issues to consider

• Should competencies be raised to high school level? (which grade?)

• Which competencies should be measured? (Reading, Mathematics, Science, Social Studies, etc.?)

• Should same competencies apply to all students? (What about the Occupational Course of Study?)

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


III: Are changes needed in the current processes?

• Maximizing benefits of assessment while minimizing intrusion

• Move to embedded field testing (starting with Mathematics)

• Use volunteer samples for item tryouts whenever necessary

• Retain same number of tested items to accommodate embedded items

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Are changes needed in the current processes (cont’d.)

Trade-offs

• Fewer items Reduced reliability

• Stricter item selection More items required

• More forms More staff required to maintain quality

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Other Assessment Methods

• Computer Based Testing (CBT)• Linear presentation

• Same length

• Computerized Adaptive Testing (CAT)• Adapts to user’s ability level

• Shorter test

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Computer Administration

• Allow for multimedia items (e.g., simulations of lab experiment, viewing movie clip and answering questions about it; or using computer to perform a task)

• Difficult to standardize for large-scale, high-stakes assessments.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Computer Administrations: Issues to consider

• Connectivity in schools and equity issues

• Location and cost of server hosting

• Bandwidth requirements

• Security and confidentiality

• School resources (management, funds, personnel, access)

• Technical assistance

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Quality Control

• Current item and test development processes have multiple checks in place

• Error rate far below industry standard

• Embedding increases chance of errors

• Use of computers (of varying quality within schools) impacts test delivery

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Quality Control (cont’d.)

• QC for scoring and data collection is highly person-intensive

• There are LEA, RAC/RCC (Regional Accountability Coordinators/Regional Computing Consultants), and DPI processes of quality control

• QC constitutes largest portion of time necessary to produce ABCs report and AYP results

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


Are changes needed? Main points:• What are the stakes involved in making

changes?

• What are the consequences of error?

• Do last year’s major successes warrant changing the model?

public schools of north carolina state board of education department of public instruction sbe...

Documents