Korea and US Korea and US

Comparison of Comparison of

Accountability Systems Accountability Systems

based on Highbased on High--stakes stakes

AssessmentAssessment

• Korea and US have similar policy direction in that they changed

educational assessment system to push forward government's

policies.

• Korean and US Education Ministers agreed to explore opportunities

for collaborative study(May 2011).

Background Background && PurposePurpose

Background

for collaborative study(May 2011).

2

Research Purpose • Comparing accountability systems based on high-stakes assessments

in Korea and US• To understand the weakness/strength of each country’s policy • To propose the direction of future educational policy

Table of ContentsTable of Contents

Section No. Title Slide No.

Ⅰ. Policy System for Basic Academic Achievement in Korea and US

4

Ⅱ.NAEA in Korea and State-level Assessment based on

7Ⅱ.NAEA in Korea and State-level Assessment based on NCLB in US

7

Ⅲ. Analysis of Policy Outcomes in Korea and US 15

Ⅳ. Policy Implications 23

3

Korea US• Proposed by MB

government’s Policy for basic academic achievement

• Proposed by President George W. Bush’s successfulexperiences in Texas

1. Comparisons of 1. Comparisons of Policy BackgroundPolicy Background

ⅠⅠⅠⅠ. Policy System

• To improve students’achievement

• To close achievement gap by SES, race, ELS students

5

ⅠⅠⅠⅠ. Policy System

2. Comparisons of 2. Comparisons of Policy PurposePolicy Purpose

Korea US• Guaranteeing all students’

progress toward achieving basic skills

• Closing achievement gaps • All students will meet equally

proficient levels in reading and math.

6

“Zero Below-Basic Plan”(School For Improvement (SFI))

math.

“100% Student Proficient Plan”

Korea US

Management National level State level

Target Grade 6th, 9th, 11th 3rd∼8th

SubjectKorean language, Mathematics, English (Social study, Science)

Reading, Mathematics

1. Comparisons of 1. Comparisons of Assessment systemAssessment system

ⅡⅡⅡⅡ. Assessment in KOR-US

SubjectEnglish (Social study, Science)

Reading, Mathematics

Achievement level 4 levels At least 4 levels

• Common grade and subject: 6th grade, Math

• For Comparison between KOR and US, California was chosenbecause California achieved good AYP results and it has high-stakes accountability system.

8

26

61.5

11.2

1.3

2128 26

19

6

26.6

49.8

22.4

1.2

2329

26

17

510

20

30

40

50

60

70

2. Co2. Comparisomparison of n of percentage at achievement levels in Korea and USpercentage at achievement levels in Korea and US

Target

level

Target

level

ⅡⅡⅡⅡ. Assessment in KOR-US

9

1.3 1.2

0

10

Advanced Proficient Basic Below

basic

Advanced Proficient Basic Below

basic

Far below

basic

Korea Califonia

(%)

‘09 ‘10 ‘09 ‘10 ‘09 ‘10 ‘09 ‘10 ‘09 ‘10 ‘09 ‘10 ‘09 ‘10 ‘09 ‘10 ‘09 ‘10

* Target level: level to accomplish in policy

Category Korea California

Typical Item for the

Basic Level

What is 470×0.01?

① 47 ② 4.7 ③ 0.47

④ 0.047 ⑤ 0.0047

What is 12/60 expressed in

Lowest terms?

① 1/8 ② 1/6 ③ 1/5

④ 1/4

A B C D① ② ③ ④ ⑤

3. Co3. Comparisomparison of Assessment tools for Mathn of Assessment tools for Math

ⅡⅡⅡⅡ. Assessment in KOR-US

Basic Level

RemarksCorrect answer. ②

Percentage of correct

answer 91%

Correct answer. C

Percentage of correct

answer 78%

response

distribution

(%)

A B C D

4 14 78 4

response

distribution

(%)

① ② ③ ④ ⑤

2.31 90.59 4.60 2.08 0.39

10

• Korean students need to understand the principle of multiplication using non-

integers, while US students calculate simple division.

Category Korea California

Typical Item for the

Proficient Level ① 40% ② 62.5% ③ 60%

What is 60% of 30?

① 1.8 ② 18 ③ 180

④ 1800

Which shows the

correct percentage

of the proportion of

black stone checkers

over the total

number of stone

checkers?

ⅡⅡⅡⅡ. Assessment in KOR-US

3. Co3. Comparisomparison of Assessment tools for Mathn of Assessment tools for Math

① 40% ② 62.5% ③ 60%

④ 37.5% ⑤ 16%

RemarksCorrect answer.②

Percentage of correct

answer 79%

Correct answer. B

Percentage of correct

answer 62%

response

distribution

(%)

A B C D

13 62 12 12

11

response

distribution

(%)

① ② ③ ④ ⑤

1.10 78.79 6.96 3.01 9.90

• US students calculate with simple question, while Korean students need to understand

the context in question to solve the problem.

US

• Content topics are introduced in early grades, then taught in more depth in and difficulty in

4. Co4. Comparisomparison of Math Curriculum of Korea and USn of Math Curriculum of Korea and US

ⅡⅡⅡⅡ. Assessment in KOR-US

Korea

• Content topics are introduced and taught with a great level of depth and difficulty in the more depth in and difficulty in

the higher grade.

12

depth and difficulty in the appropriate grade.

• Introduction grade of Math Content Topics

Grade diff.

Domain

earlier in California California&

Korea

earlier in Korea Not taughtin Cal

Not taughtin Kor3 2 1 1 2

Number and Operation 2 5 12 12 4 0 1 0

Geometry

ⅡⅡⅡⅡ. Assessment in KOR-US

4. Co4. Comparisomparison of Math Curriculum of Korea and USn of Math Curriculum of Korea and US

Geometry 0 1 7 8 0 2 5 0

Measurement 0 2 5 11 2 0 2 0

Probability and Statistics 1 1 1 3 0 0 3 8

Pattern and Solving problem 1 4 7 3 3 0 8 0

Total 4 13 32 37 9 2 19 8

13

* Number of topics in cell

• Weight of Math Content Domains(Grade 6)

Korea

Number and

OperationGeometry Measurement

Probability and

Statistics

Pattern and

Solving

problem

26.7 21.8 17.8 9.9 23.8

Algebra and Measurement Statistics,

Mathematical

(%)

ⅡⅡⅡⅡ. Assessment in KOR-US

4. Co4. Comparisomparison of Math Curriculum of Korea and USn of Math Curriculum of Korea and US

14

- Geometry and Measurement in US is relatively less than Korea.

- Probability and Statistics in Korea is much less than US.

USNumber sense

Algebra and

Functions

Measurement

and Geometry

Statistics,

Data analysis,

Probability

Mathematical

Reasoning

24.5 14.7 19.6 16.8 24.5

• Outcomes of TIMSS 1995, 1999, 2003 and 2007(8th grade)

1995 1999 2003 2007

Mean rank Mean rank Mean rank Mean rank

1. Comparison of achievement in TIMSS and PISA1. Comparison of achievement in TIMSS and PISA

ⅢⅢⅢⅢ. Analysis of Policy Outcomes in KOR-US

Math

609 3 587 2 589 2 597 2

492 18 502 19 504 15 508 9

Science

546 4 549 5 558 3 553 4

513 12 515 18 527 9 520 11

16

• Outcomes of PISA 2000, 2003, 2006 and 2009(15-year students)

ⅢⅢⅢⅢ. Analysis of Policy Outcomes in KOR-US

2000 2003 2006 2009

Mean rank Mean rank Mean rank Mean rank

Reading525 6 534 2 556 1 539 2～4

1. Comparison of achievement in TIMSS and PISA1. Comparison of achievement in TIMSS and PISA

* US was excluded because of problems in administration.

17

Reading504 15 495 18 m* m 500 11～25

Math547 2 589 2 547 1～4 546 3～6

493 18 504 15 474 32～36 487 26～36

Science552 1 558 3 522 5～9 538 4～7

499 14 527 9 489 24～35 502 19～29

• Comparing Cut-off Scores by Achievement Level in NAEA and PISA

PISANAEA

Level 6

Level 5

Level 4

670

608

546

638

553

Advanced

Proficient

2. Comparison of basic achievement in NAEA and PISA2. Comparison of basic achievement in NAEA and PISA

Basic cut-off score

ⅢⅢⅢⅢ. Analysis of Policy Outcomes in KOR-US

MathematicsMathematics

Basic

Below basic

Level 4

Level 3

Level 2

Level 1

546

483

421

359

553

456

Basic cut-off score• NAEA is higher for Math

compared to PISA 2009.

18

미국

대한민국

• Comparing cut-off scores by Math achievement Level in NAEA and PISA

2. Comparison of basic achievement in NAEA and PISA2. Comparison of basic achievement in NAEA and PISA

ⅢⅢⅢⅢ. Analysis of Policy Outcomes in KOR-US

17.1

8.0

1.9

26.3

17.7

7.8

level 4

level 5

level 6

미국

19

* Level 2: PISA 2009 Baseline(basic level)

(%)

Math basic level

cut-off for NAEA

8.1

15.3

24.4

25.2

1.9

6.2

15.6

24.4

0.0 5.0 10.0 15.0 20.0 25.0 30.0

below level 1

level 1

level 2

level 3

3. Outcomes 3. Outcomes of of NAEANAEA

• Change of percentage of below-basic students (overall percent of all subject)

Middle

High

Elementary

ⅢⅢⅢⅢ. Analysis of Policy Outcomes in KOR-US

Since change into census test in 2008, the percentage of below-basic students has averagely decreased up to more than half in all grades. ( 7.2% in 2008 ⇒ 4.8% in 2009 ⇒ 3.7% in 2010)

20

Elementary

‘08 ‘09 ‘10

(MEST announcement, November 2010)

(%)

• Performance of Schools for Improvement(SFI)(Change in percent of Below-basic)

SFI

3. Outcomes of NAEA3. Outcomes of NAEA

SFI SFI

ⅢⅢⅢⅢ. Analysis of Policy Outcomes in KOR-US

(%)

N=6,212 N=948SFI

N=3,131 N=302 N=2,221 N=410

21

‘09 ‘10 ‘09 ‘10 ‘09 ‘10‘09 ‘10 ‘09 ‘10 ‘09 ‘10

N: Number of students in 2010

• Change in Performance Achievement (California: Math, Grade 6)

4. Outcomes 4. Outcomes of of statestate--level assessment based on NCLBlevel assessment based on NCLB

49

52

50

52

54

56

58

60

proficient

ⅢⅢⅢⅢ. Analysis of Policy Outcomes in KOR-US

The 6th grade percentage of proficient and above level for Math has increased continuously in 2008, 2009, and 2010.

22

44

40

42

44

46

48

50

2008 2009 2010

proficient

level

(%)

• Korea needs to establish more various educational policies.

1. Policy Implications for Korea1. Policy Implications for Korea

ⅣⅣⅣⅣ. Policy Implications

- Establish medium and long-term policies

- Support system for non-beneficiary school

24

- Develop differentiate policy according to achievement level

- Measure year-to-year progress and develop the standard like “AYP”

- Support system for ADHD students

• US needs to develop common standards and short-term strategy.

2. Policy Implications for US2. Policy Implications for US

ⅣⅣⅣⅣ. Policy Implications

- Develop common achievement standards across states

- Develop short-term strategy like Korea

25

- Develop short-term strategy like Korea

- Establish specialized policy for below basic students like ‘SFI’ in Korea

3. Suggestion 3. Suggestion

• Benchmark each other’s strong points and outstanding examples

• Need to launch collaborative study between Korea

ⅣⅣⅣⅣ. Policy Implications

26

• Need to launch collaborative study between Korea and US (Study on policy effect from an economic view point)