sat suite o f assessme nts ess eport - maine.gov web pages 16-17/general...inter-rat inter-rate the...

Statistical R

SAT Suite o

SATEss

Main

SAT

April

2017

Report

of Assessments

AT Susay R

e

Schoo

2017

s Essay Report

uite oRepo

l Day A

of Asort

Adminis

sses

stration

sme

n

ents

of 2

4

Statistical R

SAT Suite o

Exec

This reMaine statistiinter-ra

Report

of Assessments

cutive S

eport summaduring the Acs and frequater consiste

s Essay Report

Summary

arizes the peApril 2017 aduency distribency.

y

erformance odministrationbutions, and

of 12,894 stun. This reporcorrelations

udents who rt is a summs of essay di

took the SAmary of descr

mension sco

AT Essay in riptive ores, and

of 2

4

Statistical R

SAT Suite o

Table

SAT Sui

SAT Es

Charact

Test for

Descrip

Essay A

Momen

Intercor

Reliabil

Standar

Tables...

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6Rater....

Table 7

Table 8Evidenc

Table 9

Table 9

Table 9

Table 1

Table 1Dimens

Table 1

Table 1Gender

Report

of Assessments

of Cont

ite of Asses

ssay....................

eristics of t

rms included.

ption of the an

Analyses......

ts and Score

rrelations.........

ity and standa

rdized differen

........................

. Essay Dime

. Descriptive

. Frequency D

. Frequency D

. Frequency D

. Frequency D

............................

. Correlations

. Correlationsce-Based Rea

a. Cross-tabu

b. Cross-tabu

c. Cross-tabu

0. Inter-rater

1. Inter-rater sion.....................

2. Inter-rater

3a. Essay Dimr Groups..........

s Essay Report

tents

ssments.....

............................

he Adminis

............................

nalysis sample

.......................

Distributions

............................

ard error of m

nces between

.......................

ension Score

Statistics for

Distributions o

Distributions o

Distributions o

Distributions o............................

s of the Three

s between theading and Wr

ulated Score D

ulated Score D

ulated Score D

Agreement b

Reliability (Pe............................

Consistency

mension Sco............................

........................

............................

strations of

............................

e.........................

........................

............................

............................

measurement

n groups.........

........................

Descriptions

Essay Dimen

of the Three E

of the Three E

of All Possible

of All Possible............................

e Essay Dime

e Reading Tesriting (ERW) S

Distributions

Distributions

Distributions b

between the T

earson Corre............................

(Kappa) betw

re Mean, Sta............................

........................

.............................

the SAT.....

.............................

.............................

........................

.............................

.............................

.............................

.............................

........................

and Score Ra

nsion Scores..

Essay Dimen

Essay Dimen

e Combinatio

e Combinatio.............................

ension Scores

st Score, the Section Score

between the T

between the T

between the T

Two Raters fo

lations) betwe.............................

ween the Two

ndard Deviati.............................

........................

............................

........................

............................

............................

........................

............................

............................

............................

............................

........................

anges..............

............................

sion Scores..

sion Scores b

ons of the Thr

ons of the Thr............................

s...........................

Writing & Lane, and the Dim

Two Raters fo

Two Raters fo

Two Raters fo

or Each Dimen

een the Two ............................

o Raters for E

ion, and Stan............................

........................

............................

........................

............................

............................

........................

............................

............................

............................

............................

........................

............................

............................

............................

by Rater..........

ree Essay Dim

ree Essay Dim............................

............................

nguage Test mension Scor

for Essay Rea

for Essay Ana

or Essay Writ

nsion.................

Rater Scores............................

ach Dimensio

ndardized Diff............................

.......................

............................

.......................

............................

............................

.......................

............................

............................

............................

............................

.......................

............................

............................

............................

............................

mension Scor

mension Scor............................

............................

Score, the res on Essay.

ading Score...

alysis Score...

ting Score......

............................

s for Each ............................

on.......................

ference betwe............................

of 2

.......5

..........5

.......6

..........6

..........6

.......6

..........6

..........7

..........7

..........8

.......9

..........9

.......10

.......11

.......12

res.13

res by .......15

.......16

.......16

.......17

.......17

.......18

.......19

.......19

.......19

een .......20

4

Statistical R

SAT Suite o

Table 1Racial/E

Bibliogra

About th

Appendi

A1. Pea

A2. Per

A3. Sing

A4. Sing

A5. Sing

A6. Sim

A7. We

Report

of Assessments

3b. Essay DimEthnic Groups

aphy/Refere

he College

ix A: Correl

arson product

rcentage of ag

gle-rater relia

gle-rater varia

gle-rater Stan

mple Kappa Co

ighted Kappa

s Essay Report

mension Scos..........................

ences..........

Board..........

lation Indice

t-moment cor

greement........

ability coefficie

ance..................

ndard error of

oefficient........

a Coefficient..

re Mean, Sta............................

........................

........................

es and Reli

relation coeff

............................

ent......................

............................

f measuremen

............................

............................

ndard Deviati.............................

........................

........................

iability Estim

icient.................

.............................

.............................

.............................

nt.........................

.............................

.............................

ion, and Stan............................

........................

........................

mates..........

............................

............................

............................

............................

............................

............................

............................

ndardized Diff............................

........................

........................

........................

............................

............................

............................

............................

............................

............................

............................

ference betwe............................

.......................

.......................

.......................

............................

............................

............................

............................

............................

............................

............................

of 2

een .......20

.....21

.....22

.....23

.......23

.......23

.......23

.......23

.......23

.......23

.......24

4

Statistical R

SAT Suite o

SAT Su

The SATdesignedassessmMath secTest), twsubscoreConventiMath). Fochoose toconcerns(ERW) a

SAT Ess

Test taketesting seprompt, o

A

Wpactheex(C

Two readWriting. UEnglish, test takedetails. Tthe authopersuasio

Report

of Assessments

uite of As

T Suite of Asd to measure

ment comprisction), three o cross-tests

es (Commanions, Heart oor the SAT, o also take ts the Essay, nd Math sec

say

ers opting toession to resor a nearly id

As you read t

eviden

reaso

stylistemotio

Write an essaersuade [hisuthor] uses hoice) to strehat your anassay shouldxplain how tCollege Boa

ders score eUnscorable ereceive a scr understood

The Analysisor builds an on. A high W

s Essay Report

ssessmen

sessments (e student reases two sectitests (the Rs (Analysis i

nd of Evidenof Algebra, Ptest takers athe optional although an

ction scores.

take the SAspond to thedentical one

the passage

nce, such as

ning to deve

ic or persuaon, to add p

ay in which ys/her] audienone or moreengthen the

alysis focuse not explain

the author burd, N.D.)

each essay, aessays, suc

core of 0. Thd the passags score reflecargument, in

Writing score

nts

(SAT, PSATadiness for cions (the Evieading Testin History/Soce, Words in

Problem Solvare given thrEssay are g

nalyses mak.

AT Essay rece essay. A pr, is used eve

e below, cons

s facts or ex

elop ideas an

sive elemenower to the

you explain hnce that [aute of the featu logic and pes on the mowhether you

uilds an argu

assigning a h as those te Reading sge, includingcts evidencencluding the e is given to

T/NMSQT®, college and idence-Base, the Writingocial Studiesn Context, Eving and Daree hours to given an addke use of the

ceive an addrompt appeaery time:

sider how [th

amples, to s

nd to connec

nts, such as ideas expre

how [the autthor’s claim].ures listed abersuasivenest relevant fu agree withument to pe

score from 1hat are off-toscore assessg the interplae in the essa

author’s useessays that

PSAT™ 10,postsecondaed Reading ag and Languas and Analys

Expression oata Analysis,

complete 15ditional 50 me Evidence-B

ditional 50 mars with ever

he author] u

support claim

ct claims an

word choicessed.

thor] builds a. In your essbove (or fea

ess of [his/hefeatures of thh [the authorrsuade [his/

1 to 4 to eacopic or writteses the evideay of the maay that the tee of evidencare focused

, and PSAT™ary educatioand Writing age Test, ansis in Scienc

of Ideas, Stan and Passpo54 items. Te

minutes. ThisBased Read

minutes at thery essay tex

ses

ms.

d evidence.

e or appeals

an argumensay, analyzeatures of youer] argumenthe passage.r’s] claims, bher] audienc

ch of Readinen in a languence in the ein themes a

est taker undce, reasoningd, organized

™ 8/9) is on. Each section and

nd the Math ce) and sevendard Englisort to Advanest takers wh report primaing and Writ

e end of thext. The follow

to

t to e how [the ur own t. Be sure . Your

but rather ce.

ng, Analysis,uage other thessay that thnd the impoderstands hog, and , and precise

of 2

the

en sh ced

ho arily ting

e SAT wing

and han he rtant ow

e;

4

Statistical R

SAT Suite o

that showthat have

For eachessay a sscores thscore of

This repoReading,scores ex

Charac

Test for

This repo2017. Te2017 SAacross a

Descrip

Before cothe test aSAT Suit2017, forlanguage

A score oin this refrequenc

Essay

Moment

Test takeskewnesscores. Acombinatoccurrenscores. SThese ra1), secon

Report

of Assessments

w a commane a variety of

h dimension,score of 0, ohe Essay. Th0 on one dim

ort contains , Essay Anaxamined in t

cteristics

rms include

ort summarizest takers weT Essay tesll forms and

tion of the

ompleting thadministratiote of Assessr the frequene and gende

of 0 is assignport (e.g., M

cy distribution

Analyses

ts and Sco

er performanss, and kurtoAll possible tions for threce, provides

See Tables 2ater sets are nd (Rater Se

s Essay Report

nd of languagf sentence s

the two rateor the two rathe third ratermension, the

information lysis, and Esthis report.

of the Ad

ed

zes the Essaere given twot. This reporall prompts)

analysis sa

he analyses on was cleansments Admincy of test taer.

ned to unscoMoments, cor

n of scores (

s

re Distribut

nce is descriosis), and frecombination

ee dimensions full informa2-6 for thesean aggrega

et 2), or third

ge, includingstructures an

er scores areters’ scores r’s score is den it is score

about the thssay Writing

dministrat

ays associato SAT formsrt summarize).

ample

contained inned to excludinistration R

akers in the i

orable essayrrelation, and(including al

tions

bed using dequency distns of the thren scores), a

ation on the je results. Notion of score

d (Rater Set

g the convennd consisten

e added to fodiffer by mo

doubled to yed 0 on all th

hree essay dg. Table 1 lis

tions of th

ted with the s. Three promes the data a

n this report, de any studeeport, Mainetem analysis

ys, so a scord inter-rater l three dime

escriptive stributions of see essay dimlong with thejoint distribu

ote that in sees across all3) score.

ntions of stant, precise w

orm the scoore than oneield the scor

hree dimensi

dimension scsts the descr

he SAT

SAT test formpts were aat the overa

the sample ents not in ge SAT Schos sample for

re of 0 is notreliability an

ensions).

tatistics (i.e.,scores for a

mension scoe frequency ution of the theveral tables l raters who

ndard writteord choice.

re. If one rate point, then re. If an essaions.

cores, specifriptions of th

rms adminisadministeredll level (i.e.,

available atgrade 11. Seol Day Admir this admini

t included innalyses), exc

, mean, stanll three essa

ores (512 posand percent

hree essay d‘Rater Setsprovided the

n English; a

ter gives an a third rateray receives

fically Essaye three Essa

tered in Aprd in the April aggregating

t 38 days aftee Table 3 ininistration, Astration by f

n all the analcept the

ndard deviatay dimensionssible tage of dimension ’ are mentioe first (Rater

of 2

and

r a

y ay

ril

g

ter n the April first

yses

ion, n

ned. r Set

4

Statistical R

SAT Suite o

Intercor

The Pealinear relcorrelatio

Reliabili

Reliabilityperson oobservedmany reainclude sother facof the prowith reliaother han

If the relimeasurescores. Upresentemargin oAppendix

Inter-rat

Inter-ratethe stabiraters wereliabilitythat at leraters ca

Percenta

Percentanumber oordinal aadjacent agreemetake into (Hallgren

Report

of Assessments

rrelations

rson producationship be

ons is in App

ity and stan

y refers to thor object. In td scores on tasons a perssituational vactors. If we cooportion of th

ability estimand, reliability

ability estimment (SEM)

Unlike reliabed in the unitf error arounx A5 for the

ter (single

er reliability ility of the Esere to score y coefficient cast two raten be estima

ages of agre

age of agreeof agreemennd interval d– where rat

ent, used as account agr

n, 2012). See

s Essay Report

t moment coetween two ependix A1. S

ndard error

he consistenthe context othe essay dison may scoariables, the onsider theshe measure

ates near 1 iny estimates n

ate indicates) can be conility estimate of measurend the obserformulas).

rater) relia

s the reliabilssay scores the same Escorrespondsrs score the ted.

eement

ment is an innts divided bdata, perceners are off bmeasures oreements due Tables 9a-

orrelation coessay or rate

See Tables 7

r of measur

ncy with whicof this reportimension scoore higher or

particular pase fluctuationment that is ndicating conear zero wo

s consistencnsidered a mes, which areement of the rved score. T

ability

lity of a singacross raterssay for a tes to a single

same Essa

ndex of interby the total otages of clos

by 1) can alsof inter-rate aue to chance-10 for these

oefficient proer scores. Th7 and 8 for th

rement

ch an instrumt, reliability reores, given nr lower on thassage assons in scoresnot errors. Rnsistent meaould indicate

cy of measurmeasure of ine standardizobserved sc

The SEM is

le rater scorrs: How mucest taker? Jurater, the esy for the sam

r-rater agreeobservationsse-but-not-e

so be compuagreement. Te. Therefore e results.

ovides an evahe formula fohese results

ment measurefers to the no change in

he Essay tesociated with to be errors

Reliability esasurement we fairly rando

rement, the nconsistencyzed to range core. The SEinversely re

ring the Essach would theust as with Estimation of me examine

ement. It can(see Appen

exact agreemuted and, aloThe percentit overestim

aluation of thfor calculatin.

ures some atconsistencyn actual abil

st on any givthe Essay, a

s, then reliabstimates ranwith very littlom estimate

standard ery in test takefrom 0 to 1,

EM represenlated to relia

ay. This reliae results diffeEssay reliabil

inter-rater ree. Then the

n be expressndix A2 for thment (e.g., pong with perctage of agree

mates the lev

he pairwise ng the Pears

ttribute of a y of test takelity. There ar

ven day. Theand a numb

bility is an indnge from 0 toe error. On ts of the attri

ror of ers’ observed, the SEM isnts the averaability (see

ability focuseer if two diffelity, althougheliability requreliability of

sed as the he formula).

percentage ocentage of eement does

vel of agreem

of 2

son

ers’ re

ese ber of dex

o 1, the bute.

d s age

es on erent h the uires f the

For of exact

not ment

4

Statistical R

SAT Suite o

Correlati

Correlatioscores iscorrelatio

Simple k

Cohen’s inter-ratetwo raterobservedexpectedformula fCohen’s indicatingfor these

Weighted

Weightedagreemepenalizinpenalize for calcuresults.

Standar

The test describedmean difmean difand 13b

Report

of Assessments

ion coefficie

on coefficiens a measure on coefficien

kappa statist

kappa coeffer agreemenrs while takind agreementd by chance for calculatinkappa coeffg complete r

e results.

d kappa stat

d kappa coeent between ng the disagr

the disagreelating weigh

rdized diffe

taker perford using the mfferences befferences eqfor Essay sc

s Essay Report

ent

nt between tof the inter-

nt is in Appen

tic

ficient (simplt between tw

ng into accout is defined bis defined b

ng Cohen’s kficients rangrandom agre

tistic

efficient (Cohtwo raters wreements. It ement are cted kappa c

rences bet

mance for smean and sttween the foual to 0.20 acore subgrou

the scores grater consisndix A1. See

le kappa stawo raters. It unt the possby cross-taby the margin

kappa coeffice from -1 to

eement, and

hen, 1968) iswhile correct

can be applomputed baoefficient is

tween grou

ubgroups, wtandard devocal and refeare small, 0.up moments

iven by two tency. The fe Table 11 fo

atistic; Cohencomputes th

sibility of agrulating ratingnal frequenccient is give1, with 1 ind -1 indicating

s an alternating the posslied to ordina

ased on the mgiven in App

ups

with samplesiation for all

erence group50 are medi

s and differe

raters on thformula for cor these res

n, 1960) is ahe observedreement occgs of the two

cies of ratingn in Appenddicating comg complete d

tive statistic sibility of agral ratings anmagnitude opendix A7. S

s greater thaEssay score

ps. Cohen (1ium, and 0.8nces.

e same essacomputing thults.

a statistic thad level of agrcurring by cho raters, and

gs from eachdix A6. Possmplete agree

disagreeme

that measurreement by cnd the weighof disagreemSee Table 12

n or equal toes and the s1988) sugge80 are large.

ay dimensiohe Pearson

at measuresreement betwhance. The d the agreemh rater. The ible values fment, 0 nt. See Tab

res the inter-chance and hts used to ment. The for2 for these

o 200, is standardizedests standard See Tables

of 2

on

s the ween

ment

for

le 12

-rater

rmula

d dized s 13a

4

Statistical R

SAT Suite o

Tables

Table 1.

Score

DimensScores

Rea

Ana

Wri

Note. Eac

Report

of Assessments

s

Essay Dim

sion

ading

alysis

iting

ch dimension

s Essay Report

mension Sc

What the sc

Compre

Unders

Correct

Use of t

Insightf

Evaluatpersuas

Relevan

Consist

Cohesiv

Effectiv

Variety choice;

Commafreedom

score is the s

core Descri

core reflects

ehension of

tanding of c

tness of inte

textual evide

fulness of an

tion of the ausion

nce and suff

tency of focu

veness; com

veness of pro

of sentencestyle and to

and of convem from errors

sum of two ra

iptions and

s

source text

entral ideas

rpretation of

ence

nalysis of so

uthor's use o

ficiency of su

us on releva

mmand of the

ogression of

e structures; one

entions of stas

ater scores, ea

d Score Ran

f text

urce text

of evidence,

upport for cl

nt features o

e language

f ideas throu

preciseness

andard writte

ach ranging fr

nges

reasoning,

aims

of the text

ughout the es

s of word

en English;

rom 1 to 4.

Scale

2 -

and

2 -

ssay

2 -

of 2

e

8

8

8

4

Statistical R

SAT Suite o

Table 2.

Score

Reading Rati

Rati

Dim

AnalysisRati

Rati

Dim

Writing Rati

Rati

Dim

N

Note: Zerodimensionone point,score.

Report

of Assessments

Descriptiv

ng 1

ng 2

ension Sco

s ng 1

ng 2

ension Sco

ng 1

ng 2

ension Sco

o dimension sn, the two rate, then a third

s Essay Report

ve Statistics

ore

ore

ore

scores were eer scores are rater scores t

s for Essay

excluded fromadded to form

the Essay. Th

y Dimensio

Mean

2.38

2.38

4.77

1.75

1.75

3.49

2.31

2.31

4.62

12,572

m the computam the score. Ihe third rater’s

on Scores

SD

0.72

0.70

1.26

0.77

0.76

1.37

0.74

0.73

1.32

12,572

ation of all fouIf the two rates score is dou

Skewnes

-0.31

-0.31

-0.37

0.57

0.57

0.49

-0.26

-0.27

-0.33

12,572

ur moments. ers’ scores difubled to yield

ss Kurto

-0.3

-0.3

-0.3

-0.6

-0.6

-0.7

-0.5

-0.5

-0.6

12,5

For each ffer by more tthe dimensio

of 2

osis

37

36

39

61

63

79

55

56

60

572

than on

4

Statis

SAT S

tical Report

Suite of Assessm

Table 3.

Score

0

2

3

4

5

6

7

8

Total

ments Essay Re

Frequency

Ess

Frequen

322

744

1,097

3,352

3,097

3,781

439

62

12,894

eport

y Distributi

say Readin

ncy Per

2

5

7 8

2 2

7 24

29

3

0

4 10

ons of the

g

rcent F

2.5

.77

.51

26

4.02

9.32

3.4

.48

0.00

Three Essa

Essay A

Frequency

322

4,315

2,211

2,979

1,884

1,030

143

10

12,894

ay Dimens

Analysis

Percen

2.5

33.47

17.15

23.1

14.61

7.99

1.11

0.08

100.00

sion Scores

t Frequ

32

1,0

1,2

3,4

2,8

3,4

40

3

0 12,

Page 1

s

Essay Writ

uency

22

084

258

414

883

490

04

39

894

1 of 24

ting

Percent

2.5

8.41

9.76

26.48

22.36

27.07

3.13

0.3

100.00

Statistical

SAT Suite

Table

Score

0

1

2

3

4

Total

Note. Ra

Report

of Assessments Es

4. Frequency

Rater Set

Frequency P

312

1,388

5,349

5,528

317

12,894 1

ater Set is an agg

ssay Report

Distributions

Essay Reading

t 1 R

Percent Frequ

2.42 33

10.76 1,2

41.48 5,4

42.87 5,5

2.46 29

100.00 12,8

regate of results

of the Three E

g

Rater Set 2

ency Percent

31 2.57

97 10.06

32 42.13

39 42.96

95 2.29

894 100.00

across all raters

Essay Dimensi

Rater S

t Frequency

312

5,467

4,875

2,112

128

12,894

providing the first

ion Scores by

Essay Analys

Set 1

Percent Freq

2.42 3

42.4 5

37.81 4

16.38 2

0.99

100.00 12

t (Rater Set 1) or

y Rater

sis

Rater Set 2

quency Perce

331 2.57

5,486 42.55

4,874 37.8

2,088 16.19

115 0.89

2,894 100.0

r second (Rater S

2 of 24

Rater

ent Frequency

7 311

5 1,773

8 5,426

9 5,116

9 268

00 12,894

Set 2) score.

Essay Wri

r Set 1

y Percent Fr

2.41

13.75

42.08

39.68

2.08

100.00

iting

Rater Set 2

requency Per

331 2.

1,724 13

5,454 42

5,138 39

247 1.

12,894 100

rcent

.57

3.37

2.3

9.85

.92

0.00

Statistical R

SAT Suite o

Table 5.Dimens

Essay Reading

0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4

Report

of Assessments

Frequencyion Scores

Essay Analysis

EssWrit

0 02 22 32 42 53 23 33 43 53 64 44 54 64 75 55 62 22 32 42 52 63 23 33 43 53 64 34 44 54 65 45 55 82 22 32 42 52 63 23 33 43 53 64 3

s Essay Report

y Distributis

say ting Frequenc

0 322 2 548 3 118 4 34 5 1 2 6 3 10 4 9 5 4 6 2 4 4 5 3 6 1 7 1 5 1 6 2 2 314 3 394 4 181 5 6 6 3 2 7 3 59 4 59 5 8 6 4 3 6 4 34 5 17 6 1 4 1 5 2 8 1 2 190 3 429 4 963 5 108 6 12 2 13 3 129 4 597 5 133 6 17 3 19

ons of All P

cy Percent

2.50 4.25 0.92 0.26 0.01 0.05 0.08 0.07 0.03 0.02 0.03 0.02 0.01 0.01 0.01 0.02 2.44 3.06 1.40 0.05 0.02 0.05 0.46 0.46 0.06 0.03 0.05 0.26 0.13 0.01 0.01 0.02 0.01 1.47 3.33 7.47 0.84 0.09 0.10 1.00 4.63 1.03 0.13 0.15

Possible C

EssaRead

44444444444455555555555555555555555555666666

Combination

ay ding

Essay Analysis

4 4 4 4 5 5 5 5 5 6 6 6 2 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 6 6 6 7 8 2 2 2 2 2 3

ns of the T

Essay Writing Fre

4 5 6 7 2 4 5 6 7 5 6 7 2 3 4 5 6 7 2 3 4 5 6 3 4 5 6 7 3 4 5 6 7 5 6 7 7 5 3 4 5 6 7 3

hree Essay

equency Perc

414 3.2205 1.554 0.41 0.01 0.012 0.026 0.223 0.1 0.01 0.03 0.01 0.04 0.051 0.4

300 2.3287 2.248 0.31 0.01 0.028 0.2

268 2.0371 2.875 0.55 0.0

269 2.0591 4.5243 1.8

2 0.02 0.029 0.2

274 2.178 1.3

5 0.016 0.43 0.34 0.01 0.01 0.03 0.064 0.5

120 0.9124 0.9

5 0.05 0.0

of 2

y

cent

21 59 42 01 01 09 20 18 01 01 02 01 03 40 33 23 37 01 01 22 08 88 58 04 09 58 88 02 02 22 13 38 04 12 33 03 01 01 02 50 93 96 04 04

4

Statistical R

SAT Suite o

Essay Reading

6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7

Report

of Assessments

Essay Analysis

EssWrit

3 43 53 64 44 54 64 75 45 55 65 75 86 46 56 66 76 87 57 67 72 52 62 73 64 54 64 75 55 65 7

s Essay Report

say ting Frequenc

4 63 5 157 6 184 4 86 5 297 6 680 7 7 4 23 5 191 6 963 7 40 8 1 4 4 5 40 6 650 7 48 8 3 5 2 6 13 7 8 5 2 6 3 7 2 6 2 5 3 6 14 7 19 5 13 6 43 7 49

cy Percent

0.49 1.22 1.43 0.67 2.30 5.27 0.05 0.18 1.48 7.47 0.31 0.01 0.03 0.31 5.04 0.37 0.02 0.02 0.10 0.06 0.02 0.02 0.02 0.02 0.02 0.11 0.15 0.10 0.33 0.38

EssaRead

77777777778888888888888

Tot

ay ding

Essay Analysis

6 6 6 6 7 7 7 7 8 8 4 4 5 5 6 6 6 7 7 7 8 8 8

tal

Essay Writing Fre

5 6 7 8 5 6 7 8 7 8 6 7 7 8 6 7 8 6 7 8 6 7 8

1

equency Perc

1 0.085 0.6

102 0.74 0.02 0.09 0.077 0.67 0.01 0.01 0.01 0.02 0.02 0.01 0.08 0.011 0.06 0.01 0.013 0.10 0.01 0.01 0.05 0.0

12,894 100

of 2

cent

01 66 79 03 02 07 60 05 01 01 01 02 02 01 06 09 05 01 10 08 01 01 04

0.00

4

S

S

TD

Ns

Statistical Repor

SAT Suite of Ass

Table 6. FreDimension

Essay

Reading 0 0 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4

Total

Note: Rater Sscore.

rt

sessments Essa

equency DiScores by

Essay

Analysis0 0 1 1 1 2 2 2 3 1 1 1 2 2 2 3 3 3 1 1 1 1 2 2 2 2 3 3 3 3 4 4 1 1 2 2 2 3 3 4 4

et is an aggre

ay Report

istributionsRater

s EssayWritin

0 1 1 2 3 1 2 3 3 1 2 3 1 2 3 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 3 4 2 3 2 3 4 3 4 3 4

egate of resul

s of All Pos

y ng

Freq

3

12

621

14

45

62

11

12

lts across all

ssible Com

Rater Se

quency

311 1

,036 258 6 11 61 14 2

677 ,304 193 37 ,551 495 15 76 1 8

463 517 1 2

665 ,015 3 1

107 ,661 63 12 10 1 3 1 17 3 93 93 12 94

2,894

raters providi

binations o

et 1

Percent

2.41 0.01 8.03 2.00 0.05 0.09 0.47 0.11 0.02 5.25 17.87 1.50 0.29 12.03 3.84 0.12 0.59 0.01 0.06 3.59 4.01 0.01 0.02 5.16 15.63 0.02 0.01 0.83 12.88 0.49 0.09 0.08 0.01 0.02 0.01 0.13 0.02 0.72 0.72 0.09 0.73

100.00

ing the first (R

of the Three

Freque

3310

9612616 8 51 8 2

7112,36716831

1,59548418 57 1 12 4225741 1

6592,024

2 0 81

1,68955 12 7 0 3 0 8 3 85 10018 78

12,89

Rater Set 1) o

Pa

e Essay

Rater Set 2

ncy P

7

5

4

9

94 1

or second (Ra

age 15 of 24

Percent

2.57 0.00 7.45 2.02 0.05 0.06 0.40 0.06 0.02 5.51 18.36 1.30 0.24 12.37 3.75 0.14 0.44 0.01 0.09 3.27 4.45 0.01 0.01 5.11 15.70 0.02 0.00 0.63 13.10 0.43 0.09 0.05 0.00 0.02 0.00 0.06 0.02 0.66 0.78 0.14 0.60

100.00

ater Set 2)

S

S

T

Na

Tto

N

Statistical Repor

SAT Suite of Ass

Table 7. Co

Note: Zero scacross all rate

Table 8. Cothe Evidencon Essay

Score

Reading Sc

Writing Sca

ERW

N

Note: Zero sc

Score

N

Dimension

Essay Re

Essay A

Essay W

Rater Set 1

Essay Re

Essay A

Essay W

Rater Set 2

Essay Re

Essay A

Essay W

rt

sessments Essa

orrelations o

ores were exers providing t

orrelations bce-Based R

cale Score

ale Score

ores were ex

Score

eading

nalysis

Writing

eading

nalysis

Writing

2

eading

nalysis

Writing

ay Report

of the Thre

cluded from tthe first (Rate

between thReading an

Essay Rea

0.57

0.57

0.59

12,572

cluded from t

Essay Reading

12,572

1

0.67

0.82

1

0.61

0.75

1

0.61

0.76

ee Essay Di

the computatier Set 1) or se

he Reading d Writing (

ading Ess

2

the computati

EssaAnaly

12,57

1

0.72

1

0.65

1

0.65

imension S

on of correlatecond (Rater

Test ScoreERW) Sect

say Analysis

0.62

0.62

0.65

12,572

on of correlat

ay ysis W

72

2

5

5

Scores

tions. Rater SSet 2) score.

e, the Writition Score,

s Essay W

0.62

0.62

0.65

12,57

tions.

Essay Writing

12,572

1

1

1

Set is an aggr

ng & Languand the Di

Writing

2

2

5

72

Pa

regate of resu

uage Test Simension S

age 16 of 24

ults

Score, Scores

S

S

TS

Ns

TS

Ns

Statistical Repor

SAT Suite of Ass

Table 9a. CScore

Rater S

0

1

2

3

4

Tot

Note: Rater Sscore.

Table 9b. CScore

Rater

0

1

2

3

4

Tot

Note: Rater Sscore.

rt

sessments Essa

ross-tabula

Set 1

tal

et is an aggre

Cross-tabula

Set 1

tal

et is an aggre

ay Report

ated Score

0

8 0.06

6 0.05

2 0.02

0 0.00

0 0.00

16 0.13

egate of resul

ated Score

0

8 0.06

8 0.06

0 0.00

0 0.00

0 0.00

16 0.13

egate of resul

Distributio

1

6 0.05

704 5.60

517 4.11

64 0.51

0 0.00

1,291 10.27

lts across all

e Distributio

1

100.08

4,21633.53

1,1208.91

1291.03

40.03

5,47943.58

lts across all

ons betwee

R

2

4 0.03

579 4.61

3,26125.94

1,56012.41

27 0.21

5,43143.20

raters providi

ons betwee

R

2

8 0

0.0

6 3

1,08.6

0

2,722.

3 957.5

3 45

0.3

9 8

4,838.

raters providi

en the Two

Rater Set 2

3

10.01

740.59

1,53512.21

3,69629.40

2331.85

5,53944.06

ing the first (R

en the Two

Rater Set 2

2

0 00 0

91 68

10

787 17

97

51 56

97

5 36

70

874 77

2,16

ing the first (R

Raters for

4

0

0.0

2

0.0

5 1

30.2

6 0

201.6

5 5

0.4

9 6

292.3

Rater Set 1) o

Raters for

3

0 .00 0

22 .97 0

933 .42 0

961 .64 0

72 .57 0

088 6.61 0

Rater Set 1) o

Pa

Essay Rea

4 T

0 00

2 02

11

0 24

54

06 64

54

7 45

95 35

11

or second (Ra

r Essay Ana

4

1 0.01

2 0.02

54

34 0.27

43

71 0.56

2

7 0.06

115 0.91

11

or second (Ra

age 17 of 24

ading

Total

19 0.15

1,365 10.86

5,345 42.52

5,526 43.95

317 2.52

2,572 00.00

ater Set 2)

alysis

Total

19 0.15

5,439 43.26

4,874 38.77

2,112 16.80

128 1.02

12,572 100.00

ater Set 2)

S

S

TS

Ns

Statistical Repor

SAT Suite of Ass

Table 9c. CScore

Rater

0

1

2

3

4

To

Note: Rater Sscore.

rt

sessments Essa

ross-tabula

Set 1

0

1

2

3

4

tal

et is an aggre

ay Report

ated Score

0

8 0.06

6 0.05

2 0.02

0 0.00

0 0.00

16 0.13

egate of resul

Distributio

1

70.06

1,0538.38

6084.84

490.39

00.00

1,71713.66

lts across all

ons betwee

R

2

6 3

0.0

3 8

645.1

4 3,326.

9 1,411.

0 23

0.1

7 6

5,443.

raters providi

en the Two

Rater Set 2

2

3 02 0

47 15 0

346 .61

1,11

435 .41

3,27

3 18

21

454 .38

5,40

ing the first (R

Raters for

3

0 0.00

43 0.34

,447 1.51

,437 7.34

211 .68

,138 0.87

Rater Set 1) o

Pa

Essay Wri

4

1 0.01

0 0.00

19 0.15

193 1.54

34 0.27

247 1.96

11

or second (Ra

age 18 of 24

ting

Total

19 0.15

1,749 13.91

5,422 43.13

5,114 40.68

268 2.13

12,572 100.00

ater Set 2)

S

S

T

A

P

P

P

T

N

TE

N

T

1

N

Statistical Repor

SAT Suite of Ass

Table 10. In

Agreement

Percent Agr

Percent Adj

Percent Mor

Total

Note: Zero sc

Table 11. InEach Dimen

Score

Essay Read

Essay Ana

Essay Writ

N

Note: Zero sc

Table 12. In

Score

Essay Rea

Essay Ana

Essay Writ

ASE repres

Note: Zero sc

rt

sessments Essa

nter-rater A

reement

acent

re than Adja

ores were ex

nter-rater Rnsion

ding

lysis

ting

ores were ex

nter-rater C

ding

alysis

ting

ents asympto

ores were ex

ay Report

Agreement b

acent

cluded from t

Reliability (P

Pearson Correlatio

0.57

0.61

0.62

12,894

cluded from t

onsistency

Kappa Sta

Simpl

Weight

Simpl

Weight

Simpl

Weight

otic standard e

cluded from t

between th

Essay

6

3

1

the computati

Pearson Co

n Sta

M

the computati

y (Kappa) b

atistic

le

ted

le

ted

le

ted

error.

the computati

he Two Rate

y Reading

61.45

36.92

1.62

00.00

on of inter-rat

orrelations)

andard ErroMeasuremen

0.47

0.47

0.45

12,894

on of correlat

between the

Value

0.37

0.46

0.42

0.51

0.40

0.50

on of correlat

ers for Eac

Essay

63

33

2

10

ter agreemen

) between t

or of nt

tions.

e Two Rate

ASE1

0.01

0.01

0.01

0.01

0.01

0.01

tions.

ch Dimensio

Analysis

3.47

3.85

2.68

00.00

nt.

the Two Ra

ers for Each

95% Co

0.36

0.44

0.41

0.50

0.39

0.49

Pa

on

Essay W

62.6

36.2

1.1

100.

ater Scores

h Dimensio

onfidence L

0

0

0

0

0

0

age 19 of 24

Writing

66

22

1

.00

for

on

Limits

.38

.47

.44

.52

.42

.51

S

S

Tb

N

Tb

S

E

E

E

S

E

E

E

Ne

Statistical Repor

SAT Suite of Ass

Table 13a. Ebetween Ge

Score

Essay Rea

Essay Ana

Essay Writ

Note: Zero sc

Table 13b. Ebetween Ra

Score

Essay Read

Essay Analy

Essay Writin

Score

Essay Read

Essay Analy

Essay Writin

Note: Zero scequal to or gre

rt

sessments Essa

Essay Dimeender Grou

ding

alysis

ting

ores were ex

Essay Dimacial/Ethnic

Wh

Mean

ding 4.97

ysis 3.70

ng 4.85

Wh

Mean

ding 4.97

ysis 3.70

ng 4.85

ores were exeater than 20

ay Report

ension Scoups

M

4

3

4

cluded from t

ension Scoc Groups

hite

n S.D. Me

1.19 4.

1.39 3.

1.25 4.4

hite

n S.D. Me

1.19 -

1.39 -

1.25 -

cluded from t0.

ore Mean, S

Male Mean

4.53

3.27

4.34

the analysis.

ore Mean, S

Black

ean S.D.

64 1.19

23 1.23

41 1.24

NHPI

ean S.D.

- -

- -

- -

the analysis. R

Standard D

S. D.

1.28

1.32

1.34

Standard D

Std. Diff.

Me

-0.28 4

-0.34 3

-0.36 4

Std. Diff.

Me

-

-

-

Results are o

Deviation, a

Mean

5.00

3.71

4.91

Deviation, a

Hispan

ean S.D.

.85 1.18

.49 1.34

.64 1.26

AIAN

ean S.D.

- -

- -

- -

only included i

nd Standar

FemaleS. D.

1.18

1.39

1.23

and Standar

nic Std. Diff.

M

-0.10 5

-0.15 3

-0.17 4

TStd. Diff.

M

- 4

- 3

- 4

if the non-wh

Pa

rdized Diffe

e Std. D

0.38

0.32

0.45

rdized Diffe

Asian

Mean S.D.

5.06 1.26

3.73 1.52

4.94 1.24

Two or mor

Mean S.D.

4.97 1.18

3.60 1.40

4.72 1.27

ite group sam

age 20 of 24

erence

Diff.

8

2

5

erence

n Std. Diff.

0.07

0.02

0.07

re races Std. Diff.

0.00

-0.07

-0.11

mple is

S

S

B

CM

CP

C

C

H

Statistical Repor

SAT Suite of Ass

Bibliograp

Cohen, J. (1Measuremen

Cohen, J. (1Partial Credi

College Boahttps://co

Cronbach, L334.

Haertel, E. H65-110).

rt

sessments Essa

phy/Refer

960). A Coent. 1960, 20,

968). Weight. Psycholog

rd, The (N.Dollegereadine

. J. (1951). C

H. (2006). RWestport, C

ay Report

rences

efficient of Ag, 37-46.

hted Kappa: gical Bulletin

D.) SAT essaess.collegeb

Coefficient a

Reliability. In CT: Praeger.

greement fo

Nominal Scn. 1968, 70, 2

ay. New Yorboard.org/sa

alpha and the

R. L. Brenna

r Nominal S

ales Agreem213-220.

k, NY: Authoat/inside-the-

e internal str

an (Ed.), Ed

cales. Educ

ment Provisio

or. Retrieved-test/essay

ructure of te

ducational m

cation and Ps

on for Scale

d from

ests. Psycho

easurement

Pa

sychological

ed Disagreem

metrika, 16,

t, fourth editi

age 21 of 24

l

ment or

297-

ion (pp.

S

S

A

TseettArw

Statistical Repor

SAT Suite of Ass

About the

The College success andeducation. Teducational ihe College Bhrough prog

Advanced Plresearch andwww.college

rt

sessments Essa

e College

Board is a md opportunityoday, the minstitutions aBoard helps grams and selacement Prd advocacy oeboard.org.

ay Report

Board

mission-drivey. Founded inembership a

and is dedicamore than s

ervices in corogram®. Theon behalf of

en not-for-prn 1900, the association iated to promseven millionollege readine organizatiostudents, ed

rofit organizaCollege Boais made up o

moting excelln students pness and colon also servducators and

ation that coard was creaof over 6,000lence and eq

prepare for allege succes

ves the educd schools. F

onnects studated to expa0 of the worquity in educ

a successful ss — includication commFor further in

Pa

dents to collend access to

rld’s leading cation. Eachtransition tong the SAT®

unity througnformation, v

age 22 of 24

ege o higher

h year, o college ® and the h

visit

Statistical R

SAT Suite o

Appen

A1. Pea

where the numb

A2. Perc

The perc

A3. Sing

The sing

correlatio

A4. Sing

The singcomputefrom the rater varirater sco

A5. Sing

The varia

A6. Sim

The simp

Where p0

expected

Report

of Assessments

dix A: Co

rson produ

and repber of test ta

centage of

centage of ag

gle-rater rel

le-rater relia

on between t

gle-rater va

le-rater variad on either tsame pool oiance is estim

ores:

gle-rater St

ance error of

ple Kappa

ple kappa co

0 is the obse

d probability

s Essay Report

orrelation

uct-momen

resent z-scoakers (Crock

agreement

greement (in

liability coe

ability coeffic

the first and

ariance

ance for athe first or seof raters, themated using

andard erro

f measurem

Coefficient

oefficient is g

erved probab

of agreeme

Indices a

nt correlatio

ores of obseker & Algina,

t

n percentage

efficient

cient for

second rate

a dimensionecond rater se two estimag the arithme

or of meas

ent for a sin

t

given by

bility of agree

nt and is com

and Reliab

on coefficie

∑

rved scores 1986)

e) is comput

.

r a given dim

er scores (re

n score or forscores. Becates are equi

etic average

12

urement

gle rater

1

1⁄

ement and i

mputed as

bility Esti

ent

and , res

ted as

mension is es

esult A1).

r the sum ofause both raivalent. In thof the varia

.

is given

.

s computed ∑ . .

mates

spectively an

stimated by

f dimension sater scores ahese analysences of the f

by:

as ∑.

nd N represe

the Pearson

scores can bare generatees, the singlefirst and sec

. pe is the

of 2

ents

n

be ed e-cond

e

4

Statistical R

SAT Suite o

The asym

Where pe

The asym

The conf

A7. Weig

The weigweights t

Where p0

is the exp

are c

The asym

Where pe

The asym

The conf

Report

of Assessments

mptotic varia

∑ 1

e is the expe

mptotic stand

fidence limits

ghted Kapp

ghted Kappato quantify th

0 is the obse

pected proba

constructed s

mptotic varia

∑

e(w) is the ex

mptotic stand

fidence limits

s Essay Report

ance of the s

1 . .

ected proba

dard error (A

s are compu

pa Coeffici

a coefficient he relative d

erved probab

ability agree

so that 0

ance of the w

∑

pected prob

dard error (A

s are compu

simple kappa

1

bility of agre

ASE) is the s

uted as

ent

is a generalifference be

bility agreem

ement and is

for al

weighted kap

∑ .

ability agree

ASE) is the s

uted as

a coefficient

1 ∑∑

1

eement and

square root o

1.96 ∗

ization of theetween categ

1⁄

ment and is c

s computed a

ll and

ppa coefficie

∑ .

1

ement and is

square root o

1.96 ∗

is computed

∑ .

∗

is computed

of the asymp

)

e simple Kapgories. It is c

computed as

as ∑

for al

ent is compu

1

∗

s computed

of the asymp

d as

.

d as ∑

ptotic varian

ppa coefficiecomputed as

s ∑

∑ ∑ . .

l , and

uted as

as ∑

ptotic varian

1

. . .

nce.

ent that usess

∑ . p

. The weigh

.

1

∑ ∑ . .

nce.

of 2

s the

pe(w)

hts

.

4

sat suite o f assessme nts ess eport - maine.gov web pages 16-17/general...inter-rat inter-rate the...

Documents