On

Dev

elop

ing

A

Com

preh

ensi

ve S

cien

ce

Ass

essm

ent P

lan

for D

elaw

are

Ric

hard

J. S

have

lson

Stan

ford

Uni

vers

ity

May

14,

200

1

Ove

rvie

w

Fram

ewor

k fo

r Sys

tem

Out

puts

(Dire

ct P

rodu

cts)

Inpu

ts(R

esou

rces

)

Proc

esse

s(R

esou

rce

Use

)

Out

com

es(G

oals

)

The

perf

orm

ance

mea

sure

(“ou

tput

“) is

an

indi

cato

r of t

he d

esire

d be

havi

or, n

ot th

e be

havi

or it

self.

In b

usin

ess,

the

outp

ut m

easu

re (e

.g.,

reve

nue

or st

ock

pric

e) is

a v

ery

clos

e pr

oxy

to v

alue

d ou

tcom

es.

It gu

ides

bus

ines

s dec

isio

ns a

nd a

ctio

ns.

You

can

’t m

anag

e a

busi

ness

if y

ou c

an’t

mea

sure

it’s

out

com

e.

In e

duca

tion,

out

com

es a

re m

any

and

deba

ted.

The

out

com

e in

dica

tor,

mos

t ofte

n a

mul

tiple

-cho

ice

achi

evem

ent t

est,

is b

ut a

pro

xyfo

r the

des

ired

outc

ome.

Whe

n th

is in

dica

tor b

ecom

es a

n en

d in

itse

lf, a

nd it

doe

s in

educ

atio

n, w

ell-i

nten

tione

d ac

coun

tabi

lity

may

ver

y w

ell d

isto

rt th

e sy

stem

it w

as in

tend

ed to

impr

ove.

Sour

ce: M

arch

(199

4)

Educ

atio

n In

dica

tor S

yste

m

Fram

ewor

k fo

r Cog

nitiv

e O

utpu

ts

(Kno

win

g th

e “t

hat”

)

(K

now

ing

the

“how

”)

(K

now

ing

the

“whi

ch,”

“w

hen,

” an

d “w

hy”)

Prof

icie

ncy

Exte

nt(H

ow m

uch?

)

Stru

ctur

e(H

ow is

it o

rgan

ized

?)

Oth

ers

(Pre

cisi

on?

Effic

ienc

y?A

utom

atic

ity?)

Cog

nitiv

eC

ogni

tive

Tool

s:To

ols:

Plan

ning

Plan

ning

Mon

itorin

gM

onito

ring

Dom

ain-

spec

ific

prod

uctio

n sy

stem

s

Prob

lem

sch

emat

a/st

rate

gies

/op

erat

ion

syst

ems

Low

Hig

h

Dom

ain-

spec

ific

cont

ent:

•Fa

cts

•C

once

pts

•Pr

inci

ples

Ass

essm

ent o

f Dec

lara

tive

Kno

wle

dge:

Mul

tiple

-Cho

ice-

-TIM

SS P

op. 2

Air

is m

ade

up o

f man

y ga

ses.

Whi

ch g

as is

fo

und

in th

e gr

eate

st a

mou

nt?

A. N

itrog

en

B. O

xyge

n

C. C

arbo

n D

ioxi

de

D. H

ydro

gen

Ass

essm

ent o

f Dec

lara

tive

Kno

wle

dge

Stru

ctur

e: E

leve

n-Y

ear-

Old

’sC

once

pt M

ap

wat

erra

in

river

scl

ouds

soil

sun

ocea

ns

is fa

lling

wat

er

com

esfr

om

cont

ain

goes

into

rive

rs

flow

to

cont

ain

shin

es o

n

From

Whi

te &

Gun

ston

e: “

Prob

ing

Und

erst

andi

ng”

(199

2, p

. 16)

Ass

essm

ent o

f Pro

cedu

ral K

now

ledg

e:

Perf

orm

ance

of a

Day

time

Astro

nom

y In

vesti

gatio

n

Flas

hlig

htSt

icky

To

wer

s

Stud

ent

Not

eboo

ks

and

Penc

ils

Stud

ents

are

ask

ed to

mod

el th

e pa

th o

f the

sun

from

su

nris

e to

sun

set a

nd u

se d

irec

tion,

leng

th, a

nd

angl

es o

f sha

dow

s to

sol

ve lo

catio

n pr

oble

ms.

Proc

edur

al K

now

ledg

e:

Perf

orm

ance

Ass

essm

ent:

TIM

SS P

op. 2

PUL

SEA

t thi

s sta

tion

you

shou

ld h

ave

A w

atch

A st

ep o

n th

e flo

or to

clim

b on

Rea

d A

LL

dire

ctio

ns c

aref

ully

.

You

r ta

sk:

Find

out

how

you

r pul

se c

hang

es w

hen

you

clim

b up

and

dow

n on

a st

ep fo

r 5 m

inut

es.

Thi

s is w

hat y

ou sh

ould

do:

•Fi

nd y

our p

ulse

and

be

sure

you

kno

w h

ow to

cou

nt it

. IF

YO

U C

AN

NO

T FI

ND

YO

UR

PU

LSE

ASK

A

TEA

CH

ER F

OR

HEL

P•

Dec

ide

how

ofte

n yo

u w

ill ta

ke m

easu

rem

ents

star

ting

from

whe

n yo

u ar

e re

st.

•C

limb

the

step

for a

bout

5 m

inut

es a

nd m

easu

re y

our p

ulse

at r

egul

ar in

terv

als.

1.M

ake

a ta

ble

and

writ

e do

wn

the

times

at w

hich

you

mea

sure

d yo

urpu

lse

and

the

mea

sure

men

ts y

ou m

ade.

2.H

ow d

id y

our p

ulse

cha

nge

durin

g th

e ex

erci

se?

3.W

hy d

o yo

u th

ink

your

pul

se c

hang

ed in

this

way

?

Ass

essm

ent o

f Stra

tegi

c K

now

ledg

e:

Men

tal M

odel

s

•(A

) A ro

cket

is m

ovin

g al

ong

side

way

s in

deep

spac

e, w

ith it

s eng

ine

off,

from

poi

nt

A to

poi

nt B

. It

is n

ot n

ear a

ny p

lane

ts o

r oth

er o

utsi

de fo

rces

. Its

eng

ine

is fi

red

at

poin

t B a

nd le

ft on

for 2

sec

whi

le th

e ro

cket

trav

els f

rom

poi

nt B

to p

oint

C.

Dra

w

in th

e sh

ape

of th

e pa

th fr

om B

to C

. (S

how

you

r bes

t gue

ss fo

rthi

s pro

blem

eve

n if

you

are

unsu

re o

f the

ans

wer

.)•

(B) S

how

the

path

from

C a

fter t

he e

ngin

e is

turn

ed o

ff o

n th

e sa

me

draw

ing.

B

C

A

C

BA

Cor

rect

Inco

rrec

t

Sour

ce: C

lem

ent,

J. (1

982)

. St

uden

ts’ p

reco

ncep

tions

in in

trodu

ctor

y m

echa

nics

. Am

eric

an Jo

urna

l of

Phys

ics,

50(1

), 66

-71.

Stra

tegi

c K

now

ledg

e:

Men

tal M

odel

s

Wha

t cau

ses d

ay a

nd n

ight

?

A. T

he e

arth

spin

s on

its a

xis (

.66)

B. T

he e

arth

mov

es a

roun

d th

e su

n (.2

6)

C. C

loud

s blo

ck o

ut th

e su

n (.0

3)

D. t

he su

n go

es ro

und

the

earth

(.04

)

Sour

ce: S

adle

r, P.

M. (

1998

) Psy

chom

etric

mod

els o

f stu

dent

con

cept

ions

in sc

ienc

e: R

econ

cilin

g qu

alita

tive

stud

ies a

nddi

stra

ctor

-driv

en a

sses

smen

t ins

trum

ents

. JR

ST, 3

5(3)

, 265

-296

.

Dev

elop

men

ts in

Ass

essm

ent:

Val

idity

Evi

denc

e fr

om T

IMSS

-RK

now

ledg

eTy

pePe

rcen

tM

ultip

le C

hoic

ePe

rcen

tO

pen

Ende

dPe

rcen

tTo

tal

Ana

lytic

(Dec

lara

tive

&10

1222

Sche

mat

ic)

Proc

edur

al6

28

Con

cept

ual

(Dec

lara

tive)

169

25

Fact

ual

(Dec

lara

tive)

381

38

“Not

Sci

ence

”5

27

Tota

l73

2710

0

Link

ing

Ass

essm

ents

to

Ach

ieve

men

t Com

pone

nts

Dec

lara

tive

Proc

edur

alSt

rate

gic

Kno

wle

dge

Kno

wle

dge

Kno

wle

dge

Exte

nt

Stru

ctur

e

Oth

ers

•Pe

rfor

man

ce A

sses

smen

ts•

Jour

nals

•C

once

pt M

aps

•C

ogni

tive

Map

s

•M

ultip

le-C

hoic

e•

Fill-

in•

Jour

nals

Proc

edur

e M

aps

•M

ultip

le-C

hoic

e•

Shor

t Ans

wer

•Pe

rfor

man

ce A

sses

smen

t

•M

ultip

le-C

hoic

e•

Inte

rvie

ws

•Pe

rfor

man

ce A

sses

smen

t

CLA

S: A

ligni

ng In

stru

ctio

n &

A

ccou

ntab

ility B.

Stan

dard

ized

Cur

ricul

um-E

mbe

dded

Ass

essm

ents

Stud

ent’s

Scor

e

A.

“On

Dem

and”

Mat

rix S

ampl

ing

of T

asks

& T

ests

C.

Portf

olio

s

Con

cept

ual F

ram

ewor

k Fo

r C

LAS

Aggr

egat

e Le

vel o

f Per

form

ance

A.

Mat

rix S

ampl

e B

ench

mar

k:M

ultip

le-C

hoic

e &

Perf

orm

ance

-Bas

edAs

sess

men

t

“Mod

erat

ed”

Scor

e:In

divi

dual

, Sch

ool

& D

istr

ict S

core

Indi

vidu

al L

evel

of P

erfo

rman

ce

B.

Stan

dard

ized

C

urric

ulum

-Em

bedd

ed

Ass

essm

ents

Teac

her

Mod

erat

ion

C.

Portf

olio

sSa

mpl

e fr

om C

lass

fo

r Agg

rega

tion

Teac

her C

alib

ratio

n &

Prof

essi

onal

Dev

elop

men

t

Tim

elin

e Fo

r Im

plem

enta

tion

A.

Stan

dard

ized

Ben

chm

arks

B.

Cur

ricul

um E

mbe

dded

Ass

essm

ents

&C

. Po

rtfol

ios

Pilo

t. .

.Im

plem

enta

tion

. . .

Ope

ratio

n19

9020

00In

crea

sing

Tea

cher

Res

pons

ibili

ty fo

r A

sses

smen

tIn

crea

sing

Tea

cher

Res

pons

ibili

ty fo

r A

sses

smen

t

Cav

eat:

Gov

erno

r Wils

on’s

R

atio

nale

For

Dis

cont

inui

ng C

LAS

SB 1

273

[CLA

S] ta

kes a

diff

eren

t app

roac

h . .

. .

Inst

ead

of m

anda

ting

indi

vidu

al st

uden

t sco

res

first

, with

per

form

ance

-bas

ed a

sses

smen

t in

corp

orat

ed in

to su

ch sc

ores

as t

his m

etho

d is

pr

oven

val

id a

nd re

liabl

e, it

man

date

s pe

rfor

man

ce-b

ased

ass

essm

ent n

ow a

nd tr

eats

the

prod

uctio

n of

indi

vidu

al st

uden

t sco

res a

s if i

t wer

e th

e ex

peri

men

tal t

echn

olog

y--w

hich

it c

lear

ly is

no

t. In

shor

t, SB

127

3 st

ands

the

prio

rity

for

indi

vidu

al sc

ores

on

its h

ead.

Brit

ish

Task

Gro

up o

n A

sses

smen

t and

Te

stin

g (T

GA

T): R

ecom

men

datio

ns

Ass

essm

ent

•R

esul

ts sh

ould

giv

e di

rect

info

rmat

ion

abou

t pup

il ac

hiev

emen

t in

rela

tion

to o

bjec

tives

(crit

erio

n-re

fere

nced

cr

iterio

n)•

Res

ults

shou

ld p

rovi

de d

iagn

ostic

info

rmat

ion

for p

upil

lear

ning

nee

ds (f

orm

ativ

e-ev

alua

tion

crite

rion)

•Sc

ales

shou

ld b

e co

mpa

rabl

e ac

ross

cla

sses

and

scho

ols

(cal

ibra

ted/

mod

erat

ed c

riter

ion)

•Sc

ales

shou

ld b

e se

t up

to tr

ace

pupi

l’s d

evel

opm

enta

l tra

ject

orie

s ove

r gra

de le

vels

(pro

gres

sion

crit

erio

n)[S

ourc

es:

Bla

ck (1

966)

, Dau

gher

ty (1

995)

]

Cav

eat:

PM T

hatc

her’

s Ev

alua

tion

Of T

GA

T

Ken

Bak

er w

arm

ly w

elco

med

the

repo

rt.

Whe

ther

he

had

read

it p

rope

rly

I do

not k

now

: if

he h

ad it

sa

ys m

uch

for h

is st

amin

a. C

erta

inly

I ha

d no

op

port

unity

to d

o so

bef

ore

agre

eing

to it

s pu

blic

atio

n . .

. th

at it

was

then

wel

com

ed b

y th

eLa

bour

part

y, th

e N

atio

nal U

nion

of T

each

ers a

nd

the

Tim

es E

duca

tiona

l Sup

plem

ent w

as e

noug

h to

co

nfir

m fo

r me

that

its a

ppro

ach

was

susp

ect.

Sour

ce:

Bla

ck (1

996,

p. 5

)

On Developing A Comprehensive Science Assessment Plan for DelawareOverviewFramework for SystemEducation Indicator SystemFramework for Cognitive OutputsAssessment of Declarative Knowledge: Multiple-Choice--TIMSS Pop. 2Assessment of Declarative Knowledge Structure: Eleven-Year-Old’s Concept MapAssessment of Procedural Knowledge: Performance of a Daytime Astronomy InvestigationProcedural Knowledge: Performance Assessment: TIMSS Pop. 2Assessment of Strategic Knowledge: Mental ModelsStrategic Knowledge: Mental ModelsDevelopments in Assessment:Validity Evidence from TIMSS-RLinking Assessments to Achievement ComponentsCLAS: Aligning Instruction & AccountabilityConceptual Framework For CLASTimeline For ImplementationCaveat: Governor Wilson’s Rationale For Discontinuing CLASBritish Task Group on Assessment and Testing (TGAT): RecommendationsCaveat: PM Thatcher’s Evaluation Of TGAT