Sum

mar

y S

lide

Scie

ntifi

c In

quiry

in E

duca

tion

Sci

enti

fic

Inqu

iry

in

Edu

cati

on

Rep

ort

of t

he N

RC

Com

mit

tee

on

Sci

enti

fic

Pri

ncip

les

in E

duca

tion

R

esea

rch

Ric

hard

J. S

have

lson

and

Li

sa T

owne

, edi

tors

An

Ove

rvie

w

NE

RP

PB

Mee

ting

11/

30/0

1

Bac

kgro

und

NER

PPB

spon

sore

d; b

egan

fal

l 200

0Pr

ompt

ed b

y Ca

stle

bill

, ong

oing

sk

eptic

ism

and

deb

ate

abou

t qu

ality

of

educ

atio

n re

sear

chCo

mm

ittee

of

expe

rts

auth

ored

‘c

onse

nsus

’ rep

ort

rele

ased

yes

terd

ayTi

mel

ine

quic

k by

NRC

stan

dard

s

Goa

ls

Info

rm O

ERI

reau

thor

izat

ion

Info

rm o

ngoi

ng p

ush

for

‘evi

denc

e-ba

sed

polic

y &

pra

ctic

e’ a

nd ‘s

cien

tific

ally

-bas

ed

educ

atio

n re

sear

ch’

Spar

k se

lf-re

flect

ion

in f

ield

Com

mit

tee

Mem

bers

hip

Ric

har

d J.

Sh

avel

son

(C

hai

r), S

tan

ford

Un

iver

sity

Don

ald

I. B

arfi

eld,

Wes

tEd

Rob

ert

F. B

oru

ch, U

niv

ersi

ty o

f P

enn

sylv

ania

Jere

Con

frey

, Un

iver

sity

of

Texa

s at

Au

stin

Ru

dolp

h C

rew

, Stu

pski

Fam

ily F

oun

dati

on

Rob

ert

L. D

eHaa

n, E

mor

y U

niv

ersi

ty

Mar

gare

t Ei

sen

har

t, U

niv

ersi

ty o

f C

olor

ado

at B

ould

er

Jack

McF

arlin

Fle

tch

er, U

niv

ersi

ty o

f Te

xas,

Hou

ston

Euge

ne

E. G

arci

a, U

niv

ersi

ty o

f C

alif

orn

ia, B

erke

ley

Nor

man

Hac

kerm

an, R

ober

t A

. Wel

ch F

oun

dati

on

Eric

Han

ush

ek, H

oove

r In

stit

uti

on

Rob

ert

Hau

ser,

Un

iver

sity

of

Wis

con

sin

-Mad

ison

Pau

l W. H

olla

nd,

Edu

cati

onal

Tes

tin

g Se

rvic

e

Elle

n C

ondl

iffe

Lag

eman

n, T

he

Spen

cer

Fou

nda

tion

an

d N

ew Y

ork

Un

iver

sity

Den

is C

. Ph

illip

s, S

tan

ford

Un

iver

sity

Car

ol H

. Wei

ss, H

arva

rd U

niv

ersi

ty

Cha

rge

and

App

roac

h

To c

onsi

der

scie

ntifi

c na

ture

of

educ

atio

n re

sear

ch a

nd h

ow a

fed

eral

age

ncy

coul

d su

ppor

t hi

gh q

ualit

y sc

ienc

eD

id n

ot c

ompr

ehen

sive

ly e

valu

ate

exis

ting

rese

arch

, res

earc

hers

, or

agen

cyAp

proa

ch is

for

war

d-lo

okin

g, in

form

ed b

y hi

stor

y an

d cl

ear

abou

t ro

les

of

stak

ehol

ders

Fram

ing

Que

stio

ns &

K

ey T

hem

es

Wha

t ar

e th

e pr

inci

ples

of

scie

ntifi

c qu

ality

in

edu

catio

n re

sear

ch?

Scie

nce

is f

unda

men

tally

the

sam

e ac

ross

all

disc

iplin

es a

nd f

ield

sAl

l fie

lds

are

char

acte

rized

by

a ra

nge

of le

gitim

ate

met

hods

and

spe

cial

izat

ion

depe

ndin

g on

obj

ects

of

inqu

iry a

nd c

onte

xtSo

me

diff

eren

ces

betw

een

soci

al a

nd n

atur

al s

cien

ces

As in

oth

er f

ield

s, f

eatu

res

of e

duca

tion

shap

e in

quiry

Fram

ing

Que

stio

ns &

K

ey T

hem

es (

cont

.)

How

can

a f

eder

al r

esea

rch

agen

cy

prom

ote

and

prot

ect

scie

ntifi

c qu

ality

in

the

educ

atio

n re

sear

ch it

sup

port

s?

Org

aniz

ed a

roun

d co

ncep

tion

of s

cien

tific

cul

ture

Fo

cuse

d on

art

icul

atin

g co

re in

fras

truc

ture

(pe

ople

, st

ruct

ures

, fun

ding

, fle

xibi

lity)

Em

phas

izes

rol

es o

f po

licy,

pra

ctic

e, a

nd r

esea

rch

com

mun

ities

Fram

ing

Que

stio

ns &

K

ey T

hem

es (

cont

.)

How

can

res

earc

h-ba

sed

know

ledg

e in

ed

ucat

ion

accu

mul

ate?

Scie

nce

is n

ever

fin

ishe

d, b

ut im

prov

es c

erta

inty

of

know

ledg

e ov

er t

ime

Nat

ure

of p

rogr

ess

com

mon

in a

ll fie

lds:

Scie

nce

adva

nces

in ‘f

its a

nd s

tart

s’ a

s re

sear

cher

s de

bate

fin

ding

s th

roug

h no

rms

enfo

rced

by

field

of

rese

arch

ers

Prog

ress

ena

bled

by

time,

mon

ey, a

nd p

ublic

sup

port

Res

earc

h-ba

sed

know

ledg

e in

edu

catio

n ha

s ac

cum

ulat

ed in

thi

s w

ay, b

ut n

ot t

o th

e sa

me

degr

ee

as o

ther

sci

entif

ic e

ndea

vors

Sci

enti

fic

Inqu

iry

in

Edu

cati

on

Rep

ort

of t

he N

RC

Com

mit

tee

on

Sci

enti

fic

Pri

ncip

les

in E

duca

tion

R

esea

rch

Ric

hard

J. S

have

lson

and

Li

sa T

owne

, edi

tors

Cha

pter

Sum

mar

ies

NE

RP

PB

Mee

tin g

11/

30/0

1

Cha

pter

1H

isto

rica

l and

Phi

loso

phic

al

Con

text

Scho

lars

hav

e de

bate

d na

ture

of

scie

nce

for

cent

urie

s, n

atur

e of

sci

ence

in e

duca

tion

for

mor

e th

an 1

00 y

ears

Skep

ticis

m a

bout

edu

catio

n re

sear

ch e

vide

nt

sinc

e its

ince

ptio

nRep

ort

take

s its

cue

fro

m e

volv

ed id

eas

rega

rdin

g m

odel

s of

hum

an n

atur

e, p

rogr

ess

in

scie

nce,

con

test

ed n

atur

e of

edu

catio

n,

unde

rsta

ndin

g ab

out

met

hod,

and

con

cept

ions

of

sci

entif

ic r

igor

Cha

pter

1H

isto

rica

l and

Phi

loso

phic

al

Con

text

(co

nt.)

Core

Ass

umpt

ions

N

o on

e de

finiti

on o

f sc

ienc

eSo

me

lines

of

inqu

iry m

ay n

ever

pan

out

Poss

ible

to

desc

ribe

phys

ical

and

soc

ial w

orld

Scie

ntifi

c qu

ality

one

asp

ect

of o

vera

ll va

lue

of e

duca

tion

rese

arch

Scie

nce

can

uniq

uely

con

trib

ute

to

unde

rsta

ndin

g, a

nd is

pow

erfu

l whe

n co

mbi

ned

with

insi

ghts

fro

m o

ther

for

ms

of

inqu

iry

Cha

pter

2A

ccum

ulat

ion

of S

cien

tifi

c K

now

ledg

eH

as s

cien

tific

kno

wle

dge

in e

duca

tion

accu

mul

ated

? Y

es, a

nd in

m

uch

the

sam

e w

ay a

s ot

her

field

s. Tr

ace

exam

ples

in:

Asse

ssm

ent

Early

rea

ding

Reso

urce

sEn

ablin

g Co

nditi

ons

Tim

e, m

oney

, pub

lic s

uppo

rtCo

mm

on C

hara

cter

istic

sAd

vanc

es in

‘fits

and

sta

rts’

Cont

este

dIn

terd

epen

denc

e an

d cy

clic

nat

ure

of e

mpi

rical

fin

ding

s,

met

hods

, and

the

ory

Stud

ying

hum

ans

inhe

rent

ly c

ompl

ex

Cha

pter

3G

uidi

ng P

rinc

iple

s fo

r S

cien

tifi

c In

quir

y

Gui

ding

Prin

cipl

esEm

body

not

ion

of ‘w

arra

nt’

Not

alg

orith

mic

Code

of

cond

uct

enfo

rced

by

norm

s of

co

mm

unity

Cha

pter

3G

uidi

ng P

rinc

iple

s fo

r S

cien

tifi

c In

quir

y (c

ont.

)

Scie

ntifi

c Pr

inci

ple

1: P

ose

Sign

ifica

nt

Que

stio

ns T

hat

Can

Be I

nves

tigat

ed

Empi

rical

lySi

gnifi

canc

eEm

piric

al N

atur

e

Scie

ntifi

c Pr

inci

ple

2: L

ink

Res

earc

h to

Rel

evan

t Th

eory

Conc

eptu

al f

ram

ewor

kTh

eory

-lade

n ob

serv

atio

ns

Cha

pter

3G

uidi

ng P

rinc

iple

s fo

r S

cien

tifi

c In

quir

y (c

ont.

)

Scie

ntifi

c Pr

inci

ple

3: U

se M

etho

ds T

hat

Perm

it D

irect

Inv

estig

atio

n of

Que

stio

nW

ide

rang

e of

legi

timat

e m

etho

ds a

vaila

ble

Mul

tiple

met

hods

oft

en s

tren

gthe

n in

fere

nces

Mea

sure

men

t of

ten

key

aspe

ct

Cha

pter

3G

uidi

ng P

rinc

iple

s fo

r S

cien

tifi

c In

quir

y (c

ont.

)

Scie

ntifi

c Pr

inci

ple

4: P

rovi

de C

oher

ent

Chai

n of

Rig

orou

s Rea

soni

ngBa

sic

logi

c of

infe

renc

e th

e sa

me

for

quan

titat

ive

and

qual

itativ

e re

sear

chAs

sum

ptio

ns c

lear

ly s

tate

dEs

timat

es o

f er

ror

prov

ided

Cons

ider

com

petin

g ex

plan

atio

nsSe

lect

ion

His

tory

Mea

sure

men

t er

ror

Cha

pter

3G

uidi

ng P

rinc

iple

s fo

r S

cien

tifi

c In

quir

y (c

ont.

)

Scie

ntifi

c Pr

inci

ple

5: R

eplic

ate

and

Gen

eral

ize

Acro

ss S

tudi

esRep

licat

ion

and

gene

raliz

atio

n st

reng

then

sc

ient

ific

theo

ries

and

conj

ectu

res

Gen

eral

izat

ion

achi

eved

thr

ough

use

of

stat

istic

al t

ools

, tria

ngul

atio

n, e

tc.

Gen

eral

izat

ion

in s

ocia

l wor

ld a

ffec

ted

by it

s

rapi

dly

chan

ging

nat

ure

Cha

pter

3G

uidi

ng P

rinc

iple

s fo

r S

cien

tifi

c In

quir

y (c

ont.

)

Scie

ntifi

c Pr

inci

ple

6: D

iscl

ose

Res

earc

h to

En

cour

age

Prof

essi

onal

Scr

utin

y an

d Cr

itiqu

e“O

pen

soci

ety

[of

rese

arch

ers]

” ke

y to

:D

ebat

ing

and

(som

etim

es)

inco

rpor

atin

g in

divi

dual

fin

ding

s in

to c

orpu

s of

kno

wle

dge

Enab

ling

repl

icat

ion

Cha

pter

3G

uidi

ng P

rinc

iple

s fo

r S

cien

tifi

c In

quir

y (c

ont.

)

Appl

icat

ion

of P

rinci

ples

No

one

set

of c

riter

ia c

an c

lear

ly d

istin

guis

h sc

ienc

e fr

om n

onsc

ienc

e an

d hi

gh-q

ualit

y sc

ienc

e fr

om lo

w-q

ualit

y sc

ienc

ePr

inci

ples

can

hel

p di

stin

guis

hEx

ampl

es

Cha

pter

4Fe

atur

es o

f E

duca

tion

and

E

duca

tion

Res

earc

h

Diff

eren

ces

in s

ocia

l and

phy

sica

l/nat

ural

ph

enom

ena

dist

ingu

ish

inqu

iry in

the

se

dom

ains

Res

earc

her

cont

rol m

ore

limite

d in

soc

ial

scie

nces

Res

earc

her

obje

ctiv

ity v

is-à

-vis

bia

sU

ses

of t

heor

yLe

vel o

f ce

rtai

nty

Cha

pter

4Fe

atur

es o

f E

duca

tion

and

E

duca

tion

Res

earc

h (c

ont.

)

Feat

ures

of

educ

atio

n th

at s

hape

res

earc

hVa

lues

and

Pol

itics

Hum

an V

oliti

onVa

riabi

lity

of E

duca

tiona

l Pro

gram

sO

rgan

izat

ion

of E

duca

tion

Div

ersi

ty

Cha

pter

4Fe

atur

es o

f E

duca

tion

and

E

duca

tion

Res

earc

h (c

ont.

)

Feat

ures

of

Educ

atio

n Res

earc

hM

ultip

le D

isci

plin

ary

Pers

pect

ives

Ethi

cal C

onsi

dera

tions

Rel

atio

nshi

ps

Cha

pter

5D

esig

ns f

or t

he C

ondu

ct o

f S

cien

tifi

c E

duca

tion

Res

earc

h

Des

igns

/met

hods

judg

ed o

nly

in t

erm

s of

re

leva

nce

to q

uest

ion

pose

dSt

udie

s ar

e ‘s

cien

tific

’ whe

n m

eet

prin

cipl

es o

f sc

ienc

e &

att

end

to f

eatu

res

of o

bjec

t of

inqu

iryTy

pes

of Q

uest

ions

Des

crip

tive

(Wha

t is

hap

peni

ng?)

Caus

al (

Is t

here

a s

yste

mat

ic e

ffec

t?)

Mec

hani

sm (

How

or

why

is it

hap

peni

ng?)

Cha

pter

5D

esig

ns f

or t

he C

ondu

ct o

f S

cien

tifi

c E

duca

tion

Res

earc

h (c

ont.

)

Des

crip

tive

Que

stio

nsEs

timat

es o

f Po

pula

tion

Char

acte

ristic

sSi

mpl

e Re

latio

nshi

psD

escr

iptio

ns o

f Lo

caliz

ed S

ettin

gs

Cha

pter

5D

esig

ns f

or t

he C

ondu

ct o

f S

cien

tifi

c E

duca

tion

Res

earc

h (c

ont.

)

Caus

al Q

uest

ions

Caus

al R

elat

ions

hips

Whe

n Ran

dom

izat

ion

is

Feas

ible

Ran

dom

ass

ignm

ent

best

way

to

ensu

re

equi

vale

nce

in g

roup

s fo

r co

mpa

rison

Freq

uent

ly u

sed

in m

any

disc

iplin

es &

app

lied

field

sN

ot f

requ

ently

use

d in

edu

catio

n bu

t m

any

exam

ples

dem

onst

rate

fea

sibi

lity

Cha

pter

5D

esig

ns f

or t

he C

ondu

ct o

f S

cien

tifi

c E

duca

tion

Res

earc

h (c

ont.

)

Caus

al Q

uest

ions

(co

nt.)

Caus

al R

elat

ions

hips

Whe

n Ran

dom

izat

ion

is

Not

Fea

sibl

e In

soc

ial s

ettin

gs r

ando

miz

atio

n is

som

etim

es

infe

asib

le‘Q

uasi

-exp

erim

ents

’ rel

y on

unt

esta

ble

assu

mpt

ions

and

sub

ject

to

sele

ctio

n bi

asQ

uasi

-exp

erim

ents

can

som

etim

es b

e pr

efer

able

to

exp

erim

ents

(e.

g., e

xter

nal v

alid

ity)

Cha

pter

5D

esig

ns f

or t

he C

ondu

ct o

f S

cien

tifi

c E

duca

tion

Res

earc

h (c

ont.

)

Mec

hani

sm Q

uest

ions

Whe

n Th

eory

is W

ell-E

stab

lishe

dW

hen

Theo

ry is

Wea

kEt

hnog

raph

ies

Des

ign

Stud

ies/

Teac

hing

Exp

erim

ents

Cha

pter

5D

esig

ns f

or t

he C

ondu

ct o

f S

cien

tifi

c E

duca

tion

Res

earc

h (c

ont.

)

Conc

lusi

ons

abou

t st

ate

of e

duca

tion

rese

arch

and

are

as t

o ta

rget

Theo

retic

al u

nder

stan

ding

wea

kKn

owle

dge

of c

ausa

l rel

atio

nshi

ps w

eak;

in

part

icul

ar, u

rge

expa

nded

use

of

rand

om

assi

gnm

ent

Cha

pter

6D

esig

n P

rinc

iple

s fo

r Fo

ster

ing

Sci

ence

in a

Fe

dera

l Edu

cati

on R

esea

rch

Age

ncy

Appr

oach

Did

not

eva

luat

e O

ERI

Forw

ard-

look

ing,

foc

used

on

‘firs

t pr

inci

ples

’ w

ith s

ugge

sted

sup

port

ing

mec

hani

sms

Info

rmed

by

hist

ory

of e

duca

tion

rese

arch

ag

ency

(N

IE &

OER

I) &

com

paris

ons

to o

ther

ag

enci

es (

NIC

HD

, NIA

, NSF

EH

R &

NSF

SBE

)

Dile

mm

asEd

ucat

ion

rese

arch

gro

unde

d in

pra

ctic

al

prob

lem

s bu

t cl

ose

rela

tions

hip

has

dow

nsid

eD

iver

sity

of

educ

atio

n re

sear

ch c

omm

unity

Cha

pter

6D

esig

n P

rinc

iple

s fo

r Fo

ster

ing

Sci

ence

in a

Fe

dera

l Edu

cati

on R

esea

rch

Age

ncy

(con

t.)

Des

ign

Prin

cipl

e 1:

Sta

ff t

he A

genc

y w

ith

Peop

le S

kille

d in

Sci

ence

, Man

agem

ent

and

Lead

ersh

ipFl

exib

ility

to

hire

per

man

ent

& t

empo

rary

st

aff

Enga

ge in

inte

rage

ncy

colla

bora

tions

Dep

ends

hea

vily

on

rela

ted

issu

es (

fund

ing,

re

puta

tion)

Cha

pter

6D

esig

n P

rinc

iple

s fo

r Fo

ster

ing

Sci

ence

in a

Fe

dera

l Edu

cati

on R

esea

rch

Age

ncy

(con

t.)

Des

ign

Prin

cipl

e 2:

Cre

ate

Stru

ctur

es t

o G

uide

Age

nda,

Inf

orm

Fun

ding

Dec

isio

ns,

& M

onito

r W

ork

Gov

erni

ng b

oard

with

age

nda-

sett

ing

com

mitt

eeSt

andi

ng p

eer

revi

ew p

anel

sM

any

stru

ctur

es c

an w

ork

Key

is g

ood

peer

sN

ot p

erfe

ct, h

ighl

y de

pend

ent

on s

tron

g fie

ld, a

nd

tric

ky in

edu

catio

n be

caus

e fie

ld is

ecl

ectic

Cha

pter

6D

esig

n P

rinc

iple

s fo

r Fo

ster

ing

Sci

ence

in a

Fe

dera

l Edu

cati

on R

esea

rch

Age

ncy

(con

t.)

Des

ign

Prin

cipl

e 3:

Ins

ulat

e th

e ag

ency

fr

om p

oliti

cal i

nter

fere

nce

Avoi

d m

icro

man

agem

ent

of d

ecis

ion-

mak

ing,

di

stor

tion

of a

gend

a to

be

sole

ly s

hort

-run

, us

e of

age

ncy

to p

rom

ote

polit

ical

pos

ition

sPo

litic

s pl

ay p

rope

r ro

le a

nd a

gend

a sh

ould

re

flect

pol

icy

conc

erns

Budg

etar

y di

scre

tion

part

icul

arly

key

Res

earc

h be

org

aniz

atio

nally

sep

arat

e bu

t in

telle

ctua

lly li

nked

to

impr

ovem

ent

mis

sion

Cha

pter

6D

esig

n P

rinc

iple

s fo

r Fo

ster

ing

Sci

ence

in a

Fe

dera

l Edu

cati

on R

esea

rch

Age

ncy

(con

t.)

Des

ign

Prin

cipl

e 4:

Dev

elop

Foc

used

and

Ba

lanc

ed P

ortf

olio

of

Res

earc

h th

at

Addr

esse

s Sh

ort-

, Med

ium

-, a

nd L

ong-

term

Iss

ues

of I

mpo

rtan

ce t

o Po

licy

and

Prac

tice

Crea

te b

ased

on

stat

e of

dev

elop

men

t of

fie

ldO

rgan

ized

in p

rogr

ams

of s

tudi

esIn

corp

orat

e sy

nthe

ses

of b

odie

s of

wor

k

Cha

pter

6D

esig

n P

rinc

iple

s fo

r Fo

ster

ing

Sci

ence

in a

Fe

dera

l Edu

cati

on R

esea

rch

Age

ncy

(con

t.)

Des

ign

Prin

cipl

e 5:

Ade

quat

ely

fund

the

ag

ency

Reso

urce

s m

ust

alig

n w

ith s

cope

of

agen

daG

iven

ass

umpt

ion

that

age

nda

will

be

roug

hly

com

para

ble

to O

ERI’s

tod

ay a

nd k

now

ledg

e of

cur

rent

app

ropr

iatio

n le

vels

, fun

ding

mus

t in

crea

se

Cha

pter

6D

esig

n P

rinc

iple

s fo

r Fo

ster

ing

Sci

ence

in a

Fe

dera

l Edu

cati

on R

esea

rch

Age

ncy

(con

t.)

Des

ign

Prin

cipl

e 6:

Inv

est

in r

esea

rch

infr

astr

uctu

reCo

mm

unity

of

rese

arch

ers

(hum

an r

esou

rces

)D

ata

deve

lopm

ent,

sha

ring

& a

cces

sH

ousi

ng o

f co

mm

on c

onst

ruct

sFa

cilit

atin

g et

hica

l acc

ess

to r

esea

rch

subj

ects

Link

to

Prac

tice

and

Polic

y Co

mm

uniti

es