development and validation of an instrument to …...final instrument •had to balance content...

Development and Validation of an Instrument to Measure Teacher

Knowledge of Inquiry

Gwen Nugent, Jon Pedersen, Greg Welch, Jim Bovaird

University of Nebraska-Lincoln

Background

• Instrument developed as part of national survey of elementary teacher professional development

– Professional development characteristics, teacher knowledge, teacher practice

• 19-item multiple-choice measuring teacher knowledge of:

– Scientific inquiry

– Classroom inquiry

– Pedagogical content knowledge

Development Process

• Review of existing instruments

• Drew questions from:

– Assessing PCK on Inquiry Science Teaching (Schuster,

Cobern, Schwarts, Velom, & Applegate)

– Evolving Inquiry (Doll, Bruning, Horn, & PytlikZillig)

• Developed new items

• Table of specifications developed

Table of Specifications

Reference Source

No. of Questions

Scientific inquiry The diverse ways in which scientists study the natural world and propose explanations based on the evidence derived from their work (NSES, pg. 23:)

6

Inquiry pedagogy [Schuster, et al.] 8

Source Source Source No. questions

Classroom Inquiry

K-4 NSES Standards

Essential features of classroom inquiry

NRC Inquiry Activities

3

Ask questions about objects, organisms, and events

Engage in scientifically oriented questions

Define questions from current knowledge

5

Plan and conduct an investigation

Propose preliminary explanations

1

Gather data Give priority to evidence

Gather evidence 3

Use data to construct explanation

Formulate explanation from evidence

Explain based on evidence

2

Connect explanation to knowledge

Consider other explanations

1

Development Process

• Focus group sessions with pre-service teachers to identify confusing questions and wording

• Resulting pilot assessment had 29 questions

• Pilot test with 164 teachers from two Midwestern states in early 2010 (convenience sample)

Pilot Test Results: Classical Test Theory

• Item difficulty .09 - .99;

• Discrimination .00 - .43

• Item-total correlations -.05 to .34

• Alpha = .54;

Final Instrument

• Had to balance content coverage, length, and psychometric properties

• Final version had 5 scientific inquiry; 7 classroom inquiry; and 7 PCK questions

• Some retained items were revised based on pilot results.

Psychometrics Comparison: Classical Test Theory

Statistic Pilot Items Retained Items

Difficulty .09 - .99 .09 - .91

Discrimination .00 - .43 .01 - .36

Item-total correlation .05 - .34 -.06 - .32

Alpha .54 .54

Psychometric Comparison: IRT

0.00

0.50

1.00

1.50

2.00

2.50

3.00

0.000

0.500

1.000

1.500

2.000

2.500

3.000

-4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4

Scaled InformationFunction (Original Items)

Scaled InformationFunction (RevisedItems)

Standard Error (OriginalItems)

Standard Error (RevisedItems)

Procedures for Final Survey Administration

• National survey was stratified random sample

• Mail survey conducted spring, 2010 through early spring, 2011

• Participants received mailed survey introduction, scan-form survey, and post card reminder

• Each teacher randomly assigned math, reading, or science version in equal numbers

Final Results from National Survey • Overall return for all subject areas was n = 595

• Teachers taking science measure (n = 142) scored 56% correct overall

– Scientific inquiry 48% correct

– Classroom inquiry 63%

– Pedagogical content knowledge 55%

CTT Comparison

Statistic Pilot Items Retained Items Final Results

Difficulty .09 - .99 .09 - .91 .05 - .91

Discrimination .00 - .43 .01 - .36 .12 - .41

Item-total correlation .05 - .34 -.06 - .32 -.03 - .34

Alpha .54 .54 .54

IRT Comparison

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

-4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Sta

nd

ard

Err

or

Info

rmati

on

Ability (Science Inquiry Knowledge)

Revised: Information Pilot: Information Revised: SE Pilot: SE

Future Directions: Data Analytic Model

Contact hr. Perceived Utility

Collaborative

Participation Perceived Knowledge Reported Practice

Practice/feedback Gain

(workshop)

Practice/feedback Knowledge

(classroom)

Future Directions

Knowledge measure will be one of the dependent variables for randomized control trial examining impacts of guided inquiry professional development, including coaching.

Final Results from National Survey

Presentation Wednesday, March 6 at 2:45 in

Curacao 8

Contacts

Gwen Nugent, Ph.D. (gnugent@unl.edu) Jon Pedersen, Ph.D Greg Welch, Ph.D. Jim Bovaird, Ph.D.

National Center for Research on Rural Education

216 Mabel Lee Hall Lincoln, NE 68588-0235

http://r2ed.unl.edu