Comparing the Effects of Two Versions of Professional

Development on Science Curriculum

Implementation and Scaling-UpSession 43.030

April 13, 2005American Educational Research Association Annual Meeting

Montreal, Canada

Introduction and Overview

Paul R. Brandon

Curriculum Research & Development GroupUniversity of Hawai‘i at Mānoa

3

Project Purpose• A randomized study of the effects of

variations in professional development (PD) on – program implementation– student achievement– scale-up

• First phase (ending 2/28/06) to prepare for five-year second phase (if funded)

4

FAST

• The project compares two versions of teachers’ PD for Foundational Approaches in Science Teaching (FAST).– An award-winning middle-school inquiry-

based science program– Shown to positively affect achievement and

other student outcomes

5

Phase-I Project Tasks

• Prepare the 5-day PD institute with electronic resource and on-line course

• Prepare instruments for the second phase:– Teacher questionnaire– Teacher log– Observation procedures– Student assessments

6

Focus on Span of PD

• Traditionally, FAST PD is delivered in a 10-day institute.

• We will compare it with an alternative 5-day institute followed by an online course, with an electronic multimedia resource.

7

Phase-II Hypothesis

The 5-day inquiry-based science training institute, followed by an on-line university-credit

course with computer-based multimedia, will have more favorable outcomes than a 10-day

institute without the follow-up university course.

8

Examining the effects of PD on:

• levels of classroom implementation

• intensiveness and extensiveness of long-term use of inquiry-based science (i.e., scale-up)

• levels of student achievement

9

Experimental Design

• Randomized cluster sample, with schools (N = 80) as clusters

• One teacher per school

• Even if attrition is 25%, statistical power will be .80.

10

Merit of the Study

• Addresses at least five deficits in the literature:– Few studies on effects of PD time span– Few studies about using technology in PD– Few randomized studies– Few studies of the effects of PD on student

learning– Few, if any studies, of inquiry science PD

11

Order of Presentations

• Gray, Nguyen, & Speitel: Description of PD • Brandon: Log and questionnaire development

and questionnaire validation• Taum: Observation guide development• Ayala: Student assessment development • Lawton: Comparison of early effects of the two

versions of PD

Introduction and Overview

Paul R. Brandon

Curriculum Research & Development GroupUniversity of Hawai‘i at Mānoa

Developing and Implementing an Alternative Version of FAST

Professional Development

Mary E. GrayThanhTruc T. Nguyen

Thomas W. SpeitelUniversity of Hawai‛i at Mānoa

14

FAST

• Foundational Approaches in Science Teaching

• Inquiry-based, middle school science

• Physical, biological, and earth sciences

• Exemplary program (USDOE, 2001)

15

FAST Professional Development

• All FAST teachers must participate in PD– Certification required to

purchase science curricula

• Content• Inquiry• Standards• Assessment• Classroom organization

and management• Safety

16

The ChallengeTo develop and implement an alternative version of FAST I professional development

Year Round Schedules

17

Considerations

• Current needs of science teachers

• Future possibilities (emerging technologies)

• Unique challenges (the dynamic nature of inquiry)

18

Review of LiteratureProfessional Development

On-Line LearningInquiry Science

19

Review of Literature Professional Development

• Strong subject area content (Kennedy, 1999)

• Focus on higher-order teaching strategies (Porter, Garet, Desimone, Yoon, & Birman, 2000)

– Collaborative, Same subject-grade-school, Active learning, Consistent with teacher goals

20

Review of Literature Professional Development

• Quantity linked to improvements (Radford, 1998; Supovitz, Mayer, & Kahle, 2000; Fishman, Marx, Best, & Revital, 2003)

– science content knowledge, process skills, and attitudes

• Quantity linked to standards based teaching (Supovitz & Turner,2000)

• Problems with researching inquiry-based teaching and student learning (Fishman et al., 2003)

21

Review of Literature On-line Learning

• 3 million adult learners on-line (Waitts & Lewis, 2003)

• Convenience and increased flexibility (Hülsmann,1999)

• Evidence of implementation of various pedagogical approaches (Schlager & Schank, 1997)

22

Review of Literature On-line Learning

• Quantifiable learning outcomes were not significantly linked to technology adoption (Jones and Paolucci,1998).

• Focus on motivation, skills and knowledge, self-directed learning, interactive competence, and technology skills (Kabilan, 2004)

• Increased efficacy and self-perception in teachers (Huai, Braden, White, & Elliot, 2003)

23

FASTPro

24

How are the traditional PD and FASTPro different?

• Two-weeks in duration, face-to-face

• One-week in duration, face-to-face

• Conduct 43 investigations

• Extensive modeling and practicing time

• Conduct 19 investigations

• Electronic resource• WebCT-based

learning community

InquiryInquiry

Teaching and learning science through inquiry is a new experience for many teachers and requires a significant change in attitude and behavior.

26

FASTPro – Addressing Inquiry• Strategies (Loucks-Horsley, Hewson,

Love, & Stiles,1998)

– Examples• Immersion in inquiry

into science• Coaching and

mentoring • Technology for

professional learning

27

FASTPro – Addressing Inquiry• Change (Hord, Rutherford, Huling-Austin, & Hall,1987; Guskey,

2000)

– takes time and persistence• awkwardness and frustration expected

– as teacher’s progress through a change process, their needs for support and assistance change

• Optimal Mix (Guskey, 2000) – A combination of teacher, program and change

agent that will help create a positive relationship for PD to be effective

28

FASTPro

29

FASTStart• One week face-to-face

institute

• Teachers conduct nearly half of actual student investigations

• Use inquiry methodology including modeling, discussion, assessment, and practice

30

31

FASTeR

• Observe interactions of students and teachers

• View and reflect upon investigations not experienced

• See detailed step-by-step procedure suggestions

• Movies and animations

• Web interface– Quicktime and Flash

plugins

• DVD-ROM medium

32

Examples from FASTeR

33

Examples from FASTeR• Slideshow 9, Density

and the Cartesian Diver

• Slideshow 22, Identifying Unknown Substances

34

35

FASTForward

36

Formative Findings

• Suggest that teachers were able to implement successfully and share teaching practice

• Added utility, focused and meaningful to teachers’ own environment

37

Future considerations• Infuse technological

aspects into the FASTStart face-to-face experience

• Address credit equivalencies to enable more robust activity requirements

• Expand to include FAQ’s, indexing, and rich descriptions

• Expand to include a Website as well as DVD-ROM

• Automate some computer assisted instruction and feedback mechanisms

38

Reflections…

39

Reflections…• Teamwork and

flexibility

• Coordination of equipment and resources

40

Future Considerations• Building capacity

• Application to other CRDG quality educational programs

41

Mahalo.

Developing and Implementing an Alternative Version of FAST

Professional Development

Mary E. GrayThanhTruc T. Nguyen

Thomas W. SpeitelUniversity of Hawai‛i at Mānoa

Instrument Development for a Study Comparing Two

Versions of Inquiry Science Professional Development

Paul R. Brandon Alice K. H. Taum

University of Hawai‘i at Mānoa

44

Identifying Constructs

• on teaching science with inquiry methods– Reviewed FAST and other inquiry science documents

and worked closely with inquiry science experts.

• on the context within which inquiry-based science is taught.– Reviewed 55 books and key articles on curriculum

indicators and school effectiveness.

45

Developing the Teacher Log

• Purpose: To identify the extent to which teachers implement the key features of inquiry-based science.

• Reviewed the recent literature on logs.

• Kept the instrument short to avoid overburdening teachers (21 items).

• Is completed immediately after finishing each student science investigation.

46

Focus of the Teacher Log

• Instrument addresses topics such as:– students’ questioning behaviors.– the teacher’s use of questioning strategies.– the teacher’s circulation about the classroom.– teacher-led discussions about variations in

the data.

47

Developing the Teacher Questionnaire

• Purposes: – To obtain information about implementation

that is most appropriately collected once a year.

– To collect information on implementation of investigations not covered on logs.

48

Focus of the Questionnaire

• 150 items address topics such as:– implementation of key features.– the investigations taught during the entire

year.– the adequacy of materials, equipment, etc.– teacher demographics.– teacher attitudes toward science.– teacher participation in science activities

outside the classroom

49

Types of Validation Studies to Date

• Two types of questionnaire-data validation studies conducted to date:– analyses addressing the relationship between

the responses on the log and the questionnaire (“concurrent” validity)

– an analysis examining the theoretical model underlying our study (construct validity)

50

Relationship Between Log and Questionnaire Responses

• Compared results on total scores for five items that the two instruments had in common

• Correlation = .54; effect size = .50

51

Examining the Theoretical Model• Second set of analyses examined whether the

expected relationships were found among some questionnaire variables.

• We regressed a variable measuring program implementation on five independent variables measuring:– teachers’ ongoing learning about science.– the resources available for teaching science.– the number of science courses taken.– the number of years have taught K–12 science– teachers’ attitudes toward teaching science

52

Validity Argument• Teachers’ ongoing learning about science

(including learning that occurred in recent PD) should predict implementation more than– science education background.– years of experience teaching science.– attitudes toward teaching science.– classroom resources available.

53

Results of Second Analysis• Teacher learning about science outside the classroom: • beta = .19 (significant at the .05 level)• Teacher attitudes toward teaching science: • beta = .17 (significant)

• Classroom and school resources for teaching science: • beta = .04 (not significant) • Number of university science courses taken: • beta = -.02 (not significant) • Number of years the teacher has taught K–12 science: beta =

-.02 (not significant)

Instrument Development for a Study Comparing Two

Versions of Inquiry Science Professional Development

Paul R. Brandon Alice K. H. Taum

University of Hawai‘i at Mānoa

Coding Teachers in Science Classrooms using the Inquiry Science Observation

Guide

Alice K. H. TaumCurriculum Research & Development Group

University of Hawai‘i at Mānoa

56

Inquiry Science Observation Guide• Introduction and Overview

– FAST, SCUP Project, Purpose and Development of ISOG

• General coding guidelines• Descriptions for each of the six

Activities• Definitions of Activity details

• Code Sheet • Recording Sheet • Reconciling Recording Sheet

57

Development of the Code Sheet

• 1.5 year process

• Collaborative efforts between researchers at the University of Hawai‘i at Mānoa, Stanford University and Sonoma State University; FAST teachers; curriculum developers, and coders

• 36+ revisions

58

Goal of the Code Sheet • Identify strings of activities/teacher

behaviors using the details in columns A, B, C and D. – questioning strategies used – level of engagement between teacher and

students– creating a profile of teachers implementation

of FAST in the classroom through science inquiry

59

Design of the Code Sheet

• 6 activities.

• 66 possible activity details to plug into the 6 activities.

• Multiple variations between an activity and the activity details which describe an activity string of teachers pedagogical practices.

60

Inquiry Science Observation Code Sheet (ISOCS)

Activity Activity details Activity

descriptors

1 - 6 A B C

D

1. Teacher directs student (s) ______A______ to ______B______ _______C______.

1A1. individually 1A2. in a small group

1B1. record 1B2. discuss 1B3. define 1B4. read materials to whole class

1C1. observations 1C2. predictions/ hypotheses 1C3. procedures 1C4. data 1C5. science (concept, vocab words, mechanics of science, etc…)

1D1. with evidence or examples 1D2. does not apply

2. Through ______A______, the teacher ______B_______ science ______C______.

2A1. direct instruction questioning: 2A2a. rhetorical 2A2b. interactive

2B1. introduces or provides an overview of 2B2. reviews/summarizes 2B3. demonstrates 2B4. collects 2B5. compares/contrasts 2B6. clarifies

2C1. concepts(idea) 2C2. procedures(activity) 2C3. tools/equipment 2C4. investigation (science experiment) 2C5. problem 2C6. goal 2C7. -related safety issues 2C8 data (unknown): 2C8a. differences in 2C8b. relationships between 2C8c. quality of 2C8d. analysis of 2C8e. synthesis of 2C8f. evaluation of 2C9. vocabulary words

2D1. new information 2D2. previously learned information 2D3. investigation 2D4. unit 2D5. does not apply

1. Teacher directs student(s)

_____A_______ to

_____B_____ _____C_____.

61

1. Teacher directs student(s) individually (1A1) to discuss (1B2) procedures (1C3).

1. Teacher directs student(s) _____A_____ to _____B______ _____C_____.

62

Inquiry Science Observation Recording Sheet

• Notes the start time of when the “broader” and “other activities” a teacher is engaged in begins

• Allows for comments to facilitate the reconciliation process

• Designed to record multiple activities occurring at the same time

63

Broader and Other Activities– The broader activity captures the larger activity

occurring:Through interactive questioning (2A2b), the teacher introduces or provides an overview of (2B1) science investigation (2C4).

– The other activities capture the details occurring within the broader activity:Teacher questions students through clarifying (3A1a) questioning and responds to student comment (3B1) by repeating (3C2) the comment and probing further (3C6).

64

65

Reconciling Codes• Two coders are paired to compare their

independent codings, identifying any differences between them.

• Differences are then discussed and each coder provides a rationale for his or her selected code.

• When necessary, a review of the DVD is conducted by the paired coders.

• Coding differences are discussed until consensus is reached between coders.

66

Inquiry Science Observation Reconciling Code Sheet

67

Challenges• greatest challenges:

– establishing universal definitions for terms and phrases

• What constitutes a “procedure”? • What is the difference between a science “term” and a

science “concept”?

– standardizing the recording of activity strings using the Recording Sheet

• broader verses other activities

– focusing on the teacher, rather than on the students

68

What Next?• To date we have:

– videotaped 15 teachers;– 140 videotape cassettes have been digitized

to DVDs;– 116 have been quality-checked for audio

clarity and teacher visibility; and – 8 coders are trained and ready to begin

coding!

Coding Teachers in Science Classrooms using the Inquiry Science Observation

Guide

Alice K. H. TaumCurriculum Research & Development Group

University of Hawai‘i at Mānoa

Developing Student Outcome Measures Frameworks

Carlos C. Ayala Sonoma State University

Scaling Up: Student Measures

The Buck Stops HereThe Buck Stops Here

Carlos Ayala

Let’s Talk Charge

Develop student assessment instruments in order to tease out differences between professional development models.

2 Session Pre Test3 Session Post Test Suite

73

Where we expect differences

• Student’s content knowledge• Student’s science inquiry skills• Student’s views of the nature of science• Student’s efficacy toward science• Student’s motivation to learn science

74

Final Assessment Suite

• Pre-test – 30 item multiple-choice/ short answer test (α= .86)– Attitudinal Survey

• Post-test – 30 item multiple-choice/ short answer test– Attitudinal survey– Mānoa River performance assessment

75

76

Knowledge Type Framework

Type of Knowledge Definition Examples Prompt

Declarative Knowing that Concepts & facts “What is

percolation?”

Procedural Knowing how Actions, steps, “How do

measure how much & procedures

water soil will hold?”

Schematic Knowing why Principles & “How does the water

cycle mental models work?”

77

78

Content Validity

• Reviewed curriculum and created content matrices• Curriculum developers parsed content down• Linked matrices to materials• Created test linking items to matrices• Piloted assessments• Talk alouds

79

• As students become more engaged in the FAST curriculum and the teacher more fully implements the curriculum, students will be more proficient at Science Inquiry.– Design and conduct a scientific investigation– Use appropriate tools to gather, analyze and interpret data– Develop descriptions, explanations predictions and models

using evidence

• Targets based on – Duschl’s Transformations in Three Domains– Pottenger’s Deductive Explanatory Inquiry

Student Science Inquiry

NRC. (1996). National Science Education Standards . Washington D.C.: National Academy of the Sciences.

80

Transformation 1Data to Evidence: Deciding if the data are evidence, irrelevant and/or problematic.

Transformation 2: Evidence to patterns or models decisions about selecting tools for identifying patterns or models

Transformation 3:Patterns and models to explanations. Deciding how the patterns or models lead to explanations.

Reformulation: From explanations to new questions: Deciding what next questions to ask and what new data are needed.

Student Science Inquiry

81

82

• As students become more engaged in the FAST curriculum and the teacher more fully implements the curriculum, students will understand that– anyone can be a scientist.– science knowledge is useful.– science knowledge builds over time.– science is creative.

Nature of Science

Lederman, Abd-El-Khalick, Bell and Schwartz (2002) Views of Nature of Science Questionnaire; Toward Valid and Meaningful Assessments of Learner’ Conceptions of the Nature of Science.

83

• As students become more engaged in the FAST curriculum, the greater control they will feel towards science, science investigations and science knowledge.

– I can make accurate measurements during a science investigation.– I can make appropriate predictions about what will happen during a science investigation.

Self Efficacy

Britner, S. and Pajares F., (2001) Self-Efficacy Beliefs, Motivation, Race and Gender in Middle School Science.

84

• As students become more engaged in the FAST curriculum, patterns of motivation may change.

• Dweck Groups (goals and epistemic beliefs)– Mastery Orientation– Ego Orientation– Helpless Orientation

Motivation

Haydel, A., & Roser, R. (2001). On the links between students' motivation patterns and their perceptions of, beliefs about and performance on different types of science achievement, Multidimensional Approach to Achievement

85

Next Steps

• 760 Tests and Surveys completed• Collect post test data• Run analyses• Provide results to group

Developing Student Outcome Measures Frameworks

Carlos C. Ayala Sonoma State University

The Differential Effects of Two Versions of Professional Development on Teachers’ Self-Efficacy to Implement

Inquiry-Based ScienceBrian Lawton

University of Hawai‘i at Mānoa

88

Purposes

• Effects of both versions of PD institutes on teacher self-efficacy to implement program

• Differences between the two versions

• Factors associated with self-efficacy change

89

Data Collection• Qualitative and quantitative methods used• Self-efficacy scale

– Developed to facilitate focus group discussion• Focus groups

– Conducted immediately following the institutes

– Provide in-depth information about changes in self-efficacy• Five stages of inquiry science

1. Introducing new science investigations2. Facilitating valid experimental design3. Facilitating the investigation process4. Constructing meaning5. Linking knowledge to new situations

• Seven teachers from the 10-day and seven teachers from the 5-day participated in the study

90

Data Collection (cont.)

Self-Efficacy Scale

1. STAGE: Introducing new science investigations - my ability to introduce students to new science investigations by reviewing and tying in previous work.

Now Low High ability ability

Before Low Highability ability

1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10

91

Data Analysis

• Only presenting focus group results• Analyzed for changes in self-efficacy

– Two categories: • Statements implying an increase in self-efficacy• Statements implying no change in self-efficacy

• Categories analyzed to identify the influencing factors

• The statements and factors were quantified

92

Results, Influencing Factors

• Overall seven factors were identified as influencing self-efficacy.

1 Factors identified as most influential in increasing self-efficacy

2 Factor identified as most associated with no change in self-efficacy

Factors influencing increased self-efficacy

•Content/pedagogical knowledge1

•Program Structure1

•Professional Collaboration

•Experience

Factors influencing no change in self-efficacy

•Practice/feedback2

•Student characteristics

•Experience

•Time issues

93

Results: Changes in Self-Efficacy

Changes in self-efficacy10-day version

5-day version

Statements identified as increases in self-efficacy

11 8

Statements identified as no change in self-efficacy

6 9

94

Discussion

• Group increases in self-efficacy

• Changes in self-efficacy between the groups

• Groups perceived gains in content and pedagogical knowledge

• Importance of 5-day follow-on support

95

Further Study

• A study comparing the groups after the 5-day group has received all their training.

96

Thanks!

The Differential Effects of Two Versions of Professional Development on Teachers’ Self-Efficacy to Implement

Inquiry-Based ScienceBrian Lawton

University of Hawai‘i at Mānoa

Comparing the Effects of Two Versions of Professional

Development on Science Curriculum

Implementation and Scaling-UpSession 43.030

April 13, 2005American Educational Research Association Annual Meeting

Montreal, Canada

99

Questions and Comments• Paul Brandon - Overview• Mary Gray and Truc Nguyen - Description of PD• Paul Brandon - Log and questionnaire development and

questionnaire validation.• Alice Taum –Observation guide development• Carlos Ayala - Student assessment development • Brian Lawton - Comparison of early effects of the two versions

of PD

Paper, Presentation and Contact Info available at:

http://hisii.hawaii.edu/SCUP/research/