results of the 2003–04 study of the impact of the local ... · impact of the local systemic...

Results of the 2003–04 Study of the Impact of the Local Systemic Change Initiative on

Student Achievement in Science

by

Mary Ann Simpson Eric R. Banilower

December 2004

Prepared For: The National Science Foundation

4201 Wilson Boulevard Arlington, VA 22230

Prepared By: Horizon Research, Inc.

326 Cloister Court Chapel Hill, NC 27514

INTRODUCTION Increasingly, education initiatives are being pressured to demonstrate their effectiveness, particularly as it relates to increased student achievement and a reduction in achievement gaps among demographic groups. In 1998, in response to these demands, the National Science Foundation (NSF) initiated as part of their solicitation for Local Systemic Change Initiatives (LSC) a requirement that each project beginning with Cohort 5 examine its effects on students. Being sensitive to differences in local contexts, NSF has allowed projects flexibility in how they choose to show evidence of effects on student outcomes. Differences in the nature of relevant student achievement data available to each project, as well as differences in the availability of other data about students, teachers, and schools make this flexibility a necessity. The various studies that projects are producing should provide a wealth of evidence about the variety of effects the LSCs are having on student outcomes. Studying program-wide effects systematically, however, will be difficult due to the varying instrumentation and designs of the studies. In response to this concern, Horizon Research, Inc. (HRI) proposed a three-year study, beginning with the 2001–02 academic year, to meet the challenge of investigating program-wide effects on student achievement in science, with the constraint of not imposing undue burden on projects. The study was initially limited to only those projects that included an upper elementary (grades 4–6) science component. This choice was made for three reasons: (1) the majority of LSC science projects were included in this group, (2) few projects already had student achievement data in science available, and (3) items for measuring science achievement, although limited, were available for this grade range. In an effort to increase the knowledge base on the impact of the LSC program on student achievement, a middle school science component was added to the study in the third year (2003–04). During the 2002–03 academic year, HRI collected, reviewed, revised, and piloted a set of middle school science assessment items using a process very similar to the one used in the development of the elementary science assessment. The data collection and analysis processes used for the middle school study also parallel those used for the elementary school study. This report presents results from the 2003–04 cross-project study. Analyses and results for elementary school and middle school projects are presented separately. The study used longitudinal data to control for students’ prior knowledge of the science content being tested, with a pre-test being administered near the beginning of the school year, and a post-test near the end. The relationship between post-test scores and several demographic factors was also assessed. Demographic factors included eligibility for free/reduced-price lunch, limited English proficiency, whether the student has an individualized education plan, race/ethnicity, and gender. These data also allowed HRI to examine the extent of any “achievement gaps” by gender, race/ethnicity, English-language proficiency, and SES.

Horizon Research, Inc. 1 December 2004

INSTRUMENTATION For both elementary and middle school students, achievement tests in science consisting of multiple-choice items taken primarily from the National Assessment of Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS) were administered. Items from these sources have been through extensive validity, reliability, and item functioning analyses as measures of science achievement. Separate instruments were created for the elementary school and middle school assessments. The items were selected, with the assistance of experts in science assessment and the Principal Investigators of the science LSCs, to represent the science content areas central to the LSC-designated instructional materials most frequently taught in the 4th, 5th, and 6th grades for the elementary school assessment and for the 7th and 8th grades for the middle school assessment. In each assessment, a few additional items were developed for topics frequently addressed by the LSCs, but not found in the NAEP or TIMSS item sets. In addition, all of the items were reviewed for language accessibility to help ensure that the assessment measured science knowledge, not reading ability. As a whole, the items represented a range of difficulty, allowing appropriate testing of students’ science achievement across a broad range of achievement levels. Copies of both the elementary and middle school student assessments are included in Appendix A. Although the intent of the assessment was to measure student knowledge on several scales (physical science knowledge, life science knowledge, etc.), factor and cluster analyses for both the middle and elementary school assessments indicated that, with the exception of four items, each assessment was measuring a single ability/trait. This trait was labeled “general science knowledge.” Item response theory (IRT) was then used to compute difficulty, discrimination, and guessing parameters for each item. These parameters were then used to calculate “true test scores” for each student. Using the true test score removes the error associated with day-to-day fluctuations in student performance, and is a better estimate of student knowledge. A more complete description of these procedures can be found in Appendix B. A teacher questionnaire was used to gather information regarding which science units were taught in each class between the pre-test and the post-test, as well as the extent of teacher participation in LSC professional development. Projects also provided demographic information about the students in the participating classes, including eligibility for the free/reduced-price lunch program, English proficiency status, and whether a student had an individualized educational plan. A student questionnaire was used to gather race/ethnicity and gender data. Copies of the teacher and student questionnaires are located in Appendix C.


ELEMENTARY SCHOOL STUDY The Sample Twelve of the 47 current and past LSC projects targeting 4th, 5th, or 6th grade science teachers elected to participate in the 2003–04 study, including 8 of the 16 that were required to assess impact on students, and 4 of the 31 for whom studies of impact on students are optional. Three projects administered the assessment to 4th grade classes, three to 5th grade classes, and seven to 6th grade classes (see Table 1). Eight projects administered the assessment to all classes at the grade level they selected; four administered the assessment to a sample of classes. The projects that elected to administer the assessment to a sample of classes submitted the names of all teachers at the selected grade level along with treatment information (number of hours participating in LSC professional development), teacher leader status (yes/no), and number of classes taught at the selected grade level. HRI designed a stratified sampling approach in order to maximize the variation in teacher treatment levels in the sample. With the exception of one project,1 for each project sampling a subset of their classes HRI created two lists: (1) classes taught by teachers participating in a relatively large number of hours of LSC professional development (i.e., above the median number of hours of professional development for teachers in that project); and (2) classes taught by teachers with a relatively small number of hours of LSC professional development (i.e., below the median number of hours of professional development for teachers in that project). Each list was randomly ordered and projects were instructed to select half of their classes from each list, attempting to get classes from as close to the top of each list as possible. In general, the projects were able to recruit the classes at or very near the top of their sample lists, offering a measure of confidence that the samples were not biased. Table 1 provides information about the 12 projects participating in the elementary study. Note that although HRI incorporated several steps in the data collection process to help ensure data quality, a number of students and classes were excluded from the final analyses. In some cases, classes were administered the pre-test, but not the post-test. In other cases, teachers did not follow instructions for administering the assessment and HRI was unable to match students’ pre-test and post-test data. Finally, classes in which the teacher administering the post-test was not the same as the one administering the pre-test were excluded since it was not possible to determine how much science instruction the students received or if the instruction was provided by a LSC-trained teacher.

1 One project sampled one treated and one untreated teacher from each of 20 participating schools in order to control for school level effects in their analysis of their project’s data.


Table 1 Descriptive Information for Participating Elementary School Projects

Project Cohort

Number of Years

Providing Professional Development

Grade Levels

Selected

Sample or

Population

Number of

Classes at

Selected Grade Level

Number of Classes

Administering Assessment

Number of Classes Returning

Usable Data

1 2 5 6 Population 91 91 81 2 3 5 6 Sample 45 45 41 3 4 5 6 Population 59 59 47 4 4 5 5 Sample 111 16 9 5 5 5 6 Population 98 98 97 6 5 5 4 Population 160 160 80 7 5 5 4,6 Population 131 131 121 8 6 4 6 Population 227 227 224 9 6 4 4 Sample 178 6 6

10 7 3 5 Population 30 30 29 11 7 3 6 Population 10 10 9 12 7 3 5 Sample 140 40 40

As can be seen in Table 2, about two-thirds of elementary school classes in this study were taught by teachers with at least six years of teaching experience. About one-quarter of the elementary school classes in this study were taught by teacher leaders. Teacher leaders are selected by the projects to receive leadership training in addition to professional development in science content and pedagogy. Further, many teachers chosen by projects to be teacher leaders are often selected because of their enthusiasm for science, skill at teaching science, or advanced knowledge of science. Thus, they may not be representative of a “typical” teacher responsible for teaching science. Ideally, the analyses presented in this report would have excluded teachers that were teacher leaders. However, many classes participating in this study came from projects that were funded relatively recently and only a small number of their non-teacher leaders had received extensive treatment. Without including classes taught by teacher leaders, there would be few classes in the sample receiving instruction from teachers with extensive levels of LSC professional development, making a meaningful analysis of the relationship between treatment and student achievement impossible.

Table 2 Elementary School Students:

Descriptive Statistics for Categorical Class Level Variables Percent of Classes

(N = 670) Teacher Experience Level

Five or fewer years teaching 33 Six or more years teaching 67

Teacher Designated as a “Teacher Leader” No 77 Yes 23


The elementary school classes included in this sample were taught by teachers who had accrued, on the average, 82 hours of LSC professional development (see Table 3). On the average, elementary school students received over 2,500 minutes (40 hours) of science instruction between the pre- and post-tests. For a typical class in the study, this translates to approximately 20 minutes of science instruction per day between the pre- and post-test. The average elementary school class contained 26 students, of which 45 percent were non-Asian minority students.

Table 3 Elementary Students:

Descriptive Statistics for Continuous Class Level Variables

Minimum Maximum Mean Standard Deviation

Number of hours of LSC professional development 0.00 200.00 81.61 64.77 Number of minutes of science instruction between pre- and post-

tests 0.00 9,170.00 2,543.88 1,718.76

Percent of instruction based upon the LSC-designated instructional materials 0.00 100.00 65.91 32.97

Number of students in class 1.00 36.00 26.04 4.68 Percent non-Asian minority students in class 0.00 100.00 44.63 49.71

In addition to classroom and project features, the study included variables for a number of student characteristics. As can be seen in Table 4, the sample is comprised almost equally of males and females. Fifty-four percent of elementary students were classified as White or of Asian descent; 46 percent belonged to non-Asian minority groups. Approximately one-third of elementary school students participating in this study were eligible for free/reduced-price lunch, 7 percent had an individualized education plan, and 8 percent were classified as limited English proficient.

Table 4 Elementary School Student Demographics

Percent of Students (N = 12,141)

Gender Female 49 Male 51 Race/ethnicity

White 52 African-American 19 Hispanic or Latino 20 Asian 3 American Indian or Alaskan Native 2 Hawaiian or Other Pacific Islander 0 Other 3

Free/Reduced-Price Lunch Eligible 33 Individualized Education Plan 7 Limited English Proficient 8


Analysis and Results The elementary school analyses described in this report are based upon 670 classes and 12,141 students in 12 projects that submitted complete pre- and post-test data on the variables included in the analyses. Design weights were used in all analyses to account for the unequal probabilities of classes being selected to participate in the study. Descriptive statistics for pre- and post-test raw and true scores for elementary and middle school students are shown in Table 5. Overall, students scored higher on the post-test than they did on the pre-test, an indication that the assessment is sensitive to instruction. As mentioned previously, the true score is a better estimate of student knowledge as it removes the error associated with day-to-day fluctuations in student performance.

Table 5 Elementary School Students:

Raw and IRT True Scores (N = 12,141)


Raw Score Pre-Test 9.80 100.00 61.64 18.66 Post-Test 7.84 100.00 66.81 19.12

True Score Pre-Test 19.85 95.37 61.16 17.84 Post-Test 20.55 95.17 65.82 18.22

The science program study data had a nested structure, with students nested within classes, and classes nested within projects. Statistical techniques that do not account for potential shared variance within groups in nested data structures can lead to incorrect estimates of the relationship between independent factors and the outcome. Hierarchical modeling is an appropriate technique for apportioning and predicting variance within and across groups in a nested data structure (Bryk & Raudenbush, 2002). MPLUS 3.11 (Muthen & Muthen, 2004) was used for all models presented in this report. A two-level hierarchical path analysis (students nested within classes) was used to investigate the relationship between the extent of teacher participation in LSC professional development and student post-test scores, controlling for pre-test scores and demographic characteristics of the students and classes. Past evaluations of the LSC program have found that elementary science teachers participating in LSC professional development tend to teach more science and use the LSC-designated instructional materials more frequently than teachers who have not participated in LSC professional development (Weiss, et al., 2001). Consequently, the indirect effects of professional development on post-test scores through the amount of instructional time devoted to science and the percentage of science instruction based upon the LSC-designated instructional materials were also examined.


The analysis included a number of student level predictors:

• Pre-test true score; • Gender; • Race/ethnicity (White/Asian vs. non-Asian minority); • Whether the student was eligible for free/reduced-price lunch (FRL); • Whether the student had an individualized education plan (IEP); and • Whether the student was classified as limited-English proficient (LEP).

The factor of most interest in these analyses was extent of teacher participation in LSC professional development, a classroom level variable. Additionally, classroom demographics have been shown to have effects over and beyond those of individual demographics and/or to interact with individual demographics (Bryk & Raudenbush, 2002). The classroom level predictors included in these analyses were:

• Class size; • Proportion of students in the class eligible for free/reduced-price lunch (FRL); • Proportion of students in the class with an individualized education plan (IEP); • Proportion of students in the class classified as limited-English proficient (LEP). • Teacher experience level (six or more years vs. five or fewer years); • Teacher leader status; • Amount of instructional time devoted to science between the pre- and post-testes

(minutes of instruction); • Percentage of science instruction based upon the LSC-designated instructional

materials; and • Extent of teacher participation in LSC professional development.

An assumption of regression analyses is that the included variables have normal distributions, so all continuous variables used in this study were assessed for deviations from normality. Variables that were non-normally distributed were transformed as necessitated by the level and direction of skewness and kurtosis. The transformed variables were:

• Proportion of IEP students in each class—transformed by square root; and • Proportion of LEP students in each class—transformed by square root.

Note that for these analyses, teachers’ level of professional development is treated as a continuous variable although it is measured on an 11-point scale. The midpoint of each response category was used as an approximation of the amount of professional development received by a teacher. For example, a teacher that indicated s/he received between 10 and 19 hours of LSC professional development was assigned a value of 14.5 hours of LSC professional development. Additionally, amount of instructional time devoted to science and percentage of science instruction based upon the LSC-designated instructional materials were standardized to have a mean of 0 and standard deviation of 1.


In addition to examining the relationships between the variables listed above and post-test scores, each student level variable was tested to determine if its slope varied across classes (e.g., if the relationship between the post-test score and the free/reduced-price lunch status was different for different classes). When there was significant variation across classes in the slope of a student demographic variable, classroom level predictors (class size, teacher experience, and extent of teacher participation in LSC professional development) were used in an attempt to explain the variation. Due to limitations in modeling software, it was not feasible to include a third level of nesting in the model to control for project membership. In an attempt to control for possible project effects, a set of dummy-coded project membership variables was included at the class level. Although this process does not allow for the testing of specific project effects (i.e., whether the effect of a certain variable on student achievement varies across projects), it should factor these effects out of the analysis. Table 6 shows the regression coefficients and standard errors for the main variables included in the final model for the elementary school students. Project membership variables are excluded both for space reasons and because the purpose of the study is to examine the impact of the LSC across the program, not for any individual project.


Table 6 Fixed Effects for Elementary School Post-Test True Scores

Regression Coefficient (Standard Error)

Intercept 64.53 (0.70)

Student Level FRL† 0.89

(1.58) Teacher Leader -2.09*

(0.92) IEP† -7.93*

(2.10) Percent of LEP students in class (transformed) -5.64*

(2.64) LEP -1.56*

(0.58) Non-Asian minority -2.81*

(0.41) Female -0.03

(0.28) Pre-test score 0.73*

(0.02) Class Level Teacher Leader -0.48

(0.55) Teacher has six or more years experience 0.78

(0.52) Number of students 0.01

(0.05) Percent of non-Asian minority students -3.23*

(1.13) Percent of FRL students -1.04

(1.48) Percent of IEP students (transformed) 2.34

(1.41) Percent of LEP students (transformed) -3.56*

(1.68) Hours of LSC professional development -0.01

(0.00) Amount of science instruction 0.82*

(0.23) Hours of LSC professional development 0.01*

(0.01) Percent of instruction based on LSC materials 0.50

(0.28) Hours of LSC professional development 0.01

(0.01) † Non-zero random variance for this effect, p < .05. All of the teacher level variables were used to

predict this variance; however due to space limitations, the non-significant predictors are not shown in this table.

* p < 0.05 After controlling for pre-test scores and all other variables in the model, LEP, IEP, and non-Asian minority students tended to have lower post-test scores than other students. Because the relationships between post-test scores and FRL and IEP varied across classes, the classroom level variables were entered into the model as predictors of this variation. The proportion of


LEP students in a class was related to IEP student performance; the higher the proportion of LEP students, the lower IEP students tended to score. Additionally, FRL students in classes taught by a teacher leader tended to have lower post-test scores. In terms of the main variable of interest, there was no significant direct relationship between teacher participation in LSC professional development and student post-test scores. Not surprisingly, the amount of instruction devoted to science is a significant and positive predictor of post-test scores. Furthermore, hours of professional development is a significant and positive predictor of this instructional time, indicating that participation in professional development had an indirect effect on post-test achievement scores. The percentage of instruction based upon the LSC-designated instructional materials was not a significant predictor of post-test scores. Figure 1 illustrates the overall relationship between teacher participation in LSC professional development (including both direct and indirect relationships) and predicted student post-test score (i.e., the expected score for a typical student participating in this study based upon the regression equation). Although the expected score increases with high levels of professional development, the differences are extremely small—roughly one-half of a percentage point over 200 hours of professional development (approximately 0.03 standard deviations).

Predicted True Test Score, by Amount of LSC Professional Development

60

62

64

66

68

70

0 20 40 60 80 100 120 140 160 180 200

Hours of LSC Professional Development

Pred

icte

d Tr

ue T

est S

core

Figure 1


MIDDLE SCHOOL STUDY The Sample Six of the 22 current and past LSC projects targeting 7th or 8th grade science teachers elected to participate in the 2003–04 study, including 3 of the 11 that are required to assess impact on students, and 3 of the 11 for whom studies of impact on students are optional. Three projects administered the assessment to 7th grade classes, and three to 8th grade classes. Table 7 provides information on the projects participating in the middle school study. All but one participating project administered the assessment to the entire population of classes at the selected grade-level.

Table 7 Descriptive Information for Participating Middle School Projects

Project Cohort

Number of Years

Providing Professional Development

Grade Levels

Selected

Sample or

Population

Number of

Classes at

Selected Grade Level

Number of Classes

Administering Assessment

Number of Classes Returning

Usable Data

1 2 5 7 Population 41 41 41 2 4 5 7 Population 45 45 26 3 4 5 8 Sample 146 16 7 4 6 4 8 Population 132 132 126 5 7 3 8 Population 56 56 35 6 7 3 7 Population 14 14 10

Roughly 3 in 5 participating classes were taught by teachers with six or more years of experience (see Table 8). About 1 in 5 was taught by a teacher leader.

Table 8 Middle School Students:

Descriptive Statistics for Categorical Class Level Variables Percent of Classes

(N = 225) Teacher Experience Level

Five or fewer years 42 Six or more years 58

Teacher Designated as a “Teacher Leader” No 79 Yes 21

The middle school classes in this sample were taught by teachers who had accrued, on average, 89 hours of LSC professional development (see Table 9). On the average, middle school students received 3,730 minutes (62 hours) of science instruction between the pre- and post-test


administration (roughly 30 minutes per day), 47 percent of which was based on LSC-designated instructional materials. The average middle school class contained 24 students, of which about half were non-Asian minority students.

Table 9 Middle School Students:

Descriptive Statistics for Continuous Class Level Variables (N = 225)


Number of hours of LSC professional development 0.00 200.00 88.79 69.65 Number of minutes of science instruction between pre- and post-

tests 0.00 10,545.00 3,730.31 2554.26

Percent of instruction based upon the LSC-designated instructional materials 0.00 100.00 47.10 33.91

Number of students in class 7.00 35.00 23.99 4.89 Percent non-Asian minority students in class 0.00 100.00 51.82 32.14

As can be seen in Table 10, students in the middle school sample were comprised almost equally of males and females. Forty-nine percent of middle school students were classified as White or of Asian descent; 51 percent belonged to non-Asian minority groups. Twenty-nine percent of the middle school students were eligible for free/reduced-price lunch, 5 percent had an individualized education plan, and 1 percent were classified as limited English proficient.

Table 10 Middle School Student Demographics

Percent of Students (N = 3,123)

Gender Female 49 Male 51

Race/ethnicity White 47 African-American 35 Hispanic or Latino 11 Asian 2 American Indian or Alaskan Native 1 Hawaiian or Other Pacific Islander 0 Other 4

Free/Reduced-Price Lunch Eligible 29 Individualized Education Plan 5 Limited English Proficient 1

Analysis and Results The middle school analyses described in this report are based upon 225 classes and 3,123 students in six projects that submitted complete pre- and post-test data. As was the case in the elementary school study, design weights were used in all analyses to account for the unequal probabilities of classes being selected to participate in the study. HLM models for the middle school students included the same student, class, and project level variables as the models for the


elementary school students. The same variable transformations utilized in the analysis of the elementary study data were used to address non-normal distributions in the middle school data. Table 11 shows the regression coefficients and standard errors for the main variables included in the final model for the middle school students (again, project membership variables were excluded from the table). After controlling for pre-test scores, being a non-Asian minority predicted lower post-test scores. At the class level, the higher the percentage of IEP students in a class, the lower post-test scores tended to be. Classes taught by teacher leaders tended to have higher post-test scores than teachers not taught by teacher leaders. The number of hours of LSC professional development and percentage of instruction based on the LSC-designated instructional materials did not predict post-test scores. The amount of science instruction provided between the pre- and post-tests was not a significant predictor of post-test scores. Additionally, there was no evidence of an indirect effect of professional development on post-test scores via the amount of science instruction or percentage of instruction based on the LSC-designated instructional materials.


Table 11 Fixed Effects for Middle School Post-Test Scores

Regression Coefficient (Standard Error)

Intercept 55.40 (0.51)

Student Level FRL -1.14

(1.10) IEP -1.25

(1.78) LEP 2.00

(1.77) Non-Asian minority -2.71*

(0.42) Female -0.32

(0.62) Pre-test score 0.82*

(0.03) Class Level

Teacher Leader 2.44* (0.87)

Teacher has six or more years experience -0.25 (0.55)

Number of students 0.09 (0.07)

Percent of non-Asian minority students -0.66 (1.33)

Percent of FRL students 0.16 (1.40)

Percent of IEP students (transformed) -2.85* (1.38)

Percent of LEP students (transformed) -3.00 (2.35)

Hours of LSC professional development -0.01 (0.01)

Amount of science instruction -0.39 (0.29)

Hours of LSC professional development 0.01 (0.01)

Percent of instruction based on LSC materials 0.23 (0.32)

Hours of LSC professional development 0.01 (0.01)

* p < 0.05

CONCLUSIONS This study attempted to examine how the LSC program is impacting student achievement in science, both at the elementary and middle school levels, by exploring the relationships between a number of student (i.e., demographics) and classroom level factors (e.g., extent of teacher participation in LSC professional development) and post-test scores, using pre-test scores to control for initial status. Twelve LSC projects participated in the elementary grades study; six participated in the middle grades study.


The elementary school data show a positive relationship between teacher participation in LSC professional development and the amount of instructional time devoted to science between the pre- and post-test administrations. Although the elementary school data do not provide evidence for a direct relationship between the extent of teacher participation in LSC professional development and student post-test scores, they do provide evidence for an indirect relationship via the amount of instructional time devoted to science. In other words, instructional time devoted to science is a positive predictor of post-test scores, after controlling for pre-test scores, and teacher participation in LSC professional development is a positive predictor of instructional time devoted to science. However, the effect is extremely small, about 0.03 standard deviations. The elementary study data also show evidence for a number of achievement gaps. Students with an IEP and those classified as non-Asian minority or LEP tended to score lower than other students. Neither the extent of teacher participation in LSC professional development nor the percentage of science instruction based upon the LSC-designated instructional materials are associated with a closing (or widening) of these achievement gaps. In regards to the middle school study, the extent of teacher participation in LSC professional development is not, directly or indirectly, a significant predictor of post-test scores. In addition, the amount of science instruction between the pre- and post-tests is not a significant predictor, perhaps indicating that the assessment utilized was not sensitive to instruction. However, students in classes taught by teacher leaders did outperform students in classes taught by non-teacher leaders. This finding may indicate that the impact of LSC professional development on student achievement can be expected only for teachers who have participated extensively. It is important to acknowledge some of the threats to the validity of this study. In regards to the study’s internal validity (i.e., the extent to which the results, or lack thereof, can be attributed to the LSC program), there are two major concerns. First, although the analyses presented in this report may appear to be longitudinal, they are not. Teachers with varying degrees of professional development are different teachers, not the same teacher examined at different time points. This study was not designed, nor was it feasible, to follow teachers over time and examine their “effectiveness” as they participated in LSC professional development. Similarly, there is no feasible means to assess how similar, or different, teachers with varying degrees of participation in LSC professional development are to one another. Teachers participating the most in LSC professional development may be fundamentally different from teachers who participate to a lesser extent (e.g., more motivated to teach science, less experienced because they don’t feel they can say “no” to administrators, etc.). In addition to the threats to internal validity, there is an important threat to the external validity (i.e., generalizability) of this study. Only 12 of the 47 projects targeting grades 4–6 and 6 of the 22 projects targeting grades 7–8 elected to participate in this study. Thus, there is no way to determine whether these findings are indicative of the LSC program as a whole or are idiosyncratic to the set of projects participating in this study.


REFERENCES Bryk, A.S. & Raudenbush, S.W. (2002). Hierarchical linear models: Applications and data

analysis methods. Second Edition. Newbury Park, CA: Sage Publications. Muthen, B. & Muthen, L. (2004). MPLUS (Version 3.11) [Computer software]. Los Angeles,

CA: Muthen & Muthen. Weiss, I. R., Arnold, E. E., Banilower, E. R., & Soar, E. H. (2001). Local systemic change

through teacher enhancement: Year six cross-site report. Chapel Hill, NC: Horizon Research, Inc.

Appendix A

Student Assessments

Local Systemic Change 2003–04 Elementary Science Assessment

TEST BOOKLET To the student: Mark your answers on the answer sheet. Fill in only one circle for each question. Please do not write on this test booklet; you may use scrap paper if needed. Please think carefully about your answers. Do your best, but do not be concerned if you do not know the answers to all of the questions. This test was designed for students in multiple grades and classes who may or may not have studied the topics being assessed. If you finish before time is called, you should go over your work again.

WHEN YOUR TEACHER TELLS YOU TO, TURN THE PAGE AND BEGIN

© Horizon Research, Inc. Do Not Copy Without Permission

Copyright © 2003 by Horizon Research, Inc.

All rights reserved. No part of this assessment may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from Horizon Research, Inc. For information contact: Horizon Research, Inc. 326 Cloister Court Chapel Hill, NC 27514-2296 Email: [email protected]


1. When this caterpillar becomes an adult, what will it look like?

2. A metal spoon, a wooden spoon, and a plastic spoon are placed in hot water. After 15 seconds, which spoon will feel hottest?

A. The metal spoon

B. The wooden spoon

C. The plastic spoon

D. The three spoons will feel the same

3. The morning after a rain storm, Julio noticed a big puddle outside his door when he went to school, even though the sun was shining brightly. When he came home in the afternoon, the puddle was gone. What is the most likely explanation?

A. The water stuck to the shoes of people walking by his house.

B. Neighborhood dogs drank the water.

C. A westerly wind moved the puddle to another location nearby.

D. The liquid water went into the air as water vapor.

© Horizon Research, Inc. Page 1 Do Not Copy Without Permission

4. The picture shows the ways three students connected a battery and a bulb with wires. Which of the bulbs will light?

A. 1 only

B. 2 only

C. 3 only

D. 1 and 3

E. 2 and 3

1

2

3

5. To find out whether seeds grow better in the light or dark, you could put some seeds on pieces of damp paper and

A. Keep them in a warm, dark place.

B. Keep one group in a light place and another in a dark place.

C. Keep them in a warm, light place.

D. Put them in a light or dark place that is cool.

6. Look at the picture of the organs on the left. What is the main job of the organ labeled 1?

A. Carrying air

B. Carrying food

C. Carrying blood

D. Carrying messages from the brain


7. Louisa walks around her house, testing several types of objects with a magnet. Which of the following objects will her magnet attract?

A. All metal objects

B. All shiny objects

C. Other magnets

D. A and C

E. B and C

8. How hot is it on the surface of the Sun? A. Not quite as hot as boiling water

B. About as hot as fire

C. About 100 °F

D. Much hotter than almost anything on Earth

9. Karen tried letting a cart go from the top of a ramp three times, once empty, once with a single wooden block, and once with two wooden blocks. Each time, she measured how long it took for the cart to get to the bottom. What idea was she testing?

A. The higher the cart is when you release it, the

faster it will go.

B. The heavier a cart is, the faster it will go.

C. A cart’s wheel size is not related to how fast it goes.

D. What the block is made of is not related to how fast it goes.


10. Which of the following is NOT a form of precipitation?

A. Hail

B. Wind

C. Rain

D. Snow

11. In the diagram to the left, magnets 1 and 3 are stuck in place, but magnet 2 is attached to a string that allows it to swing back and forth. When magnet 2 is released, which way will it swing?

A. Towards magnet 1.

B. Towards magnet 3.

C. It will swing back and forth.

D. It will not swing at all.

12. A teacher taps lightly on the table. Which of the following best describes what her students hear?

A. All the students hear the sound at the same loudness.

B. The three students who are standing will hear the sound louder because the higher your head is, the better you can hear.

C. The three students with an ear on the table will hear nothing because the table is blocking the sound from reaching their ears.

D. The students with an ear on the table will hear the sound louder than those standing because sound travels better through solids than through gases like air.


13. John kept some seeds on moist cotton in a dish. Mike put the same kind of seeds in a dish beside John’s dish, and covered them with water. After two days, John’s seeds sprouted, but Mike’s did not. Which is the most likely reason?

A. Mike’s seeds needed more air.

B. Mike’s seeds needed more light.

C. Mike did not put the dish in a warm enough place.

D. Mike should have used a different kind of seed.

14. On which day was snowfall most likely to have occurred?

A. Monday

B. Tuesday

C. Wednesday

D. Thursday

15. Students in Mr. Wylee’s class made electromagnets as shown in the diagram. Mr. Wylee challenged the class to make the electromagnet stronger. Which of the following would work?

1. Wrap more coils around the bolt 2. Use a stronger battery 3. Make the wire between the switch and the

battery shorter

A. 1 only

B. 2 only

C. 3 only

D. 1 and 2

E. 2 and 3


16. Which of the boxes X, Y, or Z has the LEAST mass?

A. Box X

B. Box Y

C. Box Z

D. All three boxes have the same mass.

17. Whenever scientists carefully measure any quantity many times, they expect that

A. All of the measurements will be exactly the same.

B. Only two of the measurements will be exactly the same.

C. All but one of the measurements will be exactly the same.

D. Most of the measurements will be close but not exactly the same.

18. Iris found a piece of igneous rock while on vacation. Which of the following places was she most likely visiting?

A. The marble ruins in Greece

B. An inactive volcano in Hawaii

C. The limestone caves of Kentucky

D. A sandy beach in California

19. Keisha is pushing her bicycle up a hill. Where does Keisha get the energy to push her bicycle?

A. From the food she has eaten

B. From the exercise she did earlier

C. From the ground she is walking on

D. From the bicycle she is pushing


20. The diagram to the left shows two magnets. Which magnet would hold more paper clips?

A. Magnet 1

B. Magnet 2

C. They would hold the same number of paper clips.

D. There is not enough information to answer this question.

21. Which of the following is a behavior of a bird?

A. A tree

B. Singing

C. Nest

D. Worms

22. Juan thinks that water will evaporate faster in a warm place than in a cool one. He has two identical bowls and a bucket of water. He wants to do an experiment to find out if he is correct. Which of the following should he do?

A. Place two bowls with the same amount of water in a warm place.

B. Place a bowl of water in a cool place and a bowl with twice the amount of water in a warm place.

C. Place a bowl of water in a cool place and a bowl with half of the amount of water in a warm place.

D. Place a bowl of water in a cool place and a bowl with the same amount of water in a warm place.


23. The diagram shows the life cycle of a mealworm. What would happen if this larva were eaten by a bird?

A. The mealworm would die before it could reproduce.

B. The bird would become sick.

C. The mealworm species would be wiped out.

D. The mealworm eggs would be spread by the bird.

24. Ken put a thermometer in a glass filled with hot water. Why does the liquid inside the thermometer rise?

A. Gravity pushes it up.

B. Air bubbles are released.

C. Heat from the water makes it expand.

D. Air pressure above the water pulls it up.

25. The picture shows a circuit for testing whether an object or material conducts electricity. When the wires touch the paper clip the bulb lights. What other objects will light the bulb?

A. Both a rubber band and a wooden cube

B. Both a penny and a silver spoon

C. Both a marble and a ping pong ball

D. Both a dime and a piece of thread


26. Beryl finds a rock and wants to know what kind it is. What information about the rock will best help her to identify it?

A. The size of the rock

B. The weight of the rock

C. The temperature where the rock was found

D. The minerals the rock contains

27. Some children were trying to find out which of three light bulbs was brightest. Which one of these gives the best START toward finding the answer?

A. “One bulb looks brightest to me, so I already know the answer.”

B. “All the bulbs look bright to me, so there cannot be an answer.”

C. “It would help if we had a way to measure the brightness of a light bulb.”

D. “We can take a vote and each person will vote for the bulb he or she thinks is the brightest.”

28. The Grand Canyon was most likely created by:

A. The plates of the earth moving past each other.

B. Water flowing over rock over a long period of time.

C. An unusually strong earthquake.

D. A glacier moving down from the north during the last ice age.


29. The diagram shows three identical light bulbs, a switch, and a battery. When the switch is closed what will you observe?

A. Bulb A is the brightest because it is closest to

the switch.

B. Bulb B is the brightest because it is getting electricity from both sides.

C. Bulb C is the brightest because it is closest to the battery.

D. All three are equally bright.

30. When you are riding a bicycle at night, your bicycle’s reflectors help people in cars see your bicycle. How do bicycle reflectors work?

A. They are made of a special material that gives

off its own light.

B. They are connected to batteries that allow them to produce light.

C. They bounce light back from other sources.

D. They are covered with paint that glows in the dark.

31. A girl wanted to play on a seesaw with her little brother. Which picture shows the best way for the girl, who weighed 50 kg (kilograms), to balance her brother, who weighed 25 kg?

B. A.

C. .

© Horizon Research, Inc. Page 10

D

Do Not Copy Without Permission

32. A student observed a spider and its web. Which of the following is NOT an observation?

A. The web has some threads that are straight.

B. The spider has eight legs.

C. The spider did not make any noise.

D. The spider evolved from insects.

33. If you wanted to be able to look at the stars, the planets, and the Moon more closely, what should you use?

A. Telescope

B. Periscope

C. Microscope

D. Magnifying glass

34. In each of diagrams to the left, arrows are used to represent the direction of electricity passing through the wires. Which diagram shows the direction that the electricity flows when they light up the bulb?

A. Diagram A

B. Diagram B

C. Diagram C

D. None are correct

35. What do playing a guitar, banging a drum, and dropping a pebble in the water have in common?

A. They all produce light.

B. They all cause vibrations.

C. They all convert heat to energy.

D. They all need gravity to move.


36. Which of the following living things in the pond

system uses the energy from sunlight to make its own food?

A. Insect

B. Frog

C. Water lily

D. Small fish

37. Which of the following should a science class do to find out which wind direction is most common during times of cloudy skies and wet weather in their town?

A. Check a weathervane, thermometer, and barometer daily.

B. Make a chart of the different cloud formations shown in an encyclopedia.

C. Keep a record of daily rainfall for an entire year.

D. Record wind direction, cloud conditions, and rainfall daily for at least four months.

38. Look at the picture of a sprouting seed. What is happening?

A. It is swelling and making flowers.

B. The seed coat is coming off and pulling in a stem.

C. It is performing photosynthesis and fruits are forming.

D. The root is growing down and the seed coat is coming off.


39. There is a thunderstorm close to your house. The windows rattle at the same time that you hear the thunder. What causes the windows to rattle?

A. Sound waves from the thunder

B. Light from the lightning

C. Rain from the clouds

D. The high humidity during the storm

40. According to the information in the table, which of the following should you do to decide whether an unknown powder is powder X or powder Y?

Property Powder X Powder Y Color White White Melting Point 80 °C 120 °C Shape Crystals Crystals Mixed with Water Dissolves Dissolves

A. Examine the shape of the grains of powder.

B. Check the color of the powder.

C. Measure the melting point of the powder.

D. Dissolve the powder in water.

41. Sally found a rock near the street by school. What two things below are most important for figuring out what type of rock she has found?

1. The size of the rock 2. The color of the rock 3. The texture of the rock 4. The shape of the rock

A. 1 and 2

B. 2 and 3

C. 2 and 4

D. 3 and 4


42. A bar magnet was placed near two objects. Object

A did not move. Object B moved away form the magnet. Which of the following could explain the motion of the two objects?

A. Object A is made of iron and Object B is a

magnet.

B. Object A is made of wood and Object B is made of iron.

C. Object A is made of wood, and Object B is a magnet.

D. Both objects are made of iron.

E. Both objects are magnets.

43. Seeds develop from which part of a plant?

A. Flower

B. Leaf

C. Root

D. Stem

44. Mr. Smith’s class wanted to learn about the ducks in a pond near their school. The class decided to visit the pond each month during the school year, starting in September. Which of the following questions could the class answer by making observations?

A. Which part of the pond do the ducks use for feeding?

B. How many years have pairs of ducks been together?

C. How many ducks were there last May?

D. Where do the ducks go when they migrate?


45. This table shows the temperature and precipitation (rain and snow) in four different towns on the same day. Where did it snow?

A. Town A

B. Town B

C. Town C

D. Town D

46. A block is placed in a beaker of fresh water. It floats in the water as shown. If the same block is placed in oil and in seawater, it will:

A. Float on the top of both.

B. Sink to the bottom in both.

C. Float on the top of the oil and sink to the bottom in the seawater.

D. Float lower or sink in the oil and float higher in the seawater.

47. Gregory built a circuit like the one shown that has a switch between the battery and the light bulb. When the switch is closed, the electrons flow through the wires. Where are the electrons when the switch is open?

A. The electrons are located throughout the wires, but they are not moving.

B. The electrons are resting in the battery.

C. There are no electrons in the circuit when the switch is open.

D. The electrons are moving through the wires, and escaping through the gap into the air.


48. What property of water is most important for living organisms?

A. It is odorless.

B. It does not conduct electricity.

C. It is tasteless.

D. It is liquid at most temperatures on Earth.

49. Person A and Person B hold identical magnets near a refrigerator as in the diagram to the left. Which person feels a stronger pull on the magnet?

A. Person A.

B. Person B.

C. They feel the same pull.

D. Neither feels a pull.

50. This question refers to an experiment in which moths were captured by attracting them to either white or yellow light. The graph shows that

A. Moths prefer yellow light.

B. Only white moths are attracted to white light.

C. The number of moths captured per day using yellow light decreased after Day 5.

D. The number of moths captured per day using white light decreased during the experiment.


51. A curved groove is placed on a level table as shown in the diagram to the left. A ball is pushed in the groove at P, so that it leaves at Q. These diagrams show the level table and the groove from above. Which shows how the ball will move when it leaves the groove?

52. You want to find out which will empty from a can fastest—water, alcohol, cooking oil, syrup, or soda pop. To answer this question you will need equal amounts of the liquids as well as:

A. A can with a hole in the bottom and a stopwatch

B. A stopwatch only

C. Cans with different size holes

D. Cans of different sizes, one for each liquid


53. Mr. Conrad wanted to help his students understand how water takes part in the weathering of rocks. Mr. Conrad filled a glass jar with water and screwed the top on the jar. Then he put the jar in a clear plastic box, closed the lid, and put the box in the freezer. About three hours later, the students looked in the freezer. They observed that the water was frozen and the glass jar broken into many pieces. What property of water, that makes it an important weathering agent, was demonstrated by Mr. Conrad?

A. Water is a liquid, so that it can get into cracks easily.

B. Water is a solvent so when it gets into cracks in rocks it dissolves rock material and makes the cracks larger.

C. Water expands on freezing so when it gets into cracks in rocks and freezes, it exerts lots of force on the rock making it crack even more.

D. Water is all over the surface of the earth so it has many opportunities to interact with rock, including in frozen areas.

54. Data about the hair color of fifteen students are shown in the table. Which of the following bar graphs represents the data shown in the table?

END © Horizon Research, Inc. Page 18 Do Not Copy Without Permission

Local Systemic Change 2003–04 Middle School Science Assessment

TEST BOOKLET To the student: Mark your answers on the answer sheet. Fill in only one circle for each question. Please do not write on this test booklet; you may use scrap paper if needed. Please think carefully about your answers. Do your best, but do not be concerned if you do not know the answers to all of the questions. This test was designed for students in multiple grades and classes who may or may not have studied the topics being assessed. If you finish before time is called, you should go over your work again.

WHEN YOUR TEACHER TELLS YOU TO, TURN THE PAGE AND BEGIN


Copyright ©2003 by Horizon Research, Inc.

All rights reserved. No part of this assessment may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from Horizon Research, Inc. For information contact:

Horizon Research, Inc. 326 Cloister Court Chapel Hill, NC 27514-2296 Email: [email protected]


mailto:[email protected]

1. In the pond ecosystem pictured to the left, which

of the following living things uses the energy from sunlight to make its own food?

A. Insect B. Frog C. Water lily D. Small fish

2. A cupful of water and a similar cupful of

gasoline were placed on a table near a window on a hot sunny day. A few hours later it was observed that both the cups had less liquid in them, but that there was less gasoline left than water. What does this experiment show?

A. All liquids evaporate. B. Gasoline gets hotter than water. C. Some liquids evaporate faster than others. D. Liquids will only evaporate in sunshine. E. Water gets hotter than gasoline.

3. If you are burning wood, the reaction will:

A. Release energy. B. Absorb energy. C. Neither absorb nor release energy. D. Sometimes release and sometimes absorb

energy, depending on the kind of wood.


4. Todd’s father is driving him to school, when

suddenly Todd realizes that he left his backpack on top of the car. He tells his father that it is on the roof, and his father stops the car very quickly. What happens to the backpack when Todd’s father hits the brakes?

A. It falls off the front of the car, because there

is no force to stop its forward motion. B. It falls off the back of the car, because the

quick stop creates a backward force. C. It falls off the car, but it is not possible to

know in which direction it will fall. D. It stays on top of the car, because there is no

longer a horizontal force acting on it.

5. Why do mountain climbers use oxygen

equipment at the top of the world’s highest mountains?

A. There is less oxygen in the air at great

heights. B. There is little nitrogen in the air at great

heights. C. There is a hole in the ozone layer. D. There is no air at the top of very high

mountains.

6. What feature is shared by ALL insects?

A. External skeleton B. Two pairs of wings C. Jumping legs D. Stinging mechanism


7. A student put 100 mL of water in each of the

open containers and let them stand in the sun for one day. Which container would probably lose the most water due to evaporation?

A. Container A B. Container B C. Container C D. Container D

8. A tight metal lid on a jar of pickles may loosen

when being held in hot water. This is because the hot water causes the:

A. Glass jar to contract. B. Metal lid to contract. C. Glass jar to expand more than the metal lid

expands. D. Metal lid to expand more than the glass jar

expands.

9. The graph shows the progress made by an ant

moving along a straight line. If the ant keeps moving at the same speed, how far will it have traveled at the end of 30 seconds?

A. 5 cm B. 6 cm C. 20 cm D. 30 cm


10. Which of the following BEST explains why

you do not need to eat all of the time?

A. You do not need energy to do some activities.

B. You can feel energetic, even when you do

not have much energy. C. Your body also gets energy from rest and

exercise. D. Your body is able to save some energy to

use later on.

11. Linda pushes a shopping cart with a constant

amount of force. She notices that the cart is speeding up. What can you conclude?

A. The cart must be very light. B. The cart must be very heavy. C. Linda’s force on the cart must be larger

than the opposing forces (such as friction and drag).

D. Linda’s force on the cart is exactly

balanced by the opposing forces (such as friction and drag).

12. What is the primary function of the large leaves found on seedlings growing in a forest?

A. To provide shade for the root systems B. To get rid of excess water that is entering

through the roots C. To allow for leaf damage by insects D. To gather as much light as possible for

photosynthesis


13. A metal spoon, a wooden spoon, and a plastic

spoon are placed in hot water. After 15 seconds, which spoon will feel hottest?

A. The metal spoon B. The wooden spoon C. The plastic spoon D. The three spoons will feel the same.

14. Which of the following foods gives you the

MOST energy per ounce?

A. Apple B. Brownie C. Cucumber D. Water

15. Sarah is on her school’s soccer team. One day

she is playing defense and kicks the ball to the other end of the field. Which statement describes the force acting on Sarah’s foot by the ball DURING the kick?

A. The ball does not exert a force on Sarah’s

foot. B. The force by the ball is less than the force of

Sarah’s kick. C. The force by the ball is equal to Sarah’s kick. D. The force by the ball is greater than Sarah’s

kick.


16. The male insects in a population are treated to

prevent sperm production. Would this reduce this insect population?

A. No, because the insects would still mate. B. No, because it would not change the

offspring mutation rate. C. Yes, because it would sharply decrease the

reproduction rate. D. Yes, because the males would die.

17. Which of the following is most consistent with

the modern theory of evolution?

A. Parents pass their physical traits to their offspring; those offspring with traits that help them survive in the environment are able to reproduce.

B. Parents change their physical traits in order to

survive in the environment, then those parental traits are passed to their offspring.

C. Life on this planet came from another planet

far out in space. D. Living organisms have not changed for

hundreds of millions of years.

18. The diagram shows five different Celsius

thermometers. The body temperature of sick people ranges from about 36°C to 42°C. Which thermometer would be most suited for accurately measuring body temperature?

A. Thermometer A B. Thermometer B C. Thermometer C D. Thermometer D E. Thermometer E


19. Bill is trying to push a table to the other side of

the kitchen so he can clean the floor. Which of the following statements is true?

A. The table moves because Bill pushes harder

on the table than it pushes on him. B. The table does NOT move because Bill and

the table push each other with equal force. C. The table moves because Bill is pushing on

it; the force of the table on Bill does not matter.

D. The table moves because Bill is heavier than

the table.

20. The diagrams show different trials Abdul carried

out with carts having different-sized wheels. He started them from different heights and the blocks he put in them were of equal mass.

He wants to test this idea: The heavier a cart is,

the greater its speed at the bottom of a ramp. Which three trials should he compare?

A. G, T, and X B. O, T, and Z C. R, U, and Z D. S, T, and U E. S, W, and X

21. Which statement explains why daylight and

darkness occur on Earth?

A. The Earth rotates on its axis. B. The Sun rotates on its axis. C. The Earth’s axis is tilted. D. The Sun revolves around the Earth. E. The Earth revolves around the Sun.


22. Carly gets on her skateboard and pushes off with

her foot. As she glides across the gym floor, she gradually slows down and comes to a stop. Why does she come to a stop?

A. Because the force of motion runs out B. Because the skateboard runs out of its own

force C. Because friction acts against her direction of

motion D. None of the above

23. The Moon produces no light, and yet it shines at

night. Why is this?

A. The Moon has many craters. B. The Moon reflects the light from the Sun. C. The Moon is covered with a thin layer of ice. D. The Moon rotates at a very high speed.

24. When you dissolve sugar in water, sugar

molecules

A. Mix with water molecules. B. Shrink in water molecules. C. Disappear in water molecules. D. Break apart and form new molecules.


25. Which picture shows the best way for the man to

balance a ten-liter bucket of water and a five-liter bucket of water? A. Picture A B. Picture B C. Picture C D. Picture D

26. Based on the weather data in the table to the left,

on which day was snowfall most likely to have occurred?

A. Monday B. Tuesday C. Wednesday D. Thursday

27. Which is the most basic unit of living things?

A. Cells B. Bones C. Tissues D. Organs


28. The total mass of two liquids is 32 grams. When a student combines the liquids in a beaker, she observes bubbles. Then she finds that the mass of the combined liquids is 29 grams. This could be because:

A. Molecules in the system disappeared. B. Atoms in the system packed closer together. C. Atoms in the system were destroyed. D. Molecules escaped from the system.

0

25

50

75

100

125

0 15 30 45 60 75

Time (minutes)

Dis

tanc

e (k

ilom

eter

s)

29. The graph to the left shows the progress made by

a car traveling along a straight road. What is the speed of the car?

A. 25 kilometers per hour B. 50 kilometers per hour C. 75 kilometers per hour D. 100 kilometers per hour

30. Your science textbook is sitting on your desk.

Which of the following statements describes the forces on the book?

A. The only force acting on the book is gravity,

which pulls it down. B. The force of the desk pushes the book up

more strongly than the force of gravity pulls it down.

C. The force of gravity pulls the book down

with the same strength that the force of the desk pushes it up.

D. The force of gravity pulls the book down

more strongly than the force of the desk pushes it up.

E. There are no forces acting on the book.


31. Which of the following would be the best model

to show the interactions between water and the Sun’s heat energy in cycles of precipitation?

A. Model A B. Model B C. Model C D. Model D


32. Joan is sitting next to David at the dinner table.

She asks David to pass the salt, so he slides the salt shaker along the table toward Joan. As the salt shaker moves to the right, in which direction does the force of friction act?

A. To the right only B. To the left only C. Both directions (right and left) equally D. Both directions, but stronger to the left than

the right E. Both directions, but stronger to the right than

the left

33. The picture shows the three main layers of the

Earth. Where is it the hottest?

A. Layer A B. Layer B C. Layer C D. All three layers are the same temperature.

34. Which is an example of a chemical reaction?

A. The melting of ice B. The grinding of salt crystals to powder C. The burning of wood D. The evaporation of water from a puddle


35. How are warm-blooded animals different from

cold-blooded animals?

A. Warm-blooded animals have a higher metabolism in warm weather.

B. Warm-blooded animals are more aggressive

in captivity. C. Warm-blooded animals always have a higher

blood temperature. D. Warm-blooded animals normally maintain a

fairly constant internal temperature at all air temperatures.

E. Warm-blooded animals are found only in

warm climates.

36. Whenever scientists carefully measure any

quantity many times, they expect that:

A. All of the measurements will be exactly the same.

B. Only two of the measurements will be

exactly the same. C. All but one of the measurements will be

exactly the same. D. Most of the measurements will be close, but

not exactly the same.

37. The source of energy for the Earth’s water cycle

is the:

A. Wind. B. Sun’s radiation. C. Earth’s radiation. D. Sun’s gravity.


38. Maria is asked by her teacher to move a file

cabinet. As she pushes, the cabinet begins to move and speed up. What will happen if she keeps pushing just as hard as she did when it first started to move?

A. The file cabinet will keep going at the initial

speed. B. The file cabinet will speed up until she gets

to the speed that matches the push. C. The file cabinet will go faster and faster until

she pushes less. D. The file cabinet will reach a constant speed as

her push is used up.

39. Water (H2O) cannot be turned into salt (NaCl)

through a chemical reaction because:

A. Water contains liquid atoms and salt contains solid atoms. B. Water is made of three atoms. C. Salt is a mixture of atoms. D. Salt and water are made of different atoms.

40. The Earth’s Moon is:

A. Always much closer to the Sun than it is to the Earth.

B. Always much closer to the Earth than it is to

the Sun. C. About the same distance from the Sun as it is

from the Earth. D. Sometimes closer to the Sun than it is to the

Earth and sometimes closer to the Earth than it is to the Sun.


41. The primary reason scientists repeat the

measurements they take during experiments is so that they can:

A. Check that the equipment is working. B. List all the results in a table. C. Estimate experimental error. D. Change the experimental conditions.

42. Martha collected the gas given off by a glowing

piece of charcoal. The gas was then bubbled through a small amount of colorless limewater. Part of Martha’s report stated, “After the gas was put into the jar, the limewater gradually changed to a milky white color.” This statement is:

A. A hypothesis. B. A conclusion. C. A generalization. D. An assumption of the investigation. E. An observation.

43. On which day was the average windchill

temperature likely to be the lowest?

A. Monday B. Tuesday C. Wednesday D. Friday


44. A chemical reaction always results in a:

A. Mixture. B. Phase change. C. New substance. D. Different mass.

45. If the forces on an object are balanced, then it is

safe to say that the object is:

A. At rest. B. Accelerating. C. Moving at a steady speed. D. Either at rest or moving at a steady speed.

46. Gina had an idea that plants needed minerals

from the soil for healthy growth. She placed a plant in the Sun, as shown in the diagram to the left. In order to check her idea, she also needed to use another plant. Which of the following should she use?


47. Which of the following is an important factor in

explaining why seasons occur on Earth?

A. The Earth rotates on its axis. B. The Sun rotates on its axis. C. The Earth’s axis is tilted. D. The Sun’s axis is tilted.

48. The drawings show a rocket being launched from

Earth and then returning. In which of the three positions does gravity act on the rocket?

A. 3 only B. 1 and 2 only C. 2 and 3 only D. 1, 2, and 3

49. A chemical reaction occurs when substances:

A. Break apart and the atoms change into new atoms.

B. Mix together and the molecules stay the

same. C. Become hotter and the molecules spread

apart. D. Interact and the atoms composing them

recombine.


50. Which statement is always true about

conservation of mass?

A. The total mass of the reactants is equal to the total mass of the products.

B. The mass changes in a phase change, but not

in a chemical reaction. C. The total mass of a system changes in a

chemical reaction. D. The mass of one reactant is equal to the mass

of one product.

51. Angela and Shelly are playing with a hula-hoop

and a marble. They lay the hula-hoop down on the ground and roll the marble counterclockwise around the inside of the hoop. At the instant shown in the picture to the left, they quickly lift the hoop off the ground. Which path will the marble take when the hoop is lifted off the ground?

A. Path A B. Path B C. Path C D. Path D E. Path E F. Path F

52. What is the main function of chloroplasts in a plant cell?

A. To absorb light energy and manufacture food B. To remove waste materials by active

transport C. To manufacture chemical energy from food D. To control the shape of the cell


53. Jennifer and Katie stand and lean on each other

so that they are balanced. Jennifer weighs 150 pounds and Katie weighs 120 pounds. Which one pushes harder on the other?

A. Jennifer and Katie push on each other with

the same size force. B. Katie pushes harder because she weighs less. C. Jennifer pushes harder because she weighs

more. D. Neither Jennifer nor Katie is exerting a force,

because they are balanced.

54. If the locations of earthquakes over the past ten

years were plotted on a world map, which of the following would be observed?

A. Earthquakes occur with the same frequency

everywhere on Earth. B. Earthquakes generally occur along the edges

of tectonic plates. C. Earthquakes most frequently occur near the

middle of continents. D. Earthquakes do not seem to occur in any

consistent pattern.

55. Alexander Fleming noticed that bacteria growing

on a plate of agar did not grow next to a mold that was growing on the same plate. He wrote in his laboratory report: “The mold may be producing a substance that kills bacteria.” This statement is best described as:

A. An observation. B. A hypothesis. C. A generalization. D. A conclusion.


56. Jeremy is pedaling his bike so that he is riding

at a constant speed. Why is the bike moving at a constant speed?

A. There are not any forces being exerted on the

bike. B. There is a net forward force on the bike. C. The forces pushing the bike forward are

getting larger and larger. D. The forces on the bike are balanced.

57. Two open bottles, one filled with vinegar and the other with olive oil, were left on a window sill in the Sun. Several days later it was observed that the bottles were no longer full. What can be concluded from this observation?

A. Vinegar evaporates faster than olive oil. B. Olive oil evaporates faster than vinegar. C. Both vinegar and olive oil evaporate. D. Only liquids containing water evaporate. E. Direct sunlight is needed for evaporation.

58. All of the following are true statements about

energy EXCEPT?

A. Energy comes in many forms, including light and sound energy. B. Energy can be destroyed when it changes from one form to another. C. Energy can change from one form to another in living things. D. Energy cannot be created. It can only be changed from one form to another.


59. During a chemical reaction, the atoms that make

up the reactants change

A. Shape. B. Arrangement. C. Size. D. Color.

60. Insecticides are used to control insect

populations so that they do not destroy crops. Over time, some insecticides become less effective at killing insects, and new insecticides must be developed. What is the most likely reason insecticides become less effective over time?

A. Surviving insects have learned to include

insecticides as a food source. B. Surviving insects pass their resistance to

insecticides to their offspring. C. Insecticides build up in the soil. D. Insecticides are concentrated at the bottom of

the food chain.

END


Appendix B

Description of Methods Used to Conduct the IRT Analysis

Item response theory (IRT) is the study of test and item scores based on assumptions concerning the mathematical relationship between abilities (or other hypothesized traits) and item responses. The probabilities of these responses are modeled as a function of separate parameters for the item and the person; item parameters represent such properties as item difficulty and discrimination whereas the person parameters generally represent the ability level of the examinee. Thorough dimensionality assessments were performed for elementary school students’ data from the previous year, 2002-03.* For the 2003-04 elementary school data, an exploratory factor analysis was conducted to determine whether the previous dimensionality results could be applied to these data. The exploratory factor analysis, conducted in MPLUS version 3.11, suggested a dominant first factor, an indicator of unidimensionality. Because the middle school assessment was new in 2003–04, several dimensionality analyses were conducted for these data. As in the elementary school data, an exploratory factor analyses, conducted in MPLUS version 3.11, suggested a dominant first factor. A DETECT† index of 0.15 also suggested the middle-school data were essentially unidimensional. After unidimensionality was confirmed, BILOG-MG‡ was used to fit a three-parameter IRT model to both elementary and middle school data. BILOG-MG provides estimates of each student’s latent ability, and these estimates along with the item parameters were used to calculate true test scores. Items 7, 15, and 20, were dropped from the elementary school item pool, and items 15, 19, 30, 38, 45 and 53 were dropped from the middle school assessment due to their low discrimination (< 0.65). For post-test data, item parameter estimates were fixed to their pre-test values and ability estimates calculated via BILOG-MG. For both the elementary and middle school data, the item and ability estimates from the three-parameter IRT model were used to create IRT true scores for pre- and post-test data.

* Germuth, A.A. & Banilower, E.R. (2004). Results of the 2002–03 Study of the Impact of the Local Systemic Change Initiative on Student Achievement in Science. Chapel Hill, NC: Horizon Research Inc. † Stout, W. (1999). DETECT: IRT-Based Educational and Psychological Measurement Software. William Stout Institute for Measurement. ‡ Zimowski, M., Muraki, E., Mislevy, R., & Bock, D. (2002). BILOG-MG (Version 3.0) [Computer Software]. Lincolnwood, IL: Scientific Software International, Inc.

Horizon Research, Inc. B-1 June 2004

Appendix C

Teacher and Student Questionnaires

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

7. How many students are enrolled in this class, including studentsabsent on the day of the test? (Please enter your answer in thespaces provided, then darken the corresponding oval in eachcolumn. Enter your answer as a 2-digit number; e.g., if 9 students,enter as 09.)

PLEASE DO NOT WRITE IN THIS AREA

B. Your Science Teaching

3 4 5 6 76. What grade levels are represented in this class?(Darken all ovals that apply.)

0-2 3-5 6-10 11-15 16-20 21-25 26 or more

5. How many years have you taught prior to this school year? (Darken one oval.)

2. Race - Are you: (Darken one or more.)

a. Life scienceb. Earth and space sciencec. Physical science

Yes No

4. Did your college science coursework include the equivalent of at least one semester of: (Darken one oval on each line.)

None1 semester2 semesters

3. How many college science courses have you completed? (Darken one oval.)

Male Female

2003-2004 LSC Elementary Science Program Study*Pre-Test Teacher Questionnaire

Instructions: Please use a #2 pencil or a blue or black pen to complete this

questionnaire. Darken ovals completely, but do not stray intoadjacent ovals. Be sure to erase completely any stray marks.

A. Teacher Demographic Information

1. Are you:

American Indian or Alaskan NativeAsianBlack or African-American

Hispanic or LatinoNative Hawaiian or Other Pacific IslanderWhite

3 semesters4 semesters5 or more semesters

* See the cover letter accompanying this questionnaire for a description of the LSC.

0123456789

0123456789

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Example: Characteristics of Plants Harcourt; STC 5

10. How many science units has this class worked on so far this academic year? Include units you have begun but notcompleted. Do not include units you are planning, but have not yet begun. (We are defining a "unit" as a series ofrelated activities, often on a single topic such as sound, rocks, or genetics.) (Darken one oval.)

0 1 2 3 4 5 6 7 8 9 or more

11a. For each of the life science units this class has worked on so far this year, list the title/topic, the publisher of each of theinstructional materials used to teach the unit, and the number of lessons devoted to it. Include units you have begun but notcompleted; do not include units you are planning, but have not yet started. If you have not taught any life science units yet thisyear, skip to question 12a.

11b. Approximately what percent of this life science instruction has been based on LSC*-designated instructional materials? (Darken one oval.)

0 10 20 30 40 50 60 70 80 90 100

Life Science Unit - Title Number of LessonsLife Science Unit - Publisher(s)

* See the cover letter accompanying this questionnaire for a description of the LSC.DesignExpert™ by NCS Printed in U.S.A. Mark Reflex® EW-239978-1A:654321

41-50 81 or more

9. Approximately how many minutes is a typical science lesson in this class? (Darken one oval.)

21-3010 or fewer 31-40 51-6011-20 71-8061-70

Average Number of Minutes per Lesson

i)

ii)

iii)

iv)

8. Have you taught any science in this class so far this academic year? (Darken one oval.)

YesNo (skip to question 14)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Example: Earth Features EDC Insights; Silver-Burdett 9

Example: Magnets and Motors FOSS; Scott Foresman; & teacher-made materials 8

12b. Approximately what percent of this physical science instruction has been based on LSC*-designated instructional materials? (Darken one oval.)


13b. Approximately what percent of this earth/space science instruction has been based on LSC*-designated instructionalmaterials? (Darken one oval.)

0 10 20 30 40 50 60 70 80 90 100

13a. For each of the earth/space science units this class has worked on so far this year, list the title/topic, the publisher of eachof the instructional materials used to teach the unit, and the number of lessons devoted to it. Include units you have begunbut not completed; do not include units you are planning, but have not yet started. If you have not taught any earth/spacescience units yet this year, skip to question 14.

Earth/Space Science Unit - Title Number of LessonsEarth/Space Science Unit - Publisher(s)

12a. For each of the physical science units this class has worked on so far this year, list the title/topic, the publisher of each of theinstructional materials used to teach the unit, and the number of lessons devoted to it. Include units you have begun but notcompleted; do not include units you are planning, but have not yet started. If you have not taught any physical science unitsyet this year, skip to question 13a.

0 10 20 30 40 50 60 70 80 90 100

Physical Science Unit - Title Number of LessonsPhysical Science Unit - Publisher(s)

i)

ii)

iii)

iv)

i)

ii)

iii)

iv)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263


Yes No17. Have you been identified as a teacher leader for your district's NSF-supported LSC* project?

200 or greater130-159160-199

80-99100-129

40-5960-79

10-1920-39

01-9

16. Approximately how many total hours have you spent on formal professional development in science/scienceeducation as part of the LSC* since the project began? (Darken one oval.)

Thank You!

C. LSC Professional Development

14. In what year did you begin participating in professional development as part of the LSC*? (Darken one oval.)

19951996199719981999

2000200120022003Have not yet participated in the LSC*. (SKIP to question 17.)

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

d. Other issues related to science/science education

c. Earth/space science content/instructional materials

b. Physical science content/instructional materials

a. Life science content/instructional materials

15. Approximately how many hours have you spent on formal LSC*-provided professional development since the LSCproject began, focused on each of the following?


123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

2. How many students are enrolled in this class, including students absent on the day of the test? (Pleaseenter your answer in the spaces provided, then darken the corresponding oval in each column. Enteryour answer as a 2-digit number; e.g., if 9 students, enter as 09.)


A. Your Science Teaching

3 4 5 6 71. What grade levels are represented in this class?(Darken all ovals that apply.)

LSC Elementary Science Program Study*2003-04 Post-Test Teacher Questionnaire


questionnaire. Darken ovals completely, but do not stray intoadjacent ovals. Be sure to completely erase any stray marks.


0123456789

0123456789

7. How many science units has this class worked on since the pre-test? Include units that you had begun but not completedat the time of the pre-test and units that you are currently working on but have not yet finished. Do not include units youcompleted prior to the pre-test or units you are planning but have not yet started. (We are defining a "unit" as a series ofrelated activities, often on a single topic such as sound, rocks, or genetics.) (Darken one oval.)

0 1 2 3 4 5 6 7 8 9 or more

41-50 81 or more


21-3010 or fewer 31-40 51-6011-20 71-8061-70




5. Have you been the teacher responsible for teaching science to these students for the entire academic year? (Darken one oval.)

Yes (skip to question 7)No

Questions 7 through 10b ask about your science teaching since the pre-test. To assist you in answering these questions, wehave enclosed your pre-test responses to these questions (see the purple enclosure).

6a. Approximately what percent of these students' science instruction this year were you responsible for providing?0 25 50 75 100

6b. Please provide a brief description of who, other than yourself, provided science instruction for these students (e.g., "I had a student teacher for 4 weeks"):

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263


Example: Characteristics of Plants Harcourt; STC 5

8a. For each of the life science units this class has worked on since the pre-test, list the title/topic, the publisher of each of theinstructional materials used to teach the unit, and the number of lessons devoted to it since the pre-test. Include units that you hadbegun but not completed at the time of the pre-test and units that you are currently working on but have not yet finished. Do notinclude units you completed prior to the pre-test or units you are planning but have not yet started.


i)

ii)

iii)

iv)

0 10 20 30 40 50 60 70 80 90 100

8b. Approximately what percent of your life science instruction since the pre-test has been based on LSC*-designatedinstructional materials? If you have not taught any life science since the pre-test, skip to question 9a. (Darken one oval.)

Example: Magnets and Motors FOSS; Scott Foresman; & teacher-made materials 8

9b. Approximately what percent of your physical science instruction since the pre-test has been based on LSC*-designatedinstructional materials? If you have not taught any physical science since the pre-test, skip to question 10a. (Darken one oval.)

9a. For each of the physical science units this class has worked on since the pre-test, list the title/topic, the publisher of each ofthe instructional materials used to teach the unit, and the number of lessons devoted to it since the pre-test. Include units thatyou had begun but not completed at the time of the pre-test and units that you are currently working on but have not yetfinished. Do not include units you completed prior to the pre-test or units you are planning but have not yet started.

0 10 20 30 40 50 60 70 80 90 100


i)

ii)

iii)

iv)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263



Example: Earth Features EDC Insights; Silver-Burdett 9

10b. Approximately what percent of your earth/space science instruction since the pre-test has been based on LSC*-designatedinstructional materials? If you have not taught any earth/space science since the pre-test, skip to question 11. (Darken oneoval.)

0 10 20 30 40 50 60 70 80 90 100

10a. For each of the earth/space science units this class has worked on since the pre-test, list the title/topic, the publisher ofeach of the instructional materials used to teach the unit, and the number of lessons devoted to it since the pre-test. Include units that you had begun but not completed at the time of the pre-test and units that you are currently working onbut have not yet finished. Do not include units you completed prior to the pre-test or units you are planning but have notyet started.


i)

ii)

iii)

iv)


12. About how often do students in this class take part in each of thefollowing types of activities as part of their science instruction? (Darken one oval on each line.)

11. About how often do you do each of the following in your scienceinstruction in this class? (Darken one oval on each line.)

a. Use the LSC*-designated instructional materials as the basisof science lessons.

b. Arrange seating to facilitate student discussion. c. Use open-ended questions. d. Require students to supply evidence to support their claims. e. Encourage students to explain concepts to one another.f. Encourage students to consider alternative explanations.g. Assign science homework.

Rarely Sometimes Often All or (e.g., a few (e.g., once (e.g., once almost all times a or twice or twice science Never year) a month) a week) lessons

a. Participate in discussions with the teacher to further scienceunderstanding.

b. Work in cooperative learning groups. c. Make formal presentations to the class. d. Answer textbook/worksheet questions. e. Review homework/worksheet assignments.

f. Share ideas or solve problems with each other in small groups. g. Engage in hands-on science activities. h. Design or implement their own investigation. i. Work on models or simulations.


Question 10 continues on next page.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263


B. LSC Professional Development


DesignExpert™ by NCS Printed in U.S.A. Mark Reflex® EW-239978-1B:654321

j. Work on extended science investigations or projects (a week ormore in duration).

k. Participate in field work. l. Write reflections in a notebook or journal.m. Work on portfolios. n. Take short-answer tests (e.g., multiple choice, true/false,

fill-in-the-blank).



199519961997199819992000

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more





14. Approximately how many hours have you spent on formal LSC-provided professional development* since the LSCproject began, focused on each of the following?

Yes No16. Have you been identified as a teacher leader for your district's NSF-supported LSC project?

200 or greater130-159160-199

80-99100-129

40-5960-79

10-1920-39

01-9


12. (continued)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263



C. Teacher Opinions and Preparedness

17. Please indicate how prepared you feel to do each of the following in thegrades you teach. (Darken one oval on each line.)

a. Provide concrete experience before abstract concepts. b. Develop students' conceptual understanding of science. c. Take students' prior understanding into account when planning curriculum

and instruction. d. Make connections between science and other disciplines. e. Have students work in cooperative learning groups.

f. Have students participate in appropriate hands-on activities. g. Engage students in inquiry-oriented activities. h. Engage students in applications of science in a variety of contexts. i. Use performance-based assessment. j. Use portfolios. k. Use informal questioning to assess student understanding.

Not Fairly Very adequately Somewhat well well prepared prepared prepared prepared

DesignExpert™ by NCS Printed in U.S.A. Mark Reflex® EW-239978-1C:654321

19. Please indicate how well prepared you feel to do each of thefollowing. (Darken one oval on each line.)

a. Lead a class of students using investigative strategies. b. Manage a class of students engaged in hands-on/project-based work. c. Help students take responsibility for their own learning. d. Recognize and respond to student diversity. e. Encourage students' interest in science. f. Use strategies that specifically encourage participation of females and

minorities in science. g. Involve parents in the science education of their students.


a. The human body b. Ecology c. Rocks and soils d. Astronomy

e. Processes of change over time (e.g., evolution) f. Mixtures and solutions g. Electricity h. Sound

i. Forces and motion j. Machines k. Engineering and design principles (e.g., structures, models)

18. Within science, many teachers feel better prepared to teach sometopics than others. How well prepared do you feel to teach each ofthe following topics at the grade levels you teach, whether or not theyare currently included in your curriculum?

(Darken one oval on each line.)


Thank you!

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

7. How many students are enrolled in this class, including studentsabsent on the day of the test? (Please enter your answer in thespaces provided, then darken the corresponding oval in eachcolumn. Enter your answer as a 2-digit number; e.g., if 9 students,enter as 09.)


B. Your Science Teaching

6 7 86. What grade levels are represented in this class?(Darken all ovals that apply.)

0-2 3-5 6-10 11-15 16-20 21-25 26 or more

5. How many years have you taught prior to this school year? (Darken one oval.)

2. Race - Are you: (Darken one or more.)

a. Life scienceb. Earth and space sciencec. Physical science

Yes No

4. Did your college science coursework include the equivalent of at least one semester of: (Darken one oval on each line.)

None1 semester2 semesters

3. How many college science courses have you completed? (Darken one oval.)

Male Female

2003-2004 LSC Middle School Science Program Study*Pre-Test Teacher Questionnaire


questionnaire. Darken ovals completely, but do not stray intoadjacent ovals. Be sure to erase completely any stray marks.

A. Teacher Demographic Information

1. Are you:

American Indian or Alaskan NativeAsianBlack or African-American

Hispanic or LatinoNative Hawaiian or Other Pacific IslanderWhite

3 semesters4 semesters5 or more semesters


0123456789

0123456789

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Example: Cells Prentice Hall; FOSS 7

10. How many science units has this class worked on so far this academic year? Include units you have begun but notcompleted. Do not include units you are planning, but have not yet begun. (We are defining a "unit" as a series ofrelated activities, often on a single topic such as sound, rocks, or genetics.) (Darken one oval.)

0 1 2 3 4 5 6 7 8 9 or more

11a. For each of the life science units this class has worked on so far this year, list the title/topic, the publisher of each of theinstructional materials used to teach the unit, and the number of lessons devoted to it. Include units you have begun but notcompleted; do not include units you are planning, but have not yet started. If you have not taught any life science units yet thisyear, skip to question 12a.

11b. Approximately what percent of this life science instruction has been based on LSC*-designated instructional materials? (Darken one oval.)

0 10 20 30 40 50 60 70 80 90 100



41-50 81 or more


21-3010 or fewer 31-40 51-6011-20 71-8061-70


i)

ii)

iii)

iv)



123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Example: Catastrophic Events STC; teacher-made materials 5

Example: Force & Motion Glencoe; Cambridge Physics Outlet 8

12b. Approximately what percent of this physical science instruction has been based on LSC*-designated instructional materials? (Darken one oval.)



13b. Approximately what percent of this earth/space science instruction has been based on LSC*-designated instructionalmaterials? (Darken one oval.)

0 10 20 30 40 50 60 70 80 90 100

13a. For each of the earth/space science units this class has worked on so far this year, list the title/topic, the publisher of eachof the instructional materials used to teach the unit, and the number of lessons devoted to it. Include units you have begunbut not completed; do not include units you are planning, but have not yet started. If you have not taught any earth/spacescience units yet this year, skip to question 14.


12a. For each of the physical science units this class has worked on so far this year, list the title/topic, the publisher of each of theinstructional materials used to teach the unit, and the number of lessons devoted to it. Include units you have begun but notcompleted; do not include units you are planning, but have not yet started. If you have not taught any physical science unitsyet this year, skip to question 13a.

0 10 20 30 40 50 60 70 80 90 100


i)

ii)

iii)

iv)

i)

ii)

iii)

iv)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263



Yes No17. Have you been identified as a teacher leader for your district's NSF-supported LSC* project?

200 or greater130-159160-199

80-99100-129

40-5960-79

10-1920-39

01-9


Thank You!

C. LSC Professional Development


19951996199719981999


0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more





15. Approximately how many hours have you spent on formal LSC*-provided professional development since the LSCproject began, focused on each of the following?

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

2. How many students are enrolled in this class, including students absent on the day of the test? (Pleaseenter your answer in the spaces provided, then darken the corresponding oval in each column. Enteryour answer as a 2-digit number; e.g., if 9 students, enter as 09.)


A. Your Science Teaching

6 7 81. What grade levels are represented in this class?(Darken all ovals that apply.)

LSC Middle School Science Program Study*2003-04 Post-Test Teacher Questionnaire


questionnaire. Darken ovals completely, but do not stray intoadjacent ovals. Be sure to completely erase any stray marks.


0123456789

0123456789

6. How many science units has this class worked on since the pre-test? Include units that you had begun but not completedat the time of the pre-test and units that you are currently working on but have not yet finished. Do not include units youcompleted prior to the pre-test or units you are planning but have not yet started. (We are defining a "unit" as a series ofrelated activities, often on a single topic such as sound, rocks, or genetics.) (Darken one oval.)

0 1 2 3 4 5 6 7 8 9 or more

41-50 81 or more


21-3010 or fewer 31-40 51-6011-20 71-8061-70


4. Have you been the teacher responsible for teaching science to these students for the entire academic year? (Darken one oval.)

Yes (skip to question 6)No

5a. Approximately what percent of these students' science instruction this year were you responsible for providing?

Questions 6 through 9b ask about your science teaching since the pre-test. To assist you in answering these questions, wehave enclosed your pre-test responses to these questions (see the blue enclosure).

5b. Please provide a brief description of who, other than yourself, provided science instruction for these students (e.g., "I had a student teacher for 4 weeks"):

0 25 50 75 100

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263


Example: Cells Prentice Hall, FOSS 7

7a. For each of the life science units this class has worked on since the pre-test, list the title/topic, the publisher of each of theinstructional materials used to teach the unit, and the number of lessons devoted to it since the pre-test. Include units that you hadbegun but not completed at the time of the pre-test and units that you are currently working on but have not yet finished. Do notinclude units you completed prior to the pre-test or units you are planning but have not yet started. If you have not taught any lifescience since the pre-test, skip to question 8a.


i)

ii)

iii)

iv)

0 10 20 30 40 50 60 70 80 90 100

7b. Approximately what percent of your life science instruction since the pre-test has been based on LSC*-designatedinstructional materials? (Darken one oval.)

Example: Forces and Motion Glencoe, Cambridge Physics Outlet 8

8b. Approximately what percent of your physical science instruction since the pre-test has been based on LSC*-designatedinstructional materials? (Darken one oval.)

8a. For each of the physical science units this class has worked on since the pre-test, list the title/topic, the publisher of each ofthe instructional materials used to teach the unit, and the number of lessons devoted to it since the pre-test. Include units thatyou had begun but not completed at the time of the pre-test and units that you are currently working on but have not yetfinished. Do not include units you completed prior to the pre-test or units you are planning but have not yet started. If youhave not taught any physical science since the pre-test, skip to question 9a.

0 10 20 30 40 50 60 70 80 90 100


i)

ii)

iii)

iv)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263



Example: Catastrophic Events STC, teacher-made materials 5

9b. Approximately what percent of your earth/space science instruction since the pre-test has been based on LSC*-designatedinstructional materials? (Darken one oval.)

0 10 20 30 40 50 60 70 80 90 100

9a. For each of the earth/space science units this class has worked on since the pre-test, list the title/topic, the publisher ofeach of the instructional materials used to teach the unit, and the number of lessons devoted to it since the pre-test. Include units that you had begun but not completed at the time of the pre-test and units that you are currently working onbut have not yet finished. Do not include units you completed prior to the pre-test or units you are planning but have notyet started. If you have not taught any earth/space science since the pre-test, skip to question 10.


i)

ii)

iii)

iv)


11. About how often do students in this class take part in each of thefollowing types of activities as part of their science instruction? (Darken one oval on each line.)

10. About how often do you do each of the following in your scienceinstruction in this class? (Darken one oval on each line.)

a. Use the LSC*-designated instructional materials as the basisof science lessons.

b. Arrange seating to facilitate student discussion. c. Use open-ended questions. d. Require students to supply evidence to support their claims. e. Encourage students to explain concepts to one another.f. Encourage students to consider alternative explanations.g. Assign science homework.


a. Participate in discussions with the teacher to further scienceunderstanding.

b. Work in cooperative learning groups. c. Make formal presentations to the class. d. Answer textbook/worksheet questions. e. Review homework/worksheet assignments.

f. Share ideas or solve problems with each other in small groups. g. Engage in hands-on science activities. h. Design or implement their own investigation. i. Work on models or simulations.


Question 11 continues on next page.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263


B. LSC Professional Development



j. Work on extended science investigations or projects (a week ormore in duration).

k. Participate in field work. l. Write reflections in a notebook or journal.m. Work on portfolios. n. Take short-answer tests (e.g., multiple choice, true/false,

fill-in-the-blank).



199519961997199819992000

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more

0 1-4 5-9 10-14 15-19 20-24 25-29 30 or more





13. Approximately how many hours have you spent on formal LSC-provided professional development* since the LSCproject began, focused on each of the following?

Yes No15. Have you been identified as a teacher leader for your district's NSF-supported LSC project?

200 or greater130-159160-199

80-99100-129

40-5960-79

10-1920-39

01-9


11. (continued)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263



C. Teacher Opinions and Preparedness

16. Please indicate how prepared you feel to do each of the following in thegrades you teach. (Darken one oval on each line.)

a. Provide concrete experience before abstract concepts. b. Develop students' conceptual understanding of science. c. Take students' prior understanding into account when planning curriculum

and instruction. d. Make connections between science and other disciplines. e. Have students work in cooperative learning groups.

f. Have students participate in appropriate hands-on activities. g. Engage students in inquiry-oriented activities. h. Engage students in applications of science in a variety of contexts. i. Use performance-based assessment. j. Use portfolios. k. Use informal questioning to assess student understanding.


DesignExpert™ by NCS Printed in U.S.A. Mark Reflex® EW-239978-1C:654321

18. Please indicate how well prepared you feel to do each of thefollowing. (Darken one oval on each line.)

a. Lead a class of students using investigative strategies. b. Manage a class of students engaged in hands-on/project-based work. c. Help students take responsibility for their own learning. d. Recognize and respond to student diversity. e. Encourage students' interest in science. f. Use strategies that specifically encourage participation of females and

minorities in science. g. Involve parents in the science education of their students.


a. The human body b. Ecology c. Rocks and soils d. Astronomy

e. Processes of change over time (e.g., evolution) f. Mixtures and solutions g. Electricity h. Sound

i. Forces and motion j. Machines k. Engineering and design principles (e.g., structures, models)

17. Within science, many teachers feel better prepared to teach sometopics than others. How well prepared do you feel to teach each ofthe following topics at the grade levels you teach, whether or not theyare currently included in your curriculum?

(Darken one oval on each line.)


Thank you!

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Instructions: Please use a #2 pencil. Darken ovals completely, but do not stray into adjacent ovals. Be sure to completely eraseany stray marks.

1. Are you: Boy Girl

2. Which best describes you? (Darken one oval.)

White (not Hispanic)Black (not Hispanic)Hispanic or Latino (someone who is from a Mexican, Mexican American, Chicano, Puerto Rican, Cuban, or other

Spanish or Hispanic background)Asian (someone who is from a Chinese, Japanese, Korean, Filipino, Vietnamese, or other Asian background)Native Hawaiian or Other Pacific Islander (someone who is from Hawaii or other Pacific Island)American Indian or Alaskan Native (someone who is from one of the American Indian tribes or one of the original

people of Alaska)Other, specify _____________________________________________

3. What grade are you in? 4th grade 5th grade 6th grade Other, specify:__________


Continued on back...

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

For Office Use Only

Local Systemic Change2003-04 Elementary Science Program Study

Student Assessment Answer Sheet

Correct:

Incorrect:

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Thank you very much!

A B C D E

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

1) 2) 3) 4) 5)

6) 7) 8) 9) 10)

11) 12) 13) 14) 15)

16) 17) 18) 19) 20)

21) 22) 23) 24) 25)

26) 27) 28) 29) 30)

31) 32) 33) 34) 35)

36) 37) 38) 39) 40)

41) 42) 43) 44) 45)

46) 47) 48) 49) 50)

51) 52) 53) 54)

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Local Systemic Change2003-04 Middle School Science Program Study

Student Assessment Answer Sheet

Instructions: Please use a #2 pencil. Darken ovals completely, but do not stray into adjacent ovals. Be sure to completely eraseany stray marks.

1. Are you: Boy Girl

2. Which best describes you? (Darken one oval.)

White (not Hispanic)Black (not Hispanic)Hispanic or Latino (someone who is from a Mexican, Mexican American, Chicano, Puerto Rican, Cuban, or other

Spanish or Hispanic background)Asian (someone who is from a Chinese, Japanese, Korean, Filipino, Vietnamese, or other Asian background)Native Hawaiian or Other Pacific Islander (someone who is from Hawaii or other Pacific Island)American Indian or Alaskan Native (someone who is from one of the American Indian tribes or one of the original

people of Alaska)Other, specify _____________________________________________

3. What grade are you in? 6th grade 7th grade 8th grade 9th grade


Continued on back...

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

For Office Use Only

Correct:

Incorrect:

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263

Thank you very much!

A B C D

A B C D E

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D E

A B C D

A B C D E

A B C D E

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D E

A B C D

A B C D E

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D

A B C D

A B C D E F

A B C D

A B C D

1) 2) 3) 4) 5)

6) 7) 8) 9) 10)

11) 12) 13) 14) 15)

16) 17) 18) 19) 20)

21) 22) 23) 24) 25)

26) 27) 28) 29) 30)

31) 32) 33) 34) 35)

36) 37) 38) 39) 40)

41) 42) 43) 44) 45)

46) 47) 48) 49) 50)

51) 52) 53) 54) 55)

56) 57) 58) 59) 60)

A B C D

A B C D

A B C D

A B C D E

A B C D

A B C D

A B C D

results of the 2003–04 study of the impact of the local ... · impact of the local systemic...

Documents